Detected Tx Unit Hang in ixgbe, kernel 2.6.25

I'm using a 10Gbps copper(CX4) dual-port NIC from silicomusa.com. It uses the Intel chipset and ixgbe driver. I'm using kernel 2.6.25 plus some hacks (no patches to ixgbe).

This particular test case was to create 500 mac-vlans on each of the two ports and generate UDP traffic between them (I have a version of the send-to-self patch applied to my kernel and enabled.)

During the setup for this test, the interfaces would have been bounced (effectively ifdown, ifup), so that is the reason for the link going up and down.

I noticed 90%+ drop rate when I first started the test, and then after maybe 1-2 minutes, things calmed down and started working. I checked /var/log/messages and saw the messages below.

I previously ran 5Gbps of traffic through the two ports with them acting like a bridge for more than 24-hours without any obvious problems, so I think the hardware is probably OK.

May  6 09:51:41 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:41 simech-ice kernel:   TDH                  <1e>
May  6 09:51:41 simech-ice kernel:   TDT                  <3ff>
May  6 09:51:41 simech-ice kernel:   next_to_use          <3ff>
May  6 09:51:41 simech-ice kernel:   next_to_clean        <1a>
May  6 09:51:41 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:41 simech-ice kernel:   time_stamp           <11e035210>
May  6 09:51:41 simech-ice kernel:   next_to_watch        <1b>
May  6 09:51:41 simech-ice kernel:   jiffies              <11e035862>
May  6 09:51:41 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:41 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:41 simech-ice kernel:   TDH                  <3d6>
May  6 09:51:41 simech-ice kernel:   TDT                  <3b0>
May  6 09:51:41 simech-ice kernel:   next_to_use          <3b0>
May  6 09:51:41 simech-ice kernel:   next_to_clean        <3d2>
May  6 09:51:41 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:41 simech-ice kernel:   time_stamp           <11e035211>
May  6 09:51:41 simech-ice kernel:   next_to_watch        <3d3>
May  6 09:51:41 simech-ice kernel:   jiffies              <11e035887>
May  6 09:51:41 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:46 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:46 simech-ice kernel:   TDH                  <28d>
May  6 09:51:46 simech-ice kernel:   TDT                  <26c>
May  6 09:51:46 simech-ice kernel:   next_to_use          <26c>
May  6 09:51:46 simech-ice kernel:   next_to_clean        <289>
May  6 09:51:46 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:46 simech-ice kernel:   time_stamp           <11e0363e0>
May  6 09:51:46 simech-ice kernel:   next_to_watch        <28a>
May  6 09:51:46 simech-ice kernel:   jiffies              <11e036e8e>
May  6 09:51:46 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:46 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:46 simech-ice kernel:   TDH                  <1bd>
May  6 09:51:46 simech-ice kernel:   TDT                  <19c>
May  6 09:51:46 simech-ice kernel:   next_to_use          <19c>
May  6 09:51:46 simech-ice kernel:   next_to_clean        <1b9>
May  6 09:51:46 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:46 simech-ice kernel:   time_stamp           <11e036346>
May  6 09:51:46 simech-ice kernel:   next_to_watch        <1ba>
May  6 09:51:46 simech-ice kernel:   jiffies              <11e036e9a>
May  6 09:51:46 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:47 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:47 simech-ice kernel:   TDH                  <29e>
May  6 09:51:47 simech-ice kernel:   TDT                  <27c>
May  6 09:51:47 simech-ice kernel:   next_to_use          <27c>
May  6 09:51:47 simech-ice kernel:   next_to_clean        <29a>
May  6 09:51:47 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:47 simech-ice kernel:   time_stamp           <11e0363e0>
May  6 09:51:47 simech-ice kernel:   next_to_watch        <29b>
May  6 09:51:47 simech-ice kernel:   jiffies              <11e036fee>
May  6 09:51:47 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:47 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:47 simech-ice kernel:   TDH                  <33f>
May  6 09:51:47 simech-ice kernel:   TDT                  <321>
May  6 09:51:47 simech-ice kernel:   next_to_use          <321>
May  6 09:51:47 simech-ice kernel:   next_to_clean        <33b>
May  6 09:51:47 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:47 simech-ice kernel:   time_stamp           <11e0363e2>
May  6 09:51:47 simech-ice kernel:   next_to_watch        <33c>
May  6 09:51:47 simech-ice kernel:   jiffies              <11e036ff5>
May  6 09:51:47 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:51 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:51 simech-ice kernel:   TDH                  <398>
May  6 09:51:51 simech-ice kernel:   TDT                  <374>
May  6 09:51:51 simech-ice kernel:   next_to_use          <374>
May  6 09:51:51 simech-ice kernel:   next_to_clean        <394>
May  6 09:51:51 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:51 simech-ice kernel:   time_stamp           <11e037748>
May  6 09:51:51 simech-ice kernel:   next_to_watch        <395>
May  6 09:51:51 simech-ice kernel:   jiffies              <11e038251>
May  6 09:51:51 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:51:51 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:51:51 simech-ice kernel:   TDH                  <101>
May  6 09:51:51 simech-ice kernel:   TDT                  <dd>
May  6 09:51:51 simech-ice kernel:   next_to_use          <dd>
May  6 09:51:51 simech-ice kernel:   next_to_clean        <fd>
May  6 09:51:51 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:51:51 simech-ice kernel:   time_stamp           <11e037743>
May  6 09:51:51 simech-ice kernel:   next_to_watch        <fe>
May  6 09:51:51 simech-ice kernel:   jiffies              <11e03825c>
May  6 09:51:51 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:00 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:00 simech-ice kernel:   TDH                  <2b5>
May  6 09:52:00 simech-ice kernel:   TDT                  <292>
May  6 09:52:00 simech-ice kernel:   next_to_use          <292>
May  6 09:52:00 simech-ice kernel:   next_to_clean        <2b1>
May  6 09:52:00 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:00 simech-ice kernel:   time_stamp           <11e038937>
May  6 09:52:00 simech-ice kernel:   next_to_watch        <2b2>
May  6 09:52:00 simech-ice kernel:   jiffies              <11e03a29c>
May  6 09:52:00 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:00 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:00 simech-ice kernel:   TDH                  <8>
May  6 09:52:00 simech-ice kernel:   TDT                  <3e6>
May  6 09:52:00 simech-ice kernel:   next_to_use          <3e6>
May  6 09:52:00 simech-ice kernel:   next_to_clean        <4>
May  6 09:52:00 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:00 simech-ice kernel:   time_stamp           <11e038957>
May  6 09:52:00 simech-ice kernel:   next_to_watch        <5>
May  6 09:52:00 simech-ice kernel:   jiffies              <11e03a2d5>
May  6 09:52:00 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:11 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May  6 09:52:11 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May  6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May  6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:11 simech-ice kernel:   TDH                  <18c>
May  6 09:52:11 simech-ice kernel:   TDT                  <12a>
May  6 09:52:11 simech-ice kernel:   next_to_use          <12a>
May  6 09:52:11 simech-ice kernel:   next_to_clean        <188>
May  6 09:52:11 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:11 simech-ice kernel:   time_stamp           <11e03aa83>
May  6 09:52:11 simech-ice kernel:   next_to_watch        <189>
May  6 09:52:11 simech-ice kernel:   jiffies              <11e03cde1>
May  6 09:52:11 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May  6 09:52:11 simech-ice kernel: ADDRCONF(NETDEV_UP): eth3#435: link is not ready
May  6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:11 simech-ice kernel: ADDRCONF(NETDEV_CHANGE): eth3#435: link becomes ready
May  6 09:52:22 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May  6 09:52:22 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May  6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May  6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:23 simech-ice kernel:   TDH                  <19b>
May  6 09:52:23 simech-ice kernel:   TDT                  <173>
May  6 09:52:23 simech-ice kernel:   next_to_use          <173>
May  6 09:52:23 simech-ice kernel:   next_to_clean        <197>
May  6 09:52:23 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:23 simech-ice kernel:   time_stamp           <11e03d200>
May  6 09:52:23 simech-ice kernel:   next_to_watch        <198>
May  6 09:52:23 simech-ice kernel:   jiffies              <11e03fcd1>
May  6 09:52:23 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May  6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May  6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May  6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:27 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:27 simech-ice kernel:   TDH                  <6>
May  6 09:52:27 simech-ice kernel:   TDT                  <3e4>
May  6 09:52:27 simech-ice kernel:   next_to_use          <3e4>
May  6 09:52:27 simech-ice kernel:   next_to_clean        <2>
May  6 09:52:27 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:27 simech-ice kernel:   time_stamp           <11e0400bb>
May  6 09:52:27 simech-ice kernel:   next_to_watch        <3>
May  6 09:52:27 simech-ice kernel:   jiffies              <11e040d75>
May  6 09:52:27 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:34 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May  6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May  6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May  6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:34 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:42 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:42 simech-ice kernel:   TDH                  <189>
May  6 09:52:42 simech-ice kernel:   TDT                  <159>
May  6 09:52:42 simech-ice kernel:   next_to_use          <159>
May  6 09:52:42 simech-ice kernel:   next_to_clean        <184>
May  6 09:52:42 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:42 simech-ice kernel:   time_stamp           <11e042edb>
May  6 09:52:42 simech-ice kernel:   next_to_watch        <185>
May  6 09:52:42 simech-ice kernel:   jiffies              <11e0449ec>
May  6 09:52:42 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:45 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May  6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:48 simech-ice kernel:   TDH                  <d>
May  6 09:52:48 simech-ice kernel:   TDT                  <3e6>
May  6 09:52:48 simech-ice kernel:   next_to_use          <3e6>
May  6 09:52:48 simech-ice kernel:   next_to_clean        <9>
May  6 09:52:48 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:48 simech-ice kernel:   time_stamp           <11e042de5>
May  6 09:52:48 simech-ice kernel:   next_to_watch        <a>
May  6 09:52:48 simech-ice kernel:   jiffies              <11e045e0b>
May  6 09:52:48 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:52:48 simech-ice kernel:   TDH                  <78>
May  6 09:52:48 simech-ice kernel:   TDT                  <52>
May  6 09:52:48 simech-ice kernel:   next_to_use          <52>
May  6 09:52:48 simech-ice kernel:   next_to_clean        <73>
May  6 09:52:48 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:52:48 simech-ice kernel:   time_stamp           <11e042e11>
May  6 09:52:48 simech-ice kernel:   next_to_watch        <74>
May  6 09:52:48 simech-ice kernel:   jiffies              <11e045e7c>
May  6 09:52:48 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May  6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May  6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:59 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May  6 09:52:59 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May  6 09:52:59 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:52:59 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May  6 09:53:07 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:53:07 simech-ice kernel:   TDH                  <28>
May  6 09:53:07 simech-ice kernel:   TDT                  <3>
May  6 09:53:07 simech-ice kernel:   next_to_use          <3>
May  6 09:53:07 simech-ice kernel:   next_to_clean        <23>
May  6 09:53:07 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:53:07 simech-ice kernel:   time_stamp           <11e049a4d>
May  6 09:53:07 simech-ice kernel:   next_to_watch        <24>
May  6 09:53:07 simech-ice kernel:   jiffies              <11e04a866>
May  6 09:53:07 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:53:07 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:53:07 simech-ice kernel:   TDH                  <2ad>
May  6 09:53:07 simech-ice kernel:   TDT                  <28c>
May  6 09:53:07 simech-ice kernel:   next_to_use          <28c>
May  6 09:53:07 simech-ice kernel:   next_to_clean        <2a7>
May  6 09:53:07 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:53:07 simech-ice kernel:   time_stamp           <11e04979e>
May  6 09:53:07 simech-ice kernel:   next_to_watch        <2a8>
May  6 09:53:07 simech-ice kernel:   jiffies              <11e04a880>
May  6 09:53:07 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:53:10 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:53:10 simech-ice kernel:   TDH                  <129>
May  6 09:53:10 simech-ice kernel:   TDT                  <103>
May  6 09:53:10 simech-ice kernel:   next_to_use          <103>
May  6 09:53:10 simech-ice kernel:   next_to_clean        <125>
May  6 09:53:10 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:53:10 simech-ice kernel:   time_stamp           <11e04b236>
May  6 09:53:10 simech-ice kernel:   next_to_watch        <126>
May  6 09:53:10 simech-ice kernel:   jiffies              <11e04b61f>
May  6 09:53:10 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:53:14 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:53:14 simech-ice kernel:   TDH                  <18e>
May  6 09:53:14 simech-ice kernel:   TDT                  <165>
May  6 09:53:14 simech-ice kernel:   next_to_use          <165>
May  6 09:53:14 simech-ice kernel:   next_to_clean        <189>
May  6 09:53:14 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:53:14 simech-ice kernel:   time_stamp           <11e04b24c>
May  6 09:53:14 simech-ice kernel:   next_to_watch        <18a>
May  6 09:53:14 simech-ice kernel:   jiffies              <11e04c4e4>
May  6 09:53:14 simech-ice kernel:   next_to_watch.status <17a8209>
May  6 09:53:14 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May  6 09:53:14 simech-ice kernel:   TDH                  <3b>
May  6 09:53:14 simech-ice kernel:   TDT                  <16>
May  6 09:53:14 simech-ice kernel:   next_to_use          <16>
May  6 09:53:14 simech-ice kernel:   next_to_clean        <37>
May  6 09:53:14 simech-ice kernel: tx_buffer_info[next_to_clean]
May  6 09:53:14 simech-ice kernel:   time_stamp           <11e04b1d7>
May  6 09:53:14 simech-ice kernel:   next_to_watch        <38>
May  6 09:53:14 simech-ice kernel:   jiffies              <11e04c6e3>
May  6 09:53:14 simech-ice kernel:   next_to_watch.status <17a8209>

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

do you have ipv6 enabled? I've seen this behavior that if a port is flooded before the events/X thread finishes, lots of packets get dropped and the events/X thread takes a long time to complete. Not sure if it is related.

hm, snipped above to demonstrate my point. These appear to be false hangs. TDH is still moving (indicating the hardware is still processing packets.) Do you have flow control enabled? Can you try with fewer descriptors? It is truly unlikely you need more than 512, usually.

The driver (incorrectly, will patch soon) defaults to flow control enabled. I suggest you disable it with ethtool -A

You might be able to just comment out the detect_tx_hung variable being set, see if the problem goes away (false hang for sure then)

Jesse

It is enabled, though I wasn't particularly using it (on purpose).

Ok, I also noticed that softirqd was at around 100% CPU (2 of them in fact, on this 2 x 4-core system. But, the NICs were not obviously transmitting many packets (as determined by looking at the tx/rx packet counters).

In subsequent tests, I see softirqd CPU usage go quite high when adding mac-vlans, before I ever start traffic. But, other applications (ntp, etc) do seem to listen for new devices and open sockets per interface and probably attempt to send some frames.

Also, this is a 64-bit kernel, with 8GB RAM, in case that matters.

Finally, I hit this a bit later. I have no idea of the root cause here...it seems mac-vlans are implicated, but it could be something else. It is tainted by my module, but this module was supposedly not really doing anything. I will also run some more tests w/out it loaded.

BUG: soft lockup - CPU#7 stuck for 61s! [ksoftirqd/7:25]
CPU 7:
Modules linked in: arc4 michael_mic wanlink(P) e1000e e1000 8021q redirdev macvlan pktgen rfcomm l2cap bluetooth autofs4 nfs lockd nfs_acl sunrpc ipv6 loop dm_multipath i5000_edac edac_core iTCO_wdt ixgbe i2c_i801 i2c_core pcspkr button iTCO_vendor_support sg sr_mod cdrom floppy dm_snapshot dm_zero dm_mirror dm_mod ata_generic pata_acpi ata_piix libata sd_mod scsi_mod ext3 jbd mbcache uhci_hcd ohci_hcd ssb ehci_hcd [last unloaded: x_tables]
Pid: 25, comm: ksoftirqd/7 Tainted: P         2.6.25 #1
RIP: 0010:[<ffffffff8120163d>]  [<ffffffff8120163d>] skb_clone+0x5a/0x5e
RSP: 0018:ffff81022f207d98  EFLAGS: 00000202
RAX: ffff81012173f300 RBX: ffff81022f207da8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff810131b0f168 RDI: ffff81012173f368
RBP: ffff81022f207d10 R08: ffff81012173f300 R09: ffff810131b0f100
R10: 0000000000000040 R11: 0000000000000000 R12: ffffffff8100cb56
R13: ffff81022f207d10 R14: ffff810131b0f100 R15: ffff81022d5b6000
FS:  0000000000000000(0000) GS:ffff81022f0b8c80(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007faf08544a90 CR3: 0000000000201000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Call Trace:
 <IRQ>  [<ffffffff8120163d>] ? skb_clone+0x5a/0x5e
 [<ffffffff8827f54a>] ? :macvlan:macvlan_handle_frame+0x102/0x222
 [<ffffffff81094817>] ? add_partial+0x49/0x51
 [<ffffffff81206db1>] ? netif_receive_skb+0x346/0x4f3
 [<ffffffff88123ed2>] ? :ixgbe:ixgbe_clean_rx_irq+0x467/0x666
 [<ffffffff881266b7>] ? :ixgbe:ixgbe_clean_rxonly+0x4a/0xa4
 [<ffffffff8120931e>] ? net_rx_action+0xb0/0x1c6
 [<ffffffff8103a030>] ? __do_softirq+0x4a/0xa5
 [<ffffffff8103a3b8>] ? ksoftirqd+0x0/0x11e
 [<ffffffff8100d0ac>] ? call_softirq+0x1c/0x28
 <EOI>  [<ffffffff8100e978>] ? do_softirq+0x34/0x72
 [<ffffffff8103a41c>] ? ksoftirqd+0x64/0x11e
 [<ffffffff81048088>] ? kthread+0x49/0x79
 [<ffffffff8100cd38>] ? child_rip+0xa/0x12
 [<ffffffff8104803f>] ? kthread+0x0/0x79
 [<ffffffff8100cd2e>] ? child_rip+0x0/0x12

unregister_netdevice: waiting for eth3#352 to become free. Usage count = 3
unregister_netdevice: waiting for eth3#352 to become free. Usage count = 3
unregister_netdevice: waiting for eth3#352 to become free. Usage count = 3
unregister_netdevice: waiting for eth3#352 to become free. Usage count = 3
unregister_netdevice: waiting for eth3#352 to become free. Usage count = 3

I'll try disabling the flow-control, and if that doesn't help, will compile out ipv6 and try that too.

Thanks,
Ben

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com