Skip to contentSkip to conversation

ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabled

Intel-wired-lan Samudrala 2 participants 4 messages May 3, 2016

lore
Share & tools
lore

Samudrala

#0 lore raw

With the latest davem's next-next or Jeff's dev-queue branch (4.6.0-rc5+), I am seeing Tx Unit Hangs when using offloaded macvlans.

Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 name mvlan_1 type macvlan ip link set mvlan_1 up

Here is the kernel log when this happens.

May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1: link becomes ready
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5
May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang
                                 Tx Queue             <0>
                                 TDH, TDT             <0>, <4>
                                 next_to_use          <4>
                                 next_to_clean        <0>
                               tx_buffer_info[next_to_clean]
                                 time_stamp           <10005560a>
                                 jiffies              <100055df4>
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5
May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang
                                 Tx Queue             <0>
                                 TDH, TDT             <0>, <2>
                                 next_to_use          <2>
                                 next_to_clean        <0>
                               tx_buffer_info[next_to_clean]
                                 time_stamp           <1000569a0>
                                 jiffies              <1000571b0>
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue packet!
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160503/6f45bdb4/attachment-0001.html>

#1 lore raw
Quoted 69 lines · Sridhar

With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.

Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 name mvlan_1 type macvlan ip link set mvlan_1 up

Here is the kernel log when this happens.

May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1:
link becomes ready
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled:
Rx Queue count = 2, Tx Queue count = 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
enable Tx Queue 0
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
enable Tx Queue 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC
device on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5
May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx

Unit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <4> ????????????????????????????????next_to_use??????????<4> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<10005560a> ????????????????????????????????jiffies??????????????<100055df4>

May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1
detected on queue 0, resetting adapter
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating
reset due to tx timeout
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
enable Tx Queue 0
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
enable Tx Queue 2
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5
May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx

Unit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <2> ????????????????????????????????next_to_use??????????<2> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<1000569a0> ????????????????????????????????jiffies??????????????<1000571b0>

May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2
detected on queue 0, resetting adapter
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating
reset due to tx timeout
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue
packet!
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
enable Tx Queue 0
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
Quoted 1 line

enable Tx Queue 2

I just updated my tree with Dave's latest just now, so if you want to
re-pull and see if it still exists, please.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160503/cc1cbdc6/attachment.asc>

#2 lore raw

It still does it even with the latest tree pulled, however if I create the macvlan, bring it up, and THEN enable l2-fwd-offload, I don't get the tx hang. If I try to create another macvlan with l2-fwd-offload enabled, I get the tx hang.

Quoted 76 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jeff Kirsher
Sent: Tuesday, May 03, 2016 1:35 PM
To: Samudrala, Sridhar <[email protected]>; intel-wired- lan at lists.osuosl.org
Subject: Re: [Intel-wired-lan] ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabled

Quoted 60 lines · Sridhar

With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.

Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 name mvlan_1 type macvlan ip link set mvlan_1 up

Here is the kernel log when this happens.

May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1:
link becomes ready
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled:

Rx Queue count = 2, Tx Queue count = 2 May 03 10:45:30 sri02 kernel:
ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5

May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx

Unit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <4> ????????????????????????????????next_to_use??????????<4> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<10005560a> ????????????????????????????????jiffies??????????????<100055df4> May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1:
detected SFP+:
5

May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx

Unit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <2> ????????????????????????????????next_to_use??????????<2> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<1000569a0> ????????????????????????????????jiffies??????????????<1000571b0> May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue packet!

Quoted 3 lines
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not

enable Tx Queue 0 May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0
p4p1: Could not enable Tx Queue 2

I just updated my tree with Dave's latest just now, so if you want to re-pull and see if it still exists, please.

#3 lore raw
Quoted 1 line · AndrewX

It still does it even with the latest tree pulled, however if I create the macvlan, bring it up, and THEN enable l2-fwd-offload, I don't get the tx hang. If I try to create another macvlan with l2-fwd-offload enabled, I get the tx hang.

Yes. I too see this issue even with the latest tree. I remember not seeing this issue with 4.6.0-rc1 based tree.

Quoted 75 lines
Quoted 75 lines

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jeff Kirsher
Sent: Tuesday, May 03, 2016 1:35 PM
To: Samudrala, Sridhar <[email protected]>; intel-wired- lan at lists.osuosl.org
Subject: Re: [Intel-wired-lan] ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabled

Quoted 55 lines · Sridhar

With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.

Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 name mvlan_1 type macvlan ip link set mvlan_1 up

Here is the kernel log when this happens.

May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1:
link becomes ready
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled:

Rx Queue count = 2, Tx Queue count = 2 May 03 10:45:30 sri02 kernel:
ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5

May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx
Unit Hang
                                 Tx Queue             <0>
                                 TDH, TDT             <0>, <4>
                                 next_to_use          <4>
                                 next_to_clean        <0>

tx_buffer_info[next_to_clean]
time_stamp <10005560a>
jiffies <100055df4> May
03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1:
detected SFP+:
5

May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up
10 Gbps, Flow Control: RX/TX
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx
Unit Hang
                                 Tx Queue             <0>
                                 TDH, TDT             <0>, <2>
                                 next_to_use          <2>
                                 next_to_clean        <0>
                               tx_buffer_info[next_to_clean]
                                 time_stamp           <1000569a0>
                                 jiffies              <1000571b0> May
Quoted 8 lines

03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue packet!

May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not

enable Tx Queue 0 May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0
p4p1: Could not enable Tx Queue 2

I just updated my tree with Dave's latest just now, so if you want to re-pull and see if it still exists, please.