ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabled
Samudrala
With the latest davem's next-next or Jeff's dev-queue branch (4.6.0-rc5+), I am seeing Tx Unit Hangs when using offloaded macvlans.
Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 name mvlan_1 type macvlan ip link set mvlan_1 up
Here is the kernel log when this happens.
May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1: link becomes ready
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1
May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5
May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang
Tx Queue <0>
TDH, TDT <0>, <4>
next_to_use <4>
next_to_clean <0>
tx_buffer_info[next_to_clean]
time_stamp <10005560a>
jiffies <100055df4>
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout
May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2
May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5
May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang
Tx Queue <0>
TDH, TDT <0>, <2>
next_to_use <2>
next_to_clean <0>
tx_buffer_info[next_to_clean]
time_stamp <1000569a0>
jiffies <1000571b0>
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout
May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter
May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue packet!
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160503/6f45bdb4/attachment-0001.html>
Quoted 69 lines · Sridhar
With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 namemvlan_1type macvlan ip link setmvlan_1upHere is the kernel log when this happens.
May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1: link becomes ready May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5 May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected TxUnit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <4> ????????????????????????????????next_to_use??????????<4> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<10005560a> ????????????????????????????????jiffies??????????????<100055df4>May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+: 5 May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected TxUnit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <2> ????????????????????????????????next_to_use??????????<2> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<1000569a0> ????????????????????????????????jiffies??????????????<1000571b0>May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:40 sri02 kernel: Virtual device mvlan_1 asks to queue packet! May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not
Quoted 1 line
enable Tx Queue 2
I just updated my tree with Dave's latest just now, so if you want to
re-pull and see if it still exists, please.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20160503/cc1cbdc6/attachment.asc>
Bowers
It still does it even with the latest tree pulled, however if I create the macvlan, bring it up, and THEN enable l2-fwd-offload, I don't get the tx hang. If I try to create another macvlan with l2-fwd-offload enabled, I get the tx hang.
Quoted 76 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jeff Kirsher
Sent: Tuesday, May 03, 2016 1:35 PM
To: Samudrala, Sridhar <[email protected]>; intel-wired- lan at lists.osuosl.org
Subject: Re: [Intel-wired-lan] ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabledQuoted 60 lines · Sridhar
With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 namemvlan_1type macvlan ip link setmvlan_1upHere is the kernel log when this happens.
May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1: link becomes ready May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled:Rx Queue count = 2, Tx Queue count = 2 May 03 10:45:30 sri02 kernel:
ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected TxUnit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <4> ????????????????????????????????next_to_use??????????<4> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<10005560a> ????????????????????????????????jiffies??????????????<100055df4> May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1:
detected SFP+:
5May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected TxUnit Hang
????????????????????????????????Tx Queue?????????????<0> ????????????????????????????????TDH, TDT?????????????<0>, <2> ????????????????????????????????next_to_use??????????<2> ????????????????????????????????next_to_clean????????<0> ??????????????????????????????tx_buffer_info[next_to_clean] ????????????????????????????????time_stamp???????????<1000569a0> ????????????????????????????????jiffies??????????????<1000571b0> May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:40 sri02 kernel: Virtual devicemvlan_1asks to queue packet!Quoted 3 lines
May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could notenable Tx Queue 0 May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0
p4p1: Could not enable Tx Queue 2I just updated my tree with Dave's latest just now, so if you want to re-pull and see if it still exists, please.
Samudrala
Quoted 1 line · AndrewX
It still does it even with the latest tree pulled, however if I create the macvlan, bring it up, and THEN enable l2-fwd-offload, I don't get the tx hang. If I try to create another macvlan with l2-fwd-offload enabled, I get the tx hang.
Yes. I too see this issue even with the latest tree. I remember not seeing this issue with 4.6.0-rc1 based tree.
Quoted 75 lines
Quoted 75 lines
-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of Jeff Kirsher
Sent: Tuesday, May 03, 2016 1:35 PM
To: Samudrala, Sridhar <[email protected]>; intel-wired- lan at lists.osuosl.org
Subject: Re: [Intel-wired-lan] ixgbe Tx Unit Hangs seen when l2-fwd-offload is enabledQuoted 55 lines · Sridhar
With the latest davem's next-next or Jeff's dev-queue branch (4.6.0- rc5+), I am seeing Tx Unit Hangs when
using offloaded macvlans.Here is a simple sequence of commands that can reproduce this issue.
ethtool -K p4p1 l2-fwd-offload on
ip link add link p4p1 namemvlan_1type macvlan ip link setmvlan_1upHere is the kernel log when this happens.
May 03 10:45:01 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:01 sri02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): p4p1: link becomes ready May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: removed PHC on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: Multiqueue Enabled:Rx Queue count = 2, Tx Queue count = 2 May 03 10:45:30 sri02 kernel:
ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0: registered PHC device on p4p1 May 03 10:45:30 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: detected SFP+:
5May 03 10:45:31 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang Tx Queue <0> TDH, TDT <0>, <4> next_to_use <4> next_to_clean <0>tx_buffer_info[next_to_clean]
time_stamp<10005560a>
jiffies <100055df4> May
03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 1 detected on queue 0, resetting adapter May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:33 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 0 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could not enable Tx Queue 2 May 03 10:45:35 sri02 kernel: ixgbe 0000:1f:00.0 p4p1:
detected SFP+:
5May 03 10:45:36 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: NIC Link is Up 10 Gbps, Flow Control: RX/TX May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Detected Tx Unit Hang Tx Queue <0> TDH, TDT <0>, <2> next_to_use <2> next_to_clean <0> tx_buffer_info[next_to_clean] time_stamp <1000569a0> jiffies <1000571b0> MayQuoted 8 lines
03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: tx hang 2 detected on queue 0, resetting adapter May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: initiating reset due to tx timeout May 03 10:45:38 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Reset adapter May 03 10:45:40 sri02 kernel: Virtual device
mvlan_1asks to queue packet!May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0 p4p1: Could notenable Tx Queue 0 May 03 10:45:40 sri02 kernel: ixgbe 0000:1f:00.0
p4p1: Could not enable Tx Queue 2I just updated my tree with Dave's latest just now, so if you want to re-pull and see if it still exists, please.