Skip to contentSkip to conversation

New: system hung with htb QoS

Bugme-new Andrew Morton 2 participants 4 messages Jan 24, 2008 – Feb 1, 2008 1 message not in the archive

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author
#1 lore raw
Quoted 131 lines · On Thu, 24 Jan 2008 03:03:11 -0800 (PST) [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=9808

           Summary: system hung with htb QoS
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.23.9
          Platform: All
        OS/Version: Linux
              Tree: Fedora
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Netfilter/Iptables
        AssignedTo: [email protected]
        ReportedBy: [email protected]

Hi,

I've setup QoS on my ftp server to limit outgoing traffic. Apparently the server
stops responding (no output no keyboard) in an unpredictable manner. Sometimes it
takes an hour, sometimes up to 4 days for the system to hung.

I have attached my QoS startup script, dmesg output, lspci -vvv, iptables that interact with QoS.

I'm also receiving this quite often:

Jan 15 12:23:17 ftp kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit
Hang
Jan 15 12:23:17 ftp kernel:   Tx Queue             <0>
Jan 15 12:23:17 ftp kernel:   TDH                  <2a>
Jan 15 12:23:17 ftp kernel:   TDT                  <17>
Jan 15 12:23:17 ftp kernel:   next_to_use          <17>
Jan 15 12:23:17 ftp kernel:   next_to_clean        <2a>
Jan 15 12:23:17 ftp kernel: buffer_info[next_to_clean]
Jan 15 12:23:17 ftp kernel:   time_stamp           <5798144>
Jan 15 12:23:17 ftp kernel:   next_to_watch        <2d>
Jan 15 12:23:17 ftp kernel:   jiffies              <57988ef>
Jan 15 12:23:17 ftp kernel:   next_to_watch.status <0>
Jan 15 12:23:19 ftp kernel: e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit

Hang

Today for the first time (after applying options to e1000 driver in modprobe.conf) I got a kernel panic:

BUG: unable to handle kernel paging request at virtual address a0379120
EIP: 0060: [<c05db2dc>] Not Tainted VLI
EIP is at ip_rcv+0x286/0x4ba
Kernel panic - not syncing: Fatal exception in interrupt

This is what I wrote on paper cause there wasn't logged anywhere. Usually it hungs without a kernel panic.

System in Fedoca Core 8 up2date
2.6.23.9-85.fc8PAE
2x Intel(R) Xeon(TM) CPU 3.20GHz
4G RAM

Without the QoS loaded system never hungs. It must be related to this. However the e1000 error I'm receiving must have to do with the e1000 driver. I've seen this bug in the past that's why I tried to apply the options in modprobe.conf

any help will be appreciated
thanx in advance

Giannis

QoS startup script:

# default WAN limit
LIMIT="80mbit"
LOW_LIMIT="50mbit"
start() {
        echo -n "Starting QoS: (WAN limit set to ${LIMIT})"

tc qdisc del dev eth0 root 2> /dev/null > /dev/null
tc qdisc del dev eth0 ingress 2> /dev/null > /dev/null ADD_CLASS="tc class add dev eth0 "
###### uplink

# install root HTB, point default traffic to 1:25
tc qdisc add dev eth0 root handle 1: htb  default 25

tc class add dev eth0 parent 1: classid 1:1 htb rate 1000mbit

# class for outgoing SYN packets + Minimize-Delay TOS
${ADD_CLASS} parent 1:1 classid 1:11 htb rate 2mbit ceil 5mbit prio 1
# class for internal LAN traffic
${ADD_CLASS} parent 1:1 classid 1:12 htb rate 500mbit ceil 800mbit prio 2
# class for WAN traffic
${ADD_CLASS} parent 1:1 classid 1:2 htb rate ${LIMIT} ceil ${LIMIT} prio 3
# class for WAN http traffic
${ADD_CLASS} parent 1:2 classid 1:24 htb rate 30mbit ceil ${LIMIT} prio 4
# default class, rest WAN traffic
${ADD_CLASS} parent 1:2 classid 1:25 htb rate 20mbit ceil ${LIMIT} prio 5

tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 1:11 tc filter add dev eth0 protocol ip parent 1:0 prio 2 handle 2 fw flowid 1:12 tc filter add dev eth0 protocol ip parent 1:0 prio 4 u32 \

        match ip sport 80 0xffff flowid 1:24

tc qdisc add dev eth0 parent 1:11 handle 11: sfq perturb 10 tc qdisc add dev eth0 parent 1:12 handle 12: sfq perturb 10 tc qdisc add dev eth0 parent 1:24 handle 24: sfq perturb 10 tc qdisc add dev eth0 parent 1:25 handle 25: sfq perturb 10

echo
}

stop() {
echo -n "Stopping QoS: "
tc qdisc del dev eth0 root 2> /dev/null > /dev/null
tc qdisc del dev eth0 ingress 2> /dev/null > /dev/null echo
}

-------------------
QoS startup script: http://www.edu.physics.uoc.gr/~bilias/ftp/QoS
dmesg: http://www.edu.physics.uoc.gr/~bilias/ftp/dmesg
lspci -vvv: http://www.edu.physics.uoc.gr/~bilias/ftp/lspci
iptables for QoS: http://www.edu.physics.uoc.gr/~bilias/ftp/iptables

modprobe.conf options for e1000:
options e1000 XsumRX=0 Speed=1000 Duplex=2 InterruptThrottleRate=0 FlowControl=3 RxDescriptors=4096 TxDescriptors=4096 RxIntDelay=0 TxIntDelay=0

Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
#2 lore raw
Quoted 7 lines · Andrew Morton
Quoted 7 lines

I'm also receiving this quite often:

Jan 15 12:23:17 ftp kernel: e1000: eth0: e1000_clean_tx_irq:
Detected Tx Unit Hang Jan 15 12:23:17 ftp kernel:   Tx Queue        
<0> 
Jan 15 12:23:17 ftp kernel:   TDH                  <2a>
Jan 15 12:23:17 ftp kernel:   TDT                  <17>
Jan 15 12:23:17 ftp kernel:   next_to_use          <17>
Quoted 8 lines
Quoted 8 lines
Jan 15 12:23:17 ftp kernel:   next_to_clean        <2a>
Jan 15 12:23:17 ftp kernel: buffer_info[next_to_clean]
Jan 15 12:23:17 ftp kernel:   time_stamp           <5798144>
Jan 15 12:23:17 ftp kernel:   next_to_watch        <2d>
Jan 15 12:23:17 ftp kernel:   jiffies              <57988ef>
Jan 15 12:23:17 ftp kernel:   next_to_watch.status <0>
Jan 15 12:23:19 ftp kernel: e1000: eth0: e1000_clean_tx_irq:

Detected Tx Unit Hang

Looks like a real hardware hang.

Would you be willing to try the 7.6.15 driver at e1000.sourceforge.net, it has many more fixes for e1000 than what is available in the in-kernel driver. I just posted a patch in the "Tracker/Patches" area that patches 7.6.15 to support the e1000_dump code to dump rings when the tx hang occurs which will help us figure out a) what software did to the ring, b) if something is messed up in the ring which we know will hang the hardware

Quoted 11 lines
Quoted 11 lines

Today for the first time (after applying options to e1000 driver in modprobe.conf) I got a kernel panic:

BUG: unable to handle kernel paging request at virtual address
a0379120 
EIP: 0060: [<c05db2dc>] Not Tainted VLI
EIP is at ip_rcv+0x286/0x4ba
Kernel panic - not syncing: Fatal exception in interrupt

This is what I wrote on paper cause there wasn't logged anywhere. Usually it hungs without a kernel panic.

Everyone involved will need more information about the panic to make progress on the panic.

Quoted 11 lines
Quoted 10 lines

System in Fedoca Core 8 up2date
2.6.23.9-85.fc8PAE
2x Intel(R) Xeon(TM) CPU 3.20GHz
4G RAM

Without the QoS loaded system never hungs. It must be related to this. However the e1000 error I'm receiving must have to do with the e1000 driver. I've seen this bug in the past that's why I tried to apply the options in modprobe.conf

Quoted 4 lines
Quoted 4 lines

modprobe.conf options for e1000:
options e1000 XsumRX=0 Speed=1000 Duplex=2 InterruptThrottleRate=0 FlowControl=3 RxDescriptors=4096 TxDescriptors=4096 RxIntDelay=0 TxIntDelay=0

Please don't use any of these options unless you must, they seem to have come from some debian forum that someone just posted a SWAG at changing parameters that fixed him for some unknown reason.

Get back to us with the debug output, the e1000 issue can be covered on [email protected]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

author
#3 lore raw
Quoted 8 lines · Jesse

Would you be willing to try the 7.6.15 driver at e1000.sourceforge.net, it has many more fixes for e1000 than what is available in the in-kernel driver. I just posted a patch in the "Tracker/Patches" area that patches 7.6.15 to support the e1000_dump code to dump rings when the tx hang occurs which will help us figure out a) what software did to the ring, b) if something is messed up in the ring which we know will hang the hardware

General hand-waving: if you have a debug patch which helps diagnose rare problems when a tester hits them, please consider just putting them straight into the mainline tree.

We can always take them out later when everything is resolved, and we can help ourselves a lot this way.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/

#4 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton
Quoted 4 lines

On Thu, 24 Jan 2008 03:03:11 -0800 (PST)
[email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=9808

Quoted 14 lines
Quoted 13 lines
           Summary: system hung with htb QoS
           Product: Networking
           Version: 2.5
     KernelVersion: 2.6.23.9
          Platform: All
        OS/Version: Linux
              Tree: Fedora
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Netfilter/Iptables
        AssignedTo:
        [email protected]
Quoted 1 line

ReportedBy: [email protected]

This user was able to eliminate the tx hang problem with htb QoS by disabling TSO. We also have some reports of xen hangs when TSO is enabled as well. Not sure if they are related.

At this point this user has a workaround. We will continue to try to reproduce here to see if we can root cause the hardware issue, but we haven't had any luck so far.