Skip to contentSkip to conversation

NetDev! Please help!

DISCUSSION Badalian Vyacheslav 3 participants 18 messages Sep 20, 2008 – Sep 23, 2008 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello all.

We buy 10 Intel servers and paste it to shape traffic. After 5-15 hours all PC is was freeze! Kernel not see TCO watchdog at this platform and can't reboot it!. Soft Watchdog not reboot pc in this situation. =(

At screen we see messages like this (when it freeze and i was near monitor):

http://www.kerneloops.org/guilty.php?guilty=dev_watchdog&version=2.6.26-release&start=1736704&end=1769471&class=warn

Also by netconsole we was get this one time:

[ 1352.245851] netconsole: network logging started
[ 1458.400133] 802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
[ 1458.400133] All bugs added by David S. Miller <[email protected]>
[ 4956.420298] CPU 1: Machine Check Exception: 0000000000000005
[ 4956.420298] e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
[ 4956.420300]   Tx Queue             <0>
[ 4956.420300]   TDH                  <81>
[ 4956.420301]   TDT                  <81>
[ 4956.420302]   next_to_use          <81>
[ 4956.420302]   next_to_clean        <d6>
[ 4956.420303] buffer_info[next_to_clean]
[ 4956.420303]   time_stamp           <15498d>
[ 4956.420304]   next_to_watch        <d6>
[ 4956.420304]   jiffies              <15511c>
[ 4956.420305]   next_to_watch.status <1>
[ 4956.420537] eth1: Detected Tx Unit Hang:
[ 4956.420538]   TDH                  <b0>
[ 4956.420538]   TDT                  <b0>
[ 4956.420539]   next_to_use          <b0>
[ 4956.420539]   next_to_clean        <5>
[ 4956.420540] buffer_info[next_to_clean]:
[ 4956.420540]   time_stamp           <15498e>
[ 4956.420541]   next_to_watch        <5>
[ 4956.420542]   jiffies              <15511c>
[ 4956.420542]   next_to_watch.status <1>
[ 4956.423064] CPU 1: Bank 0: 3200004000000800
[ 4956.423190] CPU 1: Bank 5: 3200220024080400
[ 4956.423315] Kernel panic - not syncing: CPU context corrupt
[ 4956.423933] Rebooting in 3 seconds..[  531.843998] CPU 2: Machine
Check Exception: 0000000000000005
[  531.843998] CPU 0: Machine Check Exception: 0000000000000004
[  531.844000] CPU 0: Bank 0: 3200004000000800
[  531.844001] CPU 0: Bank 5: 3200121014040400
[  531.844002] Kernel panic - not syncing: CPU context corrupt
[  531.844916] Rebooting in 3 seconds..

This out of lspci:

00:00.0 Host bridge: Intel Corporation Server DRAM Controller 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation USB UHCI Controller #5 (rev 02) 00:1a.2 USB Controller: Intel Corporation USB UHCI Controller #6 (rev 02) 00:1a.7 USB Controller: Intel Corporation USB2 EHCI Controller #2 (rev 02) 00:1c.0 PCI bridge: Intel Corporation PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel Corporation PCI Express Port 5 (rev 02) 00:1d.0 USB Controller: Intel Corporation USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation SMBus Controller (rev 02) 02:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)
03:02.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

lspci -tv

-[0000:00]-+-00.0  Intel Corporation Server DRAM Controller
           +-19.0  Intel Corporation 82566DM-2 Gigabit Network Connection
           +-1a.0  Intel Corporation USB UHCI Controller #4
           +-1a.1  Intel Corporation USB UHCI Controller #5
           +-1a.2  Intel Corporation USB UHCI Controller #6
           +-1a.7  Intel Corporation USB2 EHCI Controller #2
           +-1c.0-[0000:01]--
           +-1c.4-[0000:02]----00.0  Matrox Graphics, Inc. MGA G200e
[Pilot] ServerEngines (SEP1)
           +-1d.0  Intel Corporation USB UHCI Controller #1
           +-1d.1  Intel Corporation USB UHCI Controller #2
           +-1d.2  Intel Corporation USB UHCI Controller #3
           +-1d.7  Intel Corporation USB2 EHCI Controller #1
           +-1e.0-[0000:03]----02.0  Intel Corporation 82541GI Gigabit
Ethernet Controller
           +-1f.0  Intel Corporation LPC Interface Controller
           +-1f.2  Intel Corporation 6 port SATA AHCI Controller
           \-1f.3  Intel Corporation SMBus Controller

All tests in 2.6.26.2 kernel... now i try compile 2.6.26.5 kernel, but if KernelOops is right - its not will help.

I hope NetDev help!
If any information or test is needed - please write!

Thanks for anyone!

Badalian Vyacheslav.

#1 lore raw

New crash.... netconsole log:

[  116.333349] ------------[ cut here ]------------
[  116.333516] WARNING: at net/sched/sch_generic.c:222
dev_watchdog+0xf1/0x110()
[  116.333690] Modules linked in: netconsole i2c_i801 i2c_core e1000e e1000
[  116.334199] Pid: 0, comm: swapper Not tainted 2.6.26-gentoo-r1-fw #2
[  116.334371]  [<c012506f>] warn_on_slowpath+0x5f/0x90
[  116.334597]  [<c011dd1a>] enqueue_task_fair+0x1a/0x30
[  116.334823]  [<c011b962>] enqueue_task+0x12/0x30
[  116.335046]  [<c011b9d3>] activate_task+0x23/0x40
[  116.335268]  [<c011e01a>] try_to_wake_up+0x6a/0x110
[  116.335491]  [<c0137c7b>] autoremove_wake_function+0x1b/0x50
[  116.335718]  [<c011be6b>] __wake_up_common+0x4b/0x80
[  116.335941]  [<c011cfde>] __wake_up+0x3e/0x60
[  116.336161]  [<c0134a2b>] insert_work+0x4b/0x70
[  116.336384]  [<c0134dd7>] __queue_work+0x27/0x40
[  116.336610]  [<c02d0651>] dev_watchdog+0xf1/0x110
[  116.337333]  [<c012e055>] run_timer_softirq+0x115/0x170
[  116.337557]  [<c0122c71>] scheduler_tick+0xa1/0xd0
[  116.337780]  [<c012a062>] __do_softirq+0x82/0x100
[  116.338002]  [<c012a117>] do_softirq+0x37/0x40
[  116.338222]  [<c0114027>] smp_apic_timer_interrupt+0x57/0x90
[  116.338448]  [<c0105660>] apic_timer_interrupt+0x28/0x30
[  116.338672]  [<c010a5e2>] mwait_idle+0x32/0x40
[  116.338894]  [<c010a5b0>] mwait_idle+0x0/0x40
[  116.339115]  [<c01036e8>] cpu_idle+0x48/0xc0
[  116.339336]  =======================
[  116.339499] ---[ end trace e25a40b7dc59df07 ]---
[  117.655918] CPU 1: Machine Check Exception: 0000000000000005
[  117.656103] CPU 1: Bank 0: 3200004000000800
[  117.656604] CPU 1: Bank 5: 3200220024080400
[  117.656604] Kernel panic - not syncing: CPU context corrupt
[  117.656624] Rebooting in 3 seconds..

Thanks

#2 lore raw
In reply to Badalian Vyacheslav
Quoted 8 lines · Badalian Vyacheslav

Hello all.

We buy 10 Intel servers and paste it to shape traffic. After 5-15 hours all PC is was freeze! Kernel not see TCO watchdog at this platform and can't reboot it!. Soft Watchdog not reboot pc in this situation. =(

At screen we see messages like this (when it freeze and i was near monitor):

Maybe try nmi_watchdog=1 ?
Also try http://www.nongnu.org/dmidecode/ - to check if it has IPMI. Mine for example:
....
Handle 0x003F, DMI type 38, 16 bytes
IPMI Device Information
Interface Type: KCS (Keyboard Control Style) Specification Version: 2.0
I2C Slave Address: 0x10
NV Storage Device Address: 0
Base Address: 0x0000000000000CA2 (I/O)

Also important to change it to newer 2.6.26.5 , because for example 2.6.26.4 have fix:

http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.26.4
....

commit 685f605a498b73759cbcbc816089e804710fcc48

Author: David S. Miller <[email protected]>
Date:   Wed Aug 27 22:35:56 2008 -0700

pkt_sched: Fix return value corruption in HTB and TBF.

[ Upstream commit 69747650c814a8a79fef412c7416adf823293a3e ]

Based upon a bug report by Josip Rodin.

Packet schedulers should only return NET_XMIT_DROP iff the packet really was dropped. If the packet does reach the device after we return NET_XMIT_DROP then TCP can crash because it depends upon the enqueue path return values being accurate.

....

P.S. For netdev, i have one more friend - who is complaining that shapers is crashing on Intel machines (who uses TSC, he have two different "Core" based servers, and both is crashing). With HPET i dont have any problem on high performance shapers (except, that it is CPU expensive). It happens on latest 2.6.26.5 too. Machine getting hard lockup, and nothing than hardware watchdog able to recover it. They dont have experience to get actual reason of this issue and they dont know english well to report this issue.

#3 lore raw

Denys Fedoryshchenko wrote, On 09/20/2008 08:11 PM:
...

Quoted 7 lines

P.S. For netdev, i have one more friend - who is complaining that shapers is crashing on Intel machines (who uses TSC, he have two different "Core" based servers, and both is crashing). With HPET i dont have any problem on high performance shapers (except, that it is CPU expensive). It happens on latest 2.6.26.5 too. Machine getting hard lockup, and nothing than hardware watchdog able to recover it. They dont have experience to get actual reason of this issue and they dont know english well to report this issue.

Is your friend sure it's because of shapers? If he/she can patch there is no need to know English well to report here:

Subject: 2.6.26.5 tc not OK

Config:
.config

tc script:
script

dmesg:
dmesg

not OK when: script run/script not run

patch #1 not OK
patch #2 not OK
...
patch #2001 OK!

Jarek P.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 9 lines · Badalian Vyacheslav

Jarek Poplawski:

Hello!
There all requested information.
I try 2.6.26.5 and again get:

[143784.513166] CPU 2: Bank 0: 3200004000000800
[143784.513241] CPU 2: Bank 5: 3200121020080400
[143784.513241] Kernel panic - not syncing: CPU context corrupt
[143784.513282] Rebooting in 3 seconds..

Hi,

Actually, I suggested you to read this Machine Check Exception help, because I think you should first try to test your hardware instead of sending configs. This type of error isn't usually seen with netdev bugs.

Since I'm not a hardware expert I added linux-kernel to Cc, and probably you should do the same (I added it to this one). But, until you have any better advice I think you should do some long and heavy testing of your PCs especially for overheating or memory problems. We can start to analyze other bugs after we are sure the hardware is OK.

BTW, probably your attachements are too big for the lists and the message could be dropped. It would be better to add some link to a server or use bugzilla for this.

Thanks,
Jarek P.

Quoted 63 lines

Attached all info that i was can get from PC. Maybe problem that we use Core Duo Quard processors? It's 64bit, but kernel and software compile as 32. On 2 x "OLD HT(2 core) Xeon 32 bit" PC all work great...

Simple step to reproduce
Add iptables and tc rules.... give above 500 mbs total traffic (we have above 300/200 mbs in/out) from any (many?) ip what preset in TC rules and run any CPU like process (like compiling)...

Thanks for answers!

Denys Fedoryshchenko:
Hello!
i try run nmi_watchdog...
i hope its helps, but this PC have hardware watchdog (bios have params for it), but kernel not have module for it - /S3210SH/ (ICH9-R chipset). I think simple not add ID to driver. I try write to author of it - [email protected].
Please ask for me... this line:

[    0.143332] APIC timer registered as dummy, due to nmi_watchdog=1!

its normal start of nmi_watchdog? or i need use nmi_watchdog=2?

Thanks for answers!

Quoted 37 lines

Denys Fedoryshchenko wrote, On 09/20/2008 08:11 PM:
...

Quoted 8 lines

P.S. For netdev, i have one more friend - who is complaining that shapers is crashing on Intel machines (who uses TSC, he have two different "Core" based servers, and both is crashing). With HPET i dont have any problem on high performance shapers (except, that it is CPU expensive). It happens on latest 2.6.26.5 too. Machine getting hard lockup, and nothing than hardware watchdog able to recover it. They dont have experience to get actual reason of this issue and they dont know english well to report this issue.

Is your friend sure it's because of shapers? If he/she can patch there is no need to know English well to report here:

Subject: 2.6.26.5 tc not OK

Config:
.config

tc script:
script

dmesg:
dmesg

not OK when: script run/script not run

patch #1 not OK
patch #2 not OK
...
patch #2001 OK!

Jarek P.

#6 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

On Mon, Sep 22, 2008 at 06:53:39AM +0000, Jarek Poplawski wrote:
...

Quoted 2 lines

[...]I think you should do some long and heavy
testing of your PCs especially for overheating or memory problems.

but also power supply, network card etc.

Jarek P.

#7 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

In reply to Jarek Poplawski

Thanks for answer Jarek!
I post it is bugtrack - http://bugzilla.kernel.org/show_bug.cgi?id=11618

I not think that its hardware error because this problem we have in 10 servers on 2.6.26.2 kernel +)
On Friday night i compile 2.6.26.5 and have 2 panic on 1 pc what have max load and 1 panic on other pc.
I write to netdev list because first messages looks like:

[ 4956.420298] CPU 1: Machine Check Exception: 0000000000000005
[ 4956.420298] e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
[ 4956.420300]   Tx Queue             <0>
[ 4956.420300]   TDH                  <81>
[ 4956.420301]   TDT                  <81>
[ 4956.420302]   next_to_use          <81>
[ 4956.420302]   next_to_clean        <d6>
[ 4956.420303] buffer_info[next_to_clean]
[ 4956.420303]   time_stamp           <15498d>
[ 4956.420304]   next_to_watch        <d6>
[ 4956.420304]   jiffies              <15511c>
[ 4956.420305]   next_to_watch.status <1>
[ 4956.420537] eth1: Detected Tx Unit Hang:
[ 4956.420538]   TDH                  <b0>
[ 4956.420538]   TDT                  <b0>
[ 4956.420539]   next_to_use          <b0>
[ 4956.420539]   next_to_clean        <5>
[ 4956.420540] buffer_info[next_to_clean]:
[ 4956.420540]   time_stamp           <15498e>
[ 4956.420541]   next_to_watch        <5>
[ 4956.420542]   jiffies              <15511c>
[ 4956.420542]   next_to_watch.status <1>
[ 4956.423064] CPU 1: Bank 0: 3200004000000800
[ 4956.423190] CPU 1: Bank 5: 3200220024080400
[ 4956.423315] Kernel panic - not syncing: CPU context corrupt
[ 4956.423933] Rebooting in 3 seconds..

But in 2.6.26.5 i not see errors like this 2 days... Also if system not have network load - i can't do panic by cpuburn or compiling sources... Anyone i think its good that my message also go to general mail-list and bugzilla...

I try get more info... if you or anyone have idea how test this bug - i can do it)

Thanks!

Quoted 109 lines · Badalian Vyacheslav
Quoted 11 lines

Jarek Poplawski:

Hello!
There all requested information.
I try 2.6.26.5 and again get:

[143784.513166] CPU 2: Bank 0: 3200004000000800
[143784.513241] CPU 2: Bank 5: 3200121020080400
[143784.513241] Kernel panic - not syncing: CPU context corrupt
[143784.513282] Rebooting in 3 seconds..

Hi,

Actually, I suggested you to read this Machine Check Exception help, because I think you should first try to test your hardware instead of sending configs. This type of error isn't usually seen with netdev bugs.

Since I'm not a hardware expert I added linux-kernel to Cc, and probably you should do the same (I added it to this one). But, until you have any better advice I think you should do some long and heavy testing of your PCs especially for overheating or memory problems. We can start to analyze other bugs after we are sure the hardware is OK.

BTW, probably your attachements are too big for the lists and the message could be dropped. It would be better to add some link to a server or use bugzilla for this.

Thanks,
Jarek P.

Quoted 64 lines

Attached all info that i was can get from PC. Maybe problem that we use Core Duo Quard processors? It's 64bit, but kernel and software compile as 32. On 2 x "OLD HT(2 core) Xeon 32 bit" PC all work great...

Simple step to reproduce
Add iptables and tc rules.... give above 500 mbs total traffic (we have above 300/200 mbs in/out) from any (many?) ip what preset in TC rules and run any CPU like process (like compiling)...

Thanks for answers!

Denys Fedoryshchenko:
Hello!
i try run nmi_watchdog...
i hope its helps, but this PC have hardware watchdog (bios have params for it), but kernel not have module for it - /S3210SH/ (ICH9-R chipset). I think simple not add ID to driver. I try write to author of it - [email protected].
Please ask for me... this line:

[    0.143332] APIC timer registered as dummy, due to nmi_watchdog=1!

its normal start of nmi_watchdog? or i need use nmi_watchdog=2?

Thanks for answers!

Quoted 39 lines

Denys Fedoryshchenko wrote, On 09/20/2008 08:11 PM:
...

Quoted 9 lines

P.S. For netdev, i have one more friend - who is complaining that shapers is crashing on Intel machines (who uses TSC, he have two different "Core" based servers, and both is crashing). With HPET i dont have any problem on high performance shapers (except, that it is CPU expensive). It happens on latest 2.6.26.5 too. Machine getting hard lockup, and nothing than hardware watchdog able to recover it. They dont have experience to get actual reason of this issue and they dont know english well to report this issue.

Is your friend sure it's because of shapers? If he/she can patch there is no need to know English well to report here:

Subject: 2.6.26.5 tc not OK

Config:
.config

tc script:
script

dmesg:
dmesg

not OK when: script run/script not run

patch #1 not OK
patch #2 not OK
...
patch #2001 OK!

Jarek P.

#8 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 27 lines · Badalian Vyacheslav

Thanks for answer Jarek!
I post it is bugtrack - http://bugzilla.kernel.org/show_bug.cgi?id=11618

I not think that its hardware error because this problem we have in 10 servers on 2.6.26.2 kernel +)
On Friday night i compile 2.6.26.5 and have 2 panic on 1 pc what have max load and 1 panic on other pc.
I write to netdev list because first messages looks like:

[ 4956.420298] CPU 1: Machine Check Exception: 0000000000000005
[ 4956.420298] e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
[ 4956.420300]   Tx Queue             <0>
[ 4956.420300]   TDH                  <81>
[ 4956.420301]   TDT                  <81>
[ 4956.420302]   next_to_use          <81>
[ 4956.420302]   next_to_clean        <d6>
[ 4956.420303] buffer_info[next_to_clean]
[ 4956.420303]   time_stamp           <15498d>
[ 4956.420304]   next_to_watch        <d6>
[ 4956.420304]   jiffies              <15511c>
[ 4956.420305]   next_to_watch.status <1>
[ 4956.420537] eth1: Detected Tx Unit Hang:
[ 4956.420538]   TDH                  <b0>
[ 4956.420538]   TDT                  <b0>
[ 4956.420539]   next_to_use          <b0>
[ 4956.420539]   next_to_clean        <5>
[ 4956.420540] buffer_info[next_to_clean]:
Quoted 8 lines
[ 4956.420540]   time_stamp           <15498e>
[ 4956.420541]   next_to_watch        <5>
[ 4956.420542]   jiffies              <15511c>
[ 4956.420542]   next_to_watch.status <1>
[ 4956.423064] CPU 1: Bank 0: 3200004000000800
[ 4956.423190] CPU 1: Bank 5: 3200220024080400
[ 4956.423315] Kernel panic - not syncing: CPU context corrupt
[ 4956.423933] Rebooting in 3 seconds..

Yes, similar messages are often netdev problems, but not with this Machine Check Exception with this CPU context corrupt, which should mean some severe hardware problem (unless some bug, probably not netdev, triggers them).

Quoted 5 lines

But in 2.6.26.5 i not see errors like this 2 days... Also if system not have network load - i can't do panic by cpuburn or compiling sources... Anyone i think its good that my message also go to general mail-list and bugzilla...

I try get more info... if you or anyone have idea how test this bug - i can do it)

I see you have some advice in bugzilla. These people really know more about these things, so you should try this first. I think, they expect you to compile the most current kernel version (tip) using git for this. You can do this using the instructions from Ingo Molnar's README. Make a script from this: from the beginning to the "git checkout ...". Of course you have to install git before. After running the commands it will download the kernel sources to a subdir (takes time). Copy your config there, make oldconfig, make etc. Then send them dmesg after rebooting. If you have any problems - write. Alternatively, I guess, you could try the current 2.6.27-rc7 kernel at least.

Jarek P.

BTW: could you try to trigger this bug with one network card off?

#9 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 3 lines

BTW: could you try to trigger this bug with one network card off?

Shire! I stop eth1 and do "/etc/init.d/bgpd stop" (this pc not get route traffic anymore)....

run "emerge portage" 2 times and get:

[25492.187405] CPU 3: Machine Check Exception: 0000000000000005
[25492.187405] MCE: The hardware reports a non fatal, correctable
incident occurred on CPU 1.
[25492.187405] Bank 0: b200004000000800
[25492.187405] MCE: The hardware reports a non fatal, correctable
incident occurred on CPU 1.
[25492.187405] Bank 5: b200120014040400
[25497.124884] CPU 1: Machine Check Exception: 0000000000000004
[25497.124884] Kernel panic - not syncing: Unable to continue
[25497.124884] Rebooting in 3 seconds..

bugtracker updated.... i can get (reproduce) error on all 10 servers at 2.6.26.5... I use TC and not wont test 2.6.27-rc because its have (if i understand) multiqueue feature that not tested...

Thanks!

#10 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 21 lines · Badalian Vyacheslav
Quoted 3 lines

BTW: could you try to trigger this bug with one network card off?

Shire! I stop eth1 and do "/etc/init.d/bgpd stop" (this pc not get route traffic anymore)....

run "emerge portage" 2 times and get:

[25492.187405] CPU 3: Machine Check Exception: 0000000000000005
[25492.187405] MCE: The hardware reports a non fatal, correctable
incident occurred on CPU 1.
[25492.187405] Bank 0: b200004000000800
[25492.187405] MCE: The hardware reports a non fatal, correctable
incident occurred on CPU 1.
[25492.187405] Bank 5: b200120014040400
[25497.124884] CPU 1: Machine Check Exception: 0000000000000004
[25497.124884] Kernel panic - not syncing: Unable to continue
[25497.124884] Rebooting in 3 seconds..
Quoted 3 lines

bugtracker updated.... i can get (reproduce) error on all 10 servers at 2.6.26.5... I use TC and not wont test 2.6.27-rc because its have (if i understand) multiqueue feature that not tested...

Actually, it's quite well tested, especially by Denys, and I doubt it will be much better in 2.6.27. BTW, maybe start eth1, stop eth0 yet?

Thanks,
Jarek P.

#11 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Hello

I stop eth1 and eth0 and run "emegre portage" and get exception. Now i think its not problem in network part.

I miss situation in 2.6.27 about multiqueue and traffic shaper because i was have many work and was can't read all netdev list =(
As i understand 2.6.27-rc have support multiqueue, but how it will work with HTB/SFQ?
Is tc rules must have in 2.6.27 one root queue (and all queue go to this tree) or need to do many qdiscs and settings it to device queues (i was read some about queue2band)?
If it is simple for you - can you sort describe this part of changes?

P.S. I think now that problem not in network part of kernel and i think i stop CC netdev and Denys Fedoryshchenko. Thanks for you doing and thanks for help!

Thanks.

Quoted 7 lines

Actually, it's quite well tested, especially by Denys, and I doubt it will be much better in 2.6.27. BTW, maybe start eth1, stop eth0 yet?

Thanks,
Jarek P.

#12 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 5 lines · Badalian Vyacheslav

Hello

I stop eth1 and eth0 and run "emegre portage" and get exception. Now i think its not problem in network part.

Quoted 8 lines

I miss situation in 2.6.27 about multiqueue and traffic shaper because i was have many work and was can't read all netdev list =(
As i understand 2.6.27-rc have support multiqueue, but how it will work with HTB/SFQ?
Is tc rules must have in 2.6.27 one root queue (and all queue go to this tree) or need to do many qdiscs and settings it to device queues (i was read some about queue2band)?
If it is simple for you - can you sort describe this part of changes?

There are two main cases:

1) The default qdiscs (created while activating a new net device):
depending on a driver (most drivers are still uniqueue), there are created independent pfifo_fast_qdiscs for each supported tx queue; if a driver doesn't change this, packets are directed to them automatically, according to some hash function, which tries to separate different flows. This should be the fastest solution because there are separate qdisc and transmit locks, which could be taken by different cpus at the same time.

2) Non-default qdiscs (any qdiscs added with tc): there is only one root qdisc (with its tree) as before, dequeued to all tx queues (if available). Since there is only one qdisc lock, and additional flag preventing other processes to run the qdisc at the same time, there is not so much advantage of SMP, except on tx locking. All previous tc configs should work without changes (except sch_prio and sch_rr used for multiqueuing, replaced by sch_multiq and act_skbedit now). Probably in some cases adding sch_multiq to a tree for separating qdisc queues per tx queues could be useful.

Cheers,
Jarek P.

#13 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 11 lines

2) Non-default qdiscs (any qdiscs added with tc): there is only one root qdisc (with its tree) as before, dequeued to all tx queues (if available). Since there is only one qdisc lock, and additional flag preventing other processes to run the qdisc at the same time, there is not so much advantage of SMP, except on tx locking. All previous tc configs should work without changes (except sch_prio and sch_rr used for multiqueuing, replaced by sch_multiq and act_skbedit now). Probably in some cases adding sch_multiq to a tree for separating qdisc queues per tx queues could be useful.

Very thanks for detailed information!
Yeh! Its sound great for me. I also can stress test this feature in our network if you will needed it.

Only i have 2 question ...

1. If kernel use default situation (no tc user create rules, simple autocreate by network card driver/module) its will normal work with traffic what must delivered "as is" (not shape)... like IPTV or other multicast/unicast video stream...
If i understand logic
we have 8 cpu/core and 4 TX queue and 4 RX... one cpu linked to 1 TX/RX.... but if 1 cpu is burned by some process - this cpu will send its packet later when other cpu and packet is shape... but streams must go packet by packet to receive device... I understand that it simple need use hash function for TX queue, but if i understand - RX can't separate packets to different queues (its doing by hardware?) by hash function and packets may shape in rx stage because one CPU get it letter when nedded?
I do not wish to spend your time... only say if it will work normal by default and i will sleep easy :)

2. I can locate module like sch_multiq at last 2.6.27-rc tree and not have information of it in google... I need to know only one thing - what params for hashing was planned for it?

Thanks and thanks again! And again sorry for my English.

Best regals, Badalyan Vyacheslav.

#14 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 11 lines · Badalian Vyacheslav
Quoted 11 lines

2) Non-default qdiscs (any qdiscs added with tc): there is only one root qdisc (with its tree) as before, dequeued to all tx queues (if available). Since there is only one qdisc lock, and additional flag preventing other processes to run the qdisc at the same time, there is not so much advantage of SMP, except on tx locking. All previous tc configs should work without changes (except sch_prio and sch_rr used for multiqueuing, replaced by sch_multiq and act_skbedit now). Probably in some cases adding sch_multiq to a tree for separating qdisc queues per tx queues could be useful.

Quoted 3 lines

Very thanks for detailed information!
Yeh! Its sound great for me. I also can stress test this feature in our network if you will needed it.

Actually, I don't use these things too much, but I guess, you'll need this more. Main issues were tested and fixed, but there could be always some details not used, not noticed or not reported until you decide to use this.

Quoted 18 lines

Only i have 2 question ...

1. If kernel use default situation (no tc user create rules, simple autocreate by network card driver/module) its will normal work with traffic what must delivered "as is" (not shape)... like IPTV or other multicast/unicast video stream...
If i understand logic
we have 8 cpu/core and 4 TX queue and 4 RX... one cpu linked to 1 TX/RX.... but if 1 cpu is burned by some process - this cpu will send its packet later when other cpu and packet is shape... but streams must go packet by packet to receive device... I understand that it simple need use hash function for TX queue, but if i understand - RX can't separate packets to different queues (its doing by hardware?) by hash function and packets may shape in rx stage because one CPU get it letter when nedded?
I do not wish to spend your time... only say if it will work normal by default and i will sleep easy :)

Yes, I'm not sure I understand question, but I think you shouldn't expect too much, at least in 2.6.27. There is many work now in drivers around this multiqueing (and RX hashing), which should be available in next kernels, but I'm not tracking this too much... Anyway, with the basic support (which really isn't common for drivers in 2.6.27 yet), this separation is done only for TX just before enqueuing.

Quoted 4 lines

2. I can locate module like sch_multiq at last 2.6.27-rc tree and not have information of it in google... I need to know only one thing - what params for hashing was planned for it?

sch_multiq doesn't use any params for hashing now - it uses mapping in packets to separate them to different bands/queues. So, by default it'll respect common hashing. You can change this using any filter with act_skbedit (Documentation/networking/multiqueue.txt).

Quoted 1 line

Thanks and thanks again! And again sorry for my English.

Don't worry, English should understand...

Jarek P.

#15 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Quoted 11 lines · Jarek Poplawski
Quoted 10 lines · Badalian Vyacheslav
Quoted 10 lines

2) Non-default qdiscs (any qdiscs added with tc): there is only one root qdisc (with its tree) as before, dequeued to all tx queues (if available). Since there is only one qdisc lock, and additional flag preventing other processes to run the qdisc at the same time, there is not so much advantage of SMP, except on tx locking. All previous tc configs should work without changes (except sch_prio and sch_rr used for multiqueuing, replaced by sch_multiq and act_skbedit now). Probably in some cases adding sch_multiq to a tree for separating qdisc queues per tx queues could be useful.

...

Quoted 8 lines
Quoted 3 lines

2. I can locate module like sch_multiq at last 2.6.27-rc tree and not have information of it in google... I need to know only one thing - what params for hashing was planned for it?

sch_multiq doesn't use any params for hashing now - it uses mapping in packets to separate them to different bands/queues. So, by default it'll respect common hashing. You can change this using any filter with act_skbedit (Documentation/networking/multiqueue.txt).

OOPS!!! This 2.6.27-rc is so ...old I forgot the sch_multiq and act_skbedit are only for the -next!

I'm very sorry for misleading!!!
Jarek P.

#16 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

In reply to Jarek Poplawski
Quoted 6 lines

Actually, I don't use these things too much, but I guess, you'll need this more. Main issues were tested and fixed, but there could be always some details not used, not noticed or not reported until you decide to use this.

Yep. I will begin test it in this week if i can get time for it...

Quoted 8 lines

Yes, I'm not sure I understand question, but I think you shouldn't expect too much, at least in 2.6.27. There is many work now in drivers around this multiqueing (and RX hashing), which should be available in next kernels, but I'm not tracking this too much... Anyway, with the basic support (which really isn't common for drivers in 2.6.27 yet), this separation is done only for TX just before enqueuing.

I simple hope that after divide to bands/queues and go back to network all packets save "First in and First Out" logic that needed in services like video streaming...

Quoted 6 lines

sch_multiq doesn't use any params for hashing now - it uses mapping in packets to separate them to different bands/queues. So, by default it'll respect common hashing. You can change this using any filter with act_skbedit (Documentation/networking/multiqueue.txt).

Ok! I will read all related information and rewrite all tc generate scripts to test it!

Quoted 3 lines

Don't worry, English should understand...

Great what world have people that read messages like this and try understand it... thanks!

Quoted 4 lines

OOPS!!! This 2.6.27-rc is so ...old I forgot the sch_multiq and act_skbedit are only for the -next!

I'm very sorry for misleading!!

not to worry ;) Thanks again. I try last -next git after someone fix Intel exception issue ;)

#17 lore raw

Re: Machine Check Exception Re: NetDev! Please help!

Badalian Vyacheslav wrote, On 09/23/2008 02:16 PM:
...

Quoted 3 lines

I simple hope that after divide to bands/queues and go back to network all packets save "First in and First Out" logic that needed in services like video streaming...

Of course this is aimed at, and if it's really first in per flow, and flows are bound to devs/irqs, after hashing should be the same. But... shit happens - e.g. recently there has been fixed a bug in the hash which could change this order. And you should remember this default qdisc is prioritized fifo, so you should be careful with TOS etc.

Quoted 8 lines
Quoted 6 lines

sch_multiq doesn't use any params for hashing now - it uses mapping in packets to separate them to different bands/queues. So, by default it'll respect common hashing. You can change this using any filter with act_skbedit (Documentation/networking/multiqueue.txt).

Ok! I will read all related information and rewrite all tc generate scripts to test it!

Do you mean net-next? I think you should first do some tests if you really need this.

...

Quoted 2 lines

not to worry ;) Thanks again. I try last -next git after someone fix Intel exception issue ;)

I think, maybe you could try what Ingo Molnar asked: -tip git, for now?

Jarek P.

#18 lore raw

Re: Machine Check Exception Was: NetDev! Please help!

In reply to Badalian Vyacheslav

Badalian Vyacheslav wrote, On 09/20/2008 03:19 PM:

Quoted 1 line

Hello all.

Hi Vyacheslav,

I think it might be something more than netdev. Please, read in the kernel config the comment to Processor type and features/Machine Check Exception.

I pasted your second message below and Cc linux-kernel.

Jarek P.

Quoted 94 lines

We buy 10 Intel servers and paste it to shape traffic. After 5-15 hours all PC is was freeze! Kernel not see TCO watchdog at this platform and can't reboot it!. Soft Watchdog not reboot pc in this situation. =(

At screen we see messages like this (when it freeze and i was near monitor):

http://www.kerneloops.org/guilty.php?guilty=dev_watchdog&version=2.6.26-release&start=1736704&end=1769471&class=warn

Also by netconsole we was get this one time:

[ 1352.245851] netconsole: network logging started
[ 1458.400133] 802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
[ 1458.400133] All bugs added by David S. Miller <[email protected]>
[ 4956.420298] CPU 1: Machine Check Exception: 0000000000000005
[ 4956.420298] e1000: eth0: e1000_clean_tx_irq: Detected Tx Unit Hang
[ 4956.420300]   Tx Queue             <0>
[ 4956.420300]   TDH                  <81>
[ 4956.420301]   TDT                  <81>
[ 4956.420302]   next_to_use          <81>
[ 4956.420302]   next_to_clean        <d6>
[ 4956.420303] buffer_info[next_to_clean]
[ 4956.420303]   time_stamp           <15498d>
[ 4956.420304]   next_to_watch        <d6>
[ 4956.420304]   jiffies              <15511c>
[ 4956.420305]   next_to_watch.status <1>
[ 4956.420537] eth1: Detected Tx Unit Hang:
[ 4956.420538]   TDH                  <b0>
[ 4956.420538]   TDT                  <b0>
[ 4956.420539]   next_to_use          <b0>
[ 4956.420539]   next_to_clean        <5>
[ 4956.420540] buffer_info[next_to_clean]:
[ 4956.420540]   time_stamp           <15498e>
[ 4956.420541]   next_to_watch        <5>
[ 4956.420542]   jiffies              <15511c>
[ 4956.420542]   next_to_watch.status <1>
[ 4956.423064] CPU 1: Bank 0: 3200004000000800
[ 4956.423190] CPU 1: Bank 5: 3200220024080400
[ 4956.423315] Kernel panic - not syncing: CPU context corrupt
[ 4956.423933] Rebooting in 3 seconds..[  531.843998] CPU 2: Machine
Check Exception: 0000000000000005
[  531.843998] CPU 0: Machine Check Exception: 0000000000000004
[  531.844000] CPU 0: Bank 0: 3200004000000800
[  531.844001] CPU 0: Bank 5: 3200121014040400
[  531.844002] Kernel panic - not syncing: CPU context corrupt
[  531.844916] Rebooting in 3 seconds..

This out of lspci:

00:00.0 Host bridge: Intel Corporation Server DRAM Controller 00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation USB UHCI Controller #5 (rev 02) 00:1a.2 USB Controller: Intel Corporation USB UHCI Controller #6 (rev 02) 00:1a.7 USB Controller: Intel Corporation USB2 EHCI Controller #2 (rev 02) 00:1c.0 PCI bridge: Intel Corporation PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel Corporation PCI Express Port 5 (rev 02) 00:1d.0 USB Controller: Intel Corporation USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) 00:1f.0 ISA bridge: Intel Corporation LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 6 port SATA AHCI Controller (rev 02)
00:1f.3 SMBus: Intel Corporation SMBus Controller (rev 02) 02:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200e [Pilot] ServerEngines (SEP1) (rev 02)
03:02.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

lspci -tv

-[0000:00]-+-00.0  Intel Corporation Server DRAM Controller
           +-19.0  Intel Corporation 82566DM-2 Gigabit Network Connection
           +-1a.0  Intel Corporation USB UHCI Controller #4
           +-1a.1  Intel Corporation USB UHCI Controller #5
           +-1a.2  Intel Corporation USB UHCI Controller #6
           +-1a.7  Intel Corporation USB2 EHCI Controller #2
           +-1c.0-[0000:01]--
           +-1c.4-[0000:02]----00.0  Matrox Graphics, Inc. MGA G200e
[Pilot] ServerEngines (SEP1)
           +-1d.0  Intel Corporation USB UHCI Controller #1
           +-1d.1  Intel Corporation USB UHCI Controller #2
           +-1d.2  Intel Corporation USB UHCI Controller #3
           +-1d.7  Intel Corporation USB2 EHCI Controller #1
           +-1e.0-[0000:03]----02.0  Intel Corporation 82541GI Gigabit
Ethernet Controller
           +-1f.0  Intel Corporation LPC Interface Controller
           +-1f.2  Intel Corporation 6 port SATA AHCI Controller
           \-1f.3  Intel Corporation SMBus Controller

All tests in 2.6.26.2 kernel... now i try compile 2.6.26.5 kernel, but if KernelOops is right - its not will help.

Quoted 6 lines

I hope NetDev help!
If any information or test is needed - please write!

Thanks for anyone!

Badalian Vyacheslav.

Badalian Vyacheslav wrote, On 09/20/2008 03:38 PM:

Quoted 42 lines

New crash.... netconsole log:

[  116.333349] ------------[ cut here ]------------
[  116.333516] WARNING: at net/sched/sch_generic.c:222
dev_watchdog+0xf1/0x110()
[  116.333690] Modules linked in: netconsole i2c_i801 i2c_core e1000e e1000
[  116.334199] Pid: 0, comm: swapper Not tainted 2.6.26-gentoo-r1-fw #2
[  116.334371]  [<c012506f>] warn_on_slowpath+0x5f/0x90
[  116.334597]  [<c011dd1a>] enqueue_task_fair+0x1a/0x30
[  116.334823]  [<c011b962>] enqueue_task+0x12/0x30
[  116.335046]  [<c011b9d3>] activate_task+0x23/0x40
[  116.335268]  [<c011e01a>] try_to_wake_up+0x6a/0x110
[  116.335491]  [<c0137c7b>] autoremove_wake_function+0x1b/0x50
[  116.335718]  [<c011be6b>] __wake_up_common+0x4b/0x80
[  116.335941]  [<c011cfde>] __wake_up+0x3e/0x60
[  116.336161]  [<c0134a2b>] insert_work+0x4b/0x70
[  116.336384]  [<c0134dd7>] __queue_work+0x27/0x40
[  116.336610]  [<c02d0651>] dev_watchdog+0xf1/0x110
[  116.337333]  [<c012e055>] run_timer_softirq+0x115/0x170
[  116.337557]  [<c0122c71>] scheduler_tick+0xa1/0xd0
[  116.337780]  [<c012a062>] __do_softirq+0x82/0x100
[  116.338002]  [<c012a117>] do_softirq+0x37/0x40
[  116.338222]  [<c0114027>] smp_apic_timer_interrupt+0x57/0x90
[  116.338448]  [<c0105660>] apic_timer_interrupt+0x28/0x30
[  116.338672]  [<c010a5e2>] mwait_idle+0x32/0x40
[  116.338894]  [<c010a5b0>] mwait_idle+0x0/0x40
[  116.339115]  [<c01036e8>] cpu_idle+0x48/0xc0
[  116.339336]  =======================
[  116.339499] ---[ end trace e25a40b7dc59df07 ]---
[  117.655918] CPU 1: Machine Check Exception: 0000000000000005
[  117.656103] CPU 1: Bank 0: 3200004000000800
[  117.656604] CPU 1: Bank 5: 3200220024080400
[  117.656604] Kernel panic - not syncing: CPU context corrupt
[  117.656624] Rebooting in 3 seconds..

Thanks
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html