Skip to contentSkip to conversation

kernel BUG at page_alloc.c:98 -- compiling with distcc

DISCUSSION Marco Fais 7 participants 20 messages Apr 2, 2004 – May 20, 2004 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

Marco Fais

author
#0 lore raw

Hi!

[1.] Kernel panic while using distcc

[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

[3.] Keywords: distcc BUG page_alloc.c

[4.] Linux version 2.4.25 (root@test1) (gcc version 3.2 20020903 (Red Hat Linux 8.0 3.2-7)) #1 mer mar 31 10:28:36 CEST 2004

[5.]
ksymoops 2.4.5 on i686 2.4.25. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.25/ (default)
-m /boot/System.map-2.4.25 (default)

Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.

kernel BUG at page_alloc.c:98!

invalid operand: 0000
CPU:    0
EIP:    0010:[<c01372ae>]    Not tainted

Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: 00000002 ebx: c14b3f00 ecx: c14b3f00 edx: 00000000
esi: 00000000 edi: dec11340 ebp: c02f1d04 esp: c02f1cd4
ds: 0018 es: 0018 ss: 0018
Process swapper (pid: 0, stackpage=c02f1000)

Stack: ddd46000 c02f1cfc c0135a76 c158f6f0 de9bcdf4 ddd45800 de9bcdf4
005207dc
      ddd45800 00000001 dec4d894 dec11340 c02f1d18 c021667b 00000282
dec4d894
      dec4d8c4 c02f1d2c c02166b4 dec4d894 dec4d894 dec4d894 c02f1d44
c0216816
Call Trace:    [<c0135a76>] [<c021667b>] [<c02166b4>] [<c0216816>]

[<c023be39>]
[<c023c385>] [<c023f51c>] [<c02465a9>] [<c0246a76>] [<c022dad0>]
[<c022dc25>]
[<c0222780>] [<c022dad0>] [<c022d88f>] [<c022dad0>] [<c022de3a>]
[<e08d7eab>]
[<c021ad14>] [<c021ae3f>] [<c021af5a>] [<c0121cd7>] [<c010a66d>]
[<c01070a0>]
[<c010cb58>] [<c01070a0>] [<c01070c6>] [<c0107142>] [<c0105000>]

Code: 0f 0b 62 00 f7 60 27 c0 e9 ad fd ff ff 90 8d 74 26 00 55 89
Quoted 7 lines
Quoted 7 lines

EIP; c01372ae <__free_pages_ok+26e/280> <=====

ebx; c14b3f00 <_end+116e728/204d48a8>
ecx; c14b3f00 <_end+116e728/204d48a8>
edi; dec11340 <_end+1e8cbb68/204d48a8>
ebp; c02f1d04 <init_task_union+1d04/2000>
esp; c02f1cd4 <init_task_union+1cd4/2000>

Trace; c0135a76 <kmem_cache_free_one+f6/210> Trace; c021667b <skb_release_data+6b/90> Trace; c02166b4 <kfree_skbmem+14/70>
Trace; c0216816 <__kfree_skb+106/160>
Trace; c023be39 <tcp_clean_rtx_queue+139/330> Trace; c023c385 <tcp_ack+c5/380>
Trace; c023f51c <tcp_rcv_state_process+19c/a90> Trace; c02465a9 <tcp_v4_do_rcv+a9/130>
Trace; c0246a76 <tcp_v4_rcv+446/560>
Trace; c022dad0 <ip_local_deliver_finish+0/180> Trace; c022dc25 <ip_local_deliver_finish+155/180> Trace; c0222780 <nf_hook_slow+b0/170>
Trace; c022dad0 <ip_local_deliver_finish+0/180> Trace; c022d88f <ip_local_deliver+4f/70> Trace; c022dad0 <ip_local_deliver_finish+0/180> Trace; c022de3a <ip_rcv_finish+1ea/270> Trace; e08d7eab <[8139too]rtl8139_rx_interrupt+6b/3b0> Trace; c021ad14 <netif_receive_skb+c4/180> Trace; c021ae3f <process_backlog+6f/120> Trace; c021af5a <net_rx_action+6a/100>
Trace; c0121cd7 <do_softirq+97/a0>
Trace; c010a66d <do_IRQ+bd/f0>
Trace; c01070a0 <default_idle+0/30>
Trace; c010cb58 <call_do_IRQ+5/d>
Trace; c01070a0 <default_idle+0/30>
Trace; c01070c6 <default_idle+26/30>
Trace; c0107142 <cpu_idle+42/60>
Trace; c0105000 <_stext+0/0>

Code; c01372ae <__free_pages_ok+26e/280>
00000000 <_EIP>:

Code;  c01372ae <__free_pages_ok+26e/280>   <=====
  0:   0f 0b                     ud2a      <=====
Code;  c01372b0 <__free_pages_ok+270/280>
  2:   62 00                     bound  %eax,(%eax)
Code;  c01372b2 <__free_pages_ok+272/280>
  4:   f7 60 27                  mull   0x27(%eax)
Code;  c01372b5 <__free_pages_ok+275/280>
  7:   c0 e9 ad                  shr    $0xad,%cl
Code;  c01372b8 <__free_pages_ok+278/280>
  a:   fd                        std
Code;  c01372b9 <__free_pages_ok+279/280>
  b:   ff                        (bad)
Code;  c01372ba <__free_pages_ok+27a/280>
  c:   ff 90 8d 74 26 00         call   *0x26748d(%eax)
Code;  c01372c0 <rmqueue+0/230>
 12:   55                        push   %ebp
Code;  c01372c1 <rmqueue+1/230>
 13:   89 00                     mov    %eax,(%eax)

<0>Kernel panic: Aiee, killing interrupt handler!

1 warning issued. Results may not be reliable.

[6.] Launch distccd --daemon on the affected system, then on the remote host, set DISTCC_HOSTS="<problematic remote system>" and launch, for example, a kernel compile: make -j2 CC=distcc bzImage.

[7.] All system are AthlonXP 2.6+, on a VIA KT400 chipset (various motherboard vendors). All using EXT3 filesystems, with various redhat distributions (8.0, 9, RHEL3 -- not using NPTL)

[7.1.]

Gnu C                  3.2
Gnu make               3.79.1
util-linux             2.11r
mount                  2.11r
modutils               2.4.18
e2fsprogs              1.27
jfsutils               1.0.17
reiserfsprogs          3.6.2
pcmcia-cs              3.1.31
quota-tools            3.06.
PPP                    2.4.1
isdn4k-utils           3.1pre4
Linux C Library        2.3.2
Dynamic linker (ldd)   2.3.2
Procps                 2.0.7
Net-tools              1.60
Kbd                    1.06
Sh-utils               2.0.12

[7.2.] Processor information (from /proc/cpuinfo):

processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) XP 2600+
stepping : 1
cpu MHz : 2075.355
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
bogomips : 4141.87

[7.3.] Module information (from /proc/modules):

Module Size Used by Not tainted

nfs                    78968   1  (autoclean)
binfmt_misc             7304   1
nfsd                   80304   8  (autoclean)
lockd                  58480   1  (autoclean) [nfs nfsd]
sunrpc                 84188   1  (autoclean) [nfs nfsd lockd]
8139too                19784   2
mii                     3944   0  [8139too]
crc32                   3680   0  [8139too]
iptable_filter          2412   0  (autoclean) (unused)
ip_tables              15392   1  [iptable_filter]
ohci1394               33608   0  (unused)
ieee1394               64676   0  [ohci1394]
mousedev                5428   0  (unused)
keybdev                 3072   0  (unused)
input                   5824   0  [mousedev keybdev]
hid                    12248   0  (unused)
rtc                     8764   0  (autoclean)

[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)

0000-001f : dma1
0020-003f : pic1
0040-005f : timer
0060-006f : keyboard
0070-007f : rtc
0080-008f : dma page reg
00a0-00bf : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial(auto)
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial(auto)
0cf8-0cff : PCI conf1
c000-c0ff : Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+
c000-c0ff : 8139too
c400-c4ff : Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (#2)
c400-c4ff : 8139too
d400-d47f : VIA Technologies, Inc. IEEE 1394 Host Controller d800-d8ff : C-Media Electronics Inc CM8738
dc00-dc1f : VIA Technologies, Inc. USB
e000-e01f : VIA Technologies, Inc. USB (#2)
e400-e41f : VIA Technologies, Inc. USB (#3)
e800-e80f : VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE
e800-e807 : ide0
e808-e80f : ide1

00000000-0009ffff : System RAM
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000f0000-000fffff : System ROM
00100000-1ffeffff : System RAM
00100000-002667cb : Kernel code
002667cc-002ef563 : Kernel data
1fff0000-1fff2fff : ACPI Non-volatile Storage 1fff3000-1fffffff : ACPI Tables
d0000000-dfffffff : PCI Bus #01
d0000000-d7ffffff : nVidia Corporation NV17 [GeForce4 MX 440] d8000000-d807ffff : nVidia Corporation NV17 [GeForce4 MX 440]
e0000000-e3ffffff : VIA Technologies, Inc. VT8377 [KT400 AGP] Host Bridge e4000000-e5ffffff : PCI Bus #01
e4000000-e4ffffff : nVidia Corporation NV17 [GeForce4 MX 440]
e6020000-e60200ff : Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (#2)
e6020000-e60200ff : 8139too
e6022000-e60220ff : Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+
e6022000-e60220ff : 8139too
e6023000-e60237ff : VIA Technologies, Inc. IEEE 1394 Host Controller
e6023000-e60237ff : ohci1394
e6024000-e60240ff : VIA Technologies, Inc. USB 2.0 fec00000-fec00fff : reserved
fee00000-fee00fff : reserved
ffff0000-ffffffff : reserved

[7.5.] PCI information

00:00.0 Host bridge: VIA Technologies, Inc.: Unknown device 3189
Subsystem: VIA Technologies, Inc.: Unknown device 3189
Flags: bus master, 66Mhz, medium devsel, latency 8 Memory at e0000000 (32-bit, prefetchable) [size=64M]
Capabilities: [a0] AGP version 2.0
Capabilities: [c0] Power Management version 2

00:01.0 PCI bridge: VIA Technologies, Inc.: Unknown device b168 (prog-if 00 [Normal decode])
Flags: bus master, 66Mhz, medium devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 Memory behind bridge: e4000000-e5ffffff
Prefetchable memory behind bridge: d0000000-dfffffff
Capabilities: [80] Power Management version 2

00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Flags: bus master, medium devsel, latency 32, IRQ 17 I/O ports at c000 [size=256]
Memory at e6022000 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2

00:0b.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10)
Subsystem: Realtek Semiconductor Co., Ltd. RT8139
Flags: bus master, medium devsel, latency 32, IRQ 19 I/O ports at c400 [size=256]
Memory at e6020000 (32-bit, non-prefetchable) [size=256]
Capabilities: [50] Power Management version 2

00:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46) (prog-if 10 [OHCI])
Subsystem: Biostar Microtech Int'l Corp: Unknown device 4200
Flags: bus master, medium devsel, latency 32, IRQ 18 Memory at e6023000 (32-bit, non-prefetchable) [size=2K] I/O ports at d400 [size=128]
Capabilities: [50] Power Management version 2

00:0f.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)
Subsystem: Biostar Microtech Int'l Corp: Unknown device 8738
Flags: bus master, medium devsel, latency 32, IRQ 19
I/O ports at d800 [size=256]
Capabilities: [c0] Power Management version 2

00:10.0 USB Controller: VIA Technologies, Inc. USB (rev 80) (prog-if 00 [UHCI])
Subsystem: VIA Technologies, Inc. USB
Flags: bus master, medium devsel, latency 32, IRQ 21 I/O ports at dc00 [size=32]
Capabilities: [80] Power Management version 2

00:10.1 USB Controller: VIA Technologies, Inc. USB (rev 80) (prog-if 00 [UHCI])
Subsystem: VIA Technologies, Inc. USB
Flags: bus master, medium devsel, latency 32, IRQ 21 I/O ports at e000 [size=32]
Capabilities: [80] Power Management version 2

00:10.2 USB Controller: VIA Technologies, Inc. USB (rev 80) (prog-if 00 [UHCI])
Subsystem: VIA Technologies, Inc. USB
Flags: bus master, medium devsel, latency 32, IRQ 21 I/O ports at e400 [size=32]
Capabilities: [80] Power Management version 2

00:10.3 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 82) (prog-if 20 [EHCI])
Subsystem: VIA Technologies, Inc. USB 2.0
Flags: bus master, medium devsel, latency 32, IRQ 19 Memory at e6024000 (32-bit, non-prefetchable) [size=256]
Capabilities: [80] Power Management version 2

00:11.0 ISA bridge: VIA Technologies, Inc. VT8233A ISA Bridge
Subsystem: VIA Technologies, Inc. VT8233A ISA Bridge
Flags: bus master, stepping, medium devsel, latency 0
Capabilities: [c0] Power Management version 2

00:11.1 IDE interface: VIA Technologies, Inc. VT82C586B PIPC Bus Master IDE (rev 06) (prog-if 8a [Master SecP PriP])
Subsystem: VIA Technologies, Inc. VT82C586B PIPC Bus Master IDE
Flags: bus master, medium devsel, latency 32, IRQ 11 I/O ports at e800 [size=16]
Capabilities: [c0] Power Management version 2

01:00.0 VGA compatible controller: nVidia Corporation NV17 [GeForce4 MX440] (rev a3) (prog-if 00 [VGA])
Flags: bus master, 66Mhz, medium devsel, latency 32, IRQ 16 Memory at e4000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (32-bit, prefetchable) [size=128M] Memory at d8000000 (32-bit, prefetchable) [size=512K] Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [60] Power Management version 2
Capabilities: [44] AGP version 2.0

CPU0

 0:     292363    IO-APIC-edge  timer
 1:          3    IO-APIC-edge  keyboard
 2:          0          XT-PIC  cascade
 8:          1    IO-APIC-edge  rtc
12:          0          XT-PIC  PS/2 Mouse
14:       8958    IO-APIC-edge  ide0
15:          4    IO-APIC-edge  ide1
17:       6482   IO-APIC-level  eth0
18:          2   IO-APIC-level  ohci1394
19:         28   IO-APIC-level  eth1
NMI:          0
LOC:     292280
ERR:          0
MIS:          0

[7.7.] Other information that might be relevant to the problem

Other systems (DL-360G3 dual Xeon 2.8 GHz, RHEL3, SMP or UP kernel) doesn't show the problem.

#1 lore raw
Quoted 6 lines · On Friday April 2nd 2004 Marco Fais wrote:

[...]

When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

Quoted 7 lines
Quoted 1 line
Quoted 1 line

EIP; c01372ae <__free_pages_ok+26e/280> <=====

...
Trace; e08d7eab <[8139too]rtl8139_rx_interrupt+6b/3b0>

<0>Kernel panic: Aiee, killing interrupt handler!

From a very superficial examination of your data, it looks like there is

something going wrong in the interrupt handling of the driver for (one of) the network cards.

Distcc can generate a lot of network traffic. You might experiment with switching the role of the two network cards (in case there might be something wrong with the hardware of one of them) or use the '--listen' directive in the distccd configuration to do so.

If the panic is indeed caused by the network driver, then it should also be possible to trigger and debug this with a tool like netcat (listen on the panicking box with 'nc -l someport' and send some stuff from another box ('cat /dev/zero | nc panicker someport' or vice versa).

Sadly, nothing of this will solve your problem of course, but it might pinpoint the cause somewhat more accurately, leading hopefully to a solution!

Signature · 1 line
Marco Roeland

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 2 lines · On Friday April 2nd 2004 Marco Fais wrote:

Mmmh, all the servers use an RTL-8139 compatible card, with the same 8139too driver. So this can be the problem.

Hey, I'm by no means an expert. Suggesting the driver is to blame was mostly based on the fact that compiling locally worked, and from a remote machine triggered a panick. The rest of your description below indicates that it probably *isnt't* the driver.

Quoted 6 lines

But in this moment I'm doing a kernel compile while receiving and sending huge amounts of data using netcat, as you suggested... and works perfectly.

Ok, next I will test the second network card on the server, just to avoid the possibility of an hardware failure -- but I have other 4 servers that show the same behaviour, so I don't think it's caused by faulty hardware.

If 4 other servers show the same behaviour, and netcatting a lot of data doesn't panick the machine, that highly suggests that the network card and driver are innocent! I thought only one machine had the problem.

Quoted 5 lines

Running this test for about an hour, using all the available bandwidth on the NIC, while compiling the kernel in a loop... no problem. Using distcc, compiling the same files, cause a kernel panic in a few seconds. So this test doesn't show the problem, but I think that anyway the network card driver (or the hardware) is involved.

Why do you think so, it seems there's nothing wrong with it; you've just tested that?

One last suggestion:

Have you tried a local distcc compile, but specifying the host name as it's IP address or its real name. Distcc treats 'localhost' differently, but if it sees an IP address it will use the network route. As specified in the man page this is slower, but if there's something peculiar with the interaction of distcc with the network layer, then perhaps this triggers it. You can also use the '--verbose' option on distccd, perhaps it reports something useful before panicking.

Signature · 1 line
Marco Roeland
author
#4 lore raw

Marco Roeland ha scritto:

Quoted 5 lines
Quoted 1 line

Mmmh, all the servers use an RTL-8139 compatible card, with the same 8139too driver. So this can be the problem.

Hey, I'm by no means an expert. Suggesting the driver is to blame was mostly based on the fact that compiling locally worked, and from a remote machine triggered a panick. The rest of your description below indicates that it probably *isnt't* the driver.

I was not saying *this is the problem*, just noticing that all the systems that show this problem have this network card, while the other systems that are working perfectly are using other network hardware (e100 driver) :)

Quoted 4 lines
Quoted 1 line

Ok, next I will test the second network card on the server, just to avoid the possibility of an hardware failure -- but I have other 4 servers that show the same behaviour, so I don't think it's caused by faulty hardware.

If 4 other servers show the same behaviour, and netcatting a lot of data doesn't panick the machine, that highly suggests that the network card and driver are innocent! I thought only one machine had the problem.

If you read Andrew's message, seems that distcc uses a function that trigger the problem -- sendfile() -- so, if netcat doesn't use it, it's clear why doesn't panic the kernel.

Quoted 3 lines

Have you tried a local distcc compile, but specifying the host name as it's IP address or its real name. Distcc treats 'localhost' differently, but if it sees an IP address it will use the network route. As specified

Good test.

Yeah, kernel panic in a few seconds. Using localhost instead, compile run perfectly for hours.
So it's definitely an issue related to distcc AND networking (and probably interaction between network driver and kernel).

Thank you again for your advice!

#5 lore raw
Quoted 4 lines · On Monday April 5th 2004 Marco Fais wrote:

I was not saying *this is the problem*, just noticing that all the systems that show this problem have this network card, while the other systems that are working perfectly are using other network hardware (e100 driver) :)

Yes, my conclusion was too hasty, it *is* driver related! ;-)

With hindsight we also should have tried, of course, a 'strace distccd --no-detach' in a crashing and a non-crashing situation. This would probably have shown that 'sendfile()' was the first missing system call (and therefore likely the culprit) in the crashing situation. Oh, well...

Quoted 3 lines

If you read Andrew's message, seems that distcc uses a function that trigger the problem -- sendfile() -- so, if netcat doesn't use it, it's clear why doesn't panic the kernel.

Yes, sendfile() in combination with the 8139too driver seems to be causing the trouble. Until that will hopefully be fixed, it doesn't seem easy to workaround against. At the moment it looks like it is not an easy configurable option to *not* want to use zero_copy functionality, either in the kernel, nor in distcc.

There is an '8139cp' driver too, it's supposed to be working better as well, perhaps that one might not free the pages that are to be zero_copied across the network before they are sent?! That is the real problem if I understand Andrew's mail correctly.

You might send a 'linux 8139too sendfile() panic' kind of bugreport to the '[email protected]' mailing list. That is the list where the networking gurus are supposed to be hanging out. Although IMVHO this bug is more on the kernel than on the network side. Also filing an entry to bugzilla.kernel.org might speed up someone fixing the real problem.

Easiest workaround might be to just use a customised distcc for the machines involved: just download the source from 'distcc.samba.org', do a regular './configure', and then in the generated 'src/config.h' hand edit '#undef HAVE_SENDFILE' and '#undef HAVE_SYS_SENDFILE_H'. That should stop distcc from using sendfile().

Signature · 1 line
Marco Roeland
author
#6 lore raw

Marco Roeland ha scritto:

Quoted 4 lines

There is an '8139cp' driver too, it's supposed to be working better as well, perhaps that one might not free the pages that are to be zero_copied across the network before they are sent?! That is the real problem if I understand Andrew's mail correctly.

Just tried that, unfortunately this network card isn't supported from 8139cp driver.

Quoted 3 lines

You might send a 'linux 8139too sendfile() panic' kind of bugreport to the '[email protected]' mailing list. That is the list where the networking gurus are supposed to be hanging out. Although IMVHO this bug

Andrew's messages are in CC: to the [email protected] list, so I think they're already aware of the problem.

Quoted 2 lines

is more on the kernel than on the network side. Also filing an entry to bugzilla.kernel.org might speed up someone fixing the real problem.

Ok, let see if we get a patch from this discussion, otherwise I'll file a new bugzilla entry.

Quoted 5 lines

Easiest workaround might be to just use a customised distcc for the machines involved: just download the source from 'distcc.samba.org', do a regular './configure', and then in the generated 'src/config.h' hand edit '#undef HAVE_SENDFILE' and '#undef HAVE_SYS_SENDFILE_H'. That should stop distcc from using sendfile().

Great! I'm going to test that right now, surely better than deploying customized kernels in all servers until an "official" patch comes out.

Thank you very much, Marco.

#7 lore raw
Quoted 2 lines · On Monday April 5th 2004 Marco Fais wrote:

Ok, let see if we get a patch from this discussion, otherwise I'll file a new bugzilla entry.

Perhaps the fact that you have *two* cards in each machine that crashes with the 8139too driver could be important? I have two Athlon XP 2000+ with Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ that distcc quite a lot, and never any crash. But network topology and timings might just trigger the panic in your situation and not with others...

Quoted 3 lines

[building distcc without sendfile()]
Great! I'm going to test that right now, surely better than deploying customized kernels in all servers until an "official" patch comes out.

Yeah, although that viewpoint might not be very popular on this mailing list. ;-) By the way the patch looks quite alright and applies (with an offset) to 2.6.5 as well. If you build 8139too modular, you might even make two modules, a modified one with the reduced advertised capabilities (so that the kernel assumes the card isn't zero-copy capable) under another name perhaps like 8139too-nosendfile, and the standard one. You can than at least distribute one kernel package, and only on the affected machines modprobe the bugfix module.

Anyway, first installing a distcc without sendfile() usages, can make you (distcc)build patched kernels much faster in the future. ;-)

Signature · 1 line
Marco Roeland
#8 lore raw
In reply to Marco Roeland

I Sent an email a couple os weeks ago about the same issue.

But it wasnt so documented and organized.

I can say that the card and hardware are inocents, maybe the driver, the "remote" machines that hang are using the latest fedore stable kernel.

I would need really good pointing to the procedure to debug the problem, I'm not expert in anything about kernel.

I think it's a problem in the network handling because it happens on different kernels, in different hardware. And it happens from a couple of months ago (we got a new faster network "arquitecture") and the problems seems to be triggered by fast transport of file over NTF, and distcc. I remember having a crash using scp too for some iso files.

If needed I can help track this problem, but I need some hints on the procedure

Max

Signature · 8 lines
Linux garaged 2.6.5-rc2-mm3 #1 Fri Mar 26 11:07:16 CST 2004 i686 Intel(R) 
Pentium(R) 4 CPU 2.80GHz GenuineIntel GNU/Linux
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GS/S d- s: a-29 C++(+++) ULAHI+++ P+ L++>+++ E--- W++ N* o-- K- w++++ O- M-- 
V-- PS+ PE Y-- PGP++ t- 5- X+ R tv++ b+ DI+++ D- G++ e++ h+ r+ z**
------END GEEK CODE BLOCK------
gpg-key: http://garaged.homeip.net/gpg-key.txt
#9 lore raw
In reply to Marco Fais

(linux-2.4.25)

Quoted 3 lines · Marco Fais

kernel BUG at page_alloc.c:98!

uh-oh.

Quoted 23 lines
Quoted 8 lines
Quoted 8 lines

EIP; c01372ae <__free_pages_ok+26e/280> <=====

ebx; c14b3f00 <_end+116e728/204d48a8>
ecx; c14b3f00 <_end+116e728/204d48a8>
edi; dec11340 <_end+1e8cbb68/204d48a8>
ebp; c02f1d04 <init_task_union+1d04/2000>
esp; c02f1cd4 <init_task_union+1cd4/2000>

Trace; c0135a76 <kmem_cache_free_one+f6/210>
Trace; c021667b <skb_release_data+6b/90>
Trace; c02166b4 <kfree_skbmem+14/70>
Trace; c0216816 <__kfree_skb+106/160>
Trace; c023be39 <tcp_clean_rtx_queue+139/330>
Trace; c023c385 <tcp_ack+c5/380>
Trace; c023f51c <tcp_rcv_state_process+19c/a90>
Trace; c02465a9 <tcp_v4_do_rcv+a9/130>
Trace; c0246a76 <tcp_v4_rcv+446/560>
Trace; c022dad0 <ip_local_deliver_finish+0/180>
Trace; c022dc25 <ip_local_deliver_finish+155/180>
Trace; c0222780 <nf_hook_slow+b0/170>
Trace; c022dad0 <ip_local_deliver_finish+0/180>
Trace; c022d88f <ip_local_deliver+4f/70>

Quoted 8 lines

Trace; c022dad0 <ip_local_deliver_finish+0/180> Trace; c022de3a <ip_rcv_finish+1ea/270> Trace; e08d7eab <[8139too]rtl8139_rx_interrupt+6b/3b0> Trace; c021ad14 <netif_receive_skb+c4/180> Trace; c021ae3f <process_backlog+6f/120> Trace; c021af5a <net_rx_action+6a/100>
Trace; c0121cd7 <do_softirq+97/a0>
Trace; c010a66d <do_IRQ+bd/f0>

distcc uses sendfile(). The 8139too hardware and driver are zerocopy-capable so the kernel uses zerocopy direct-from-user-pages for sendfile().

The bug is that the networking layer is releasing the final ref to user pages from softirq context. Those pages are still on the page LRU so __free_pages_ok() will take them off.

Problem is, removing these pages from the LRU requires that the pagemap_lru_lock be taken, and that lock may not be taken from interrupt
context. So we go BUG instead.

This was all discussed fairly extensively a couple of years back and I thought it ended up being fixed.

author
#10 lore raw

Andrew Morton ha scritto:

Quoted 2 lines
Quoted 1 line

kernel BUG at page_alloc.c:98!

uh-oh.

That was the same thing that I've said when I saw all the leds blinking in *all* the keyboards ... :)

Quoted 3 lines

distcc uses sendfile(). The 8139too hardware and driver are zerocopy-capable so the kernel uses zerocopy direct-from-user-pages for sendfile().

Ok. Other servers with e100 driver doesn't show the problem. This means that they're not "zerocopy-capable"?

Quoted 2 lines

This was all discussed fairly extensively a couple of years back and I thought it ended up being fixed.

There are any workarounds for this, until the problem is corrected?

Thank you very much.

#11 lore raw
Quoted 9 lines · Marco Fais

Andrew Morton ha scritto:

Quoted 2 lines
Quoted 1 line

kernel BUG at page_alloc.c:98!

uh-oh.

That was the same thing that I've said when I saw all the leds blinking in *all* the keyboards ... :)

Quoted 6 lines
Quoted 3 lines

distcc uses sendfile(). The 8139too hardware and driver are zerocopy-capable so the kernel uses zerocopy direct-from-user-pages for sendfile().

Ok. Other servers with e100 driver doesn't show the problem. This means that they're not "zerocopy-capable"?

They are. It could be a timing thing.

Quoted 4 lines
Quoted 2 lines

This was all discussed fairly extensively a couple of years back and I thought it ended up being fixed.

There are any workarounds for this, until the problem is corrected?

This will probably make it go away.

drivers/net/8139too.c +1 -1
@@ -983,7 +983,7 @@ static int __devinit rtl8139_init_one (s
 	 * through the use of skb_copy_and_csum_dev we enable these
 	 * features
 	 */
-	dev->features |= NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_HIGHDMA;
+	dev->features |= NETIF_F_SG | NETIF_F_HIGHDMA;
 
 	dev->irq = pdev->irq;
 
author
#12 lore raw

Hola Andrew!

Andrew Morton ha scritto:

drivers/net/8139too.c static int __devinit rtl8139_init_one (s · line 988

Quoted 13 lines
Quoted 1 line

There are any workarounds for this, until the problem is corrected?

This will probably make it go away.

drivers/net/8139too.c +1 -1
@@ -983,7 +983,7 @@ static int __devinit rtl8139_init_one (s
 	 * through the use of skb_copy_and_csum_dev we enable these
 	 * features
 	 */
-	dev->features |= NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_HIGHDMA;
+	dev->features |= NETIF_F_SG | NETIF_F_HIGHDMA;
 
 	dev->irq = pdev->irq;

Unfortunately, this doesn't solve the problem. Seems that the panic it's triggered a little later (1-2 minutes instead of a few seconds), but anyway I have a kernel panic every time, also with this patch.

The oops tracing looks very similar to the one I've posted on the linux-kernel list.

Thank you Andrew, bye!

#13 lore raw
In reply to Marco Fais
Quoted 15 lines · Marco Fais

Hi!

[1.] Kernel panic while using distcc

[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

Quoted 1 line

[3.] Keywords: distcc BUG page_alloc.c

Marco, Carson,

Can you please try to reproduce this distcc generated oops using 2.4.27-pre2?

Thanks!

#14 lore raw
Quoted 16 lines · --On Monday, May 03, 2004 22:07:14 -0300 Marcelo Tosatti…
Quoted 15 lines · Marco Fais

Hi!

[1.] Kernel panic while using distcc

[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

Quoted 6 lines
Quoted 1 line

[3.] Keywords: distcc BUG page_alloc.c

Marco, Carson,

Can you please try to reproduce this distcc generated oops using 2.4.27-pre2?

I'd love to. However 2.4.27-pre2 broke the tg3 driver. tg3.c contains WARN_ON(1). Sadly, WARN_ON doesn't exist in 2.4.x, so depmod correctly complains about an unresolved symbol.

I'm beginning to wonder if anyone actually builds these pre releases... I mean, I know the tg3 driver is really obscure, and only used by 2 people, but...

Signature · 1 line
Carson
#15 lore raw

On Wednesday 05 May 2004 18:25, Carson Gaspar wrote:

Hi Carson,

Quoted 6 lines

I'd love to. However 2.4.27-pre2 broke the tg3 driver. tg3.c contains WARN_ON(1). Sadly, WARN_ON doesn't exist in 2.4.x, so depmod correctly complains about an unresolved symbol.
I'm beginning to wonder if anyone actually builds these pre releases... I mean, I know the tg3 driver is really obscure, and only used by 2 people, but...

by 2 people? you have to be kidding.

Anyway, attached is 2.4 WARN_ON. Apply it and use tg3 :p

ciao, Marc

include/linux/kernel.h +7 -0
@@ -196,4 +196,11 @@ struct sysinfo {
 
 #define BUG_ON(condition) do { if (unlikely((condition)!=0)) BUG(); } while(0)
 
+#define WARN_ON(condition) do { \
+	if (unlikely((condition)!=0)) { \
+		printk("Badness in %s at %s:%d\n", __FUNCTION__, __FILE__, __LINE__); \
+		dump_stack(); \
+	} \
+} while (0)
+
 #endif /* _LINUX_KERNEL_H */
#16 lore raw
In reply to Carson Gaspar
Quoted 26 lines · Carson Gaspar

--On Monday, May 03, 2004 22:07:14 -0300 Marcelo Tosatti

Quoted 16 lines · <[email protected]> wrote:
Quoted 15 lines · Marco Fais

Hi!

[1.] Kernel panic while using distcc

[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

Quoted 6 lines
Quoted 1 line

[3.] Keywords: distcc BUG page_alloc.c

Marco, Carson,

Can you please try to reproduce this distcc generated oops using 2.4.27-pre2?

Quoted 7 lines

I'd love to. However 2.4.27-pre2 broke the tg3 driver. tg3.c contains WARN_ON(1). Sadly, WARN_ON doesn't exist in 2.4.x, so depmod correctly complains about an unresolved symbol.

I'm beginning to wonder if anyone actually builds these pre releases... I mean, I know the tg3 driver is really obscure, and only used by 2 people, but...

I just commited a fix to the BK tree.

Can you please apply this.

# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher. # This patch includes the following deltas:
# ChangeSet 1.1385 -> 1.1386
# include/linux/kernel.h 1.22 -> 1.23
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 04/05/05 [email protected] 1.1386
# [PATCH] copy WARN_ON() definition from 2.6
#
# --------------------------------------------
#

include/linux/kernel.h +7 -0
diff -Nru a/include/linux/kernel.h b/include/linux/kernel.h
@@ -196,4 +196,11 @@
 
 #define BUG_ON(condition) do { if (unlikely((condition)!=0)) BUG(); } while(0)
 
+#define WARN_ON(condition) do { \
+	if (unlikely((condition)!=0)) { \
+		printk("Badness in %s at %s:%d\n", __FUNCTION__, __FILE__, __LINE__); \
+		dump_stack(); \
+	} \
+} while (0)
+
 #endif /* _LINUX_KERNEL_H */
#17 lore raw
Quoted 20 lines · Marcelo Tosatti
Quoted 19 lines · Carson Gaspar

--On Monday, May 03, 2004 22:07:14 -0300 Marcelo Tosatti

Quoted 16 lines · <[email protected]> wrote:
Quoted 15 lines · Marco Fais

Hi!

[1.] Kernel panic while using distcc

[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

[3.] Keywords: distcc BUG page_alloc.c

Marco, Carson,

Hi Carson,

So did Andrea's fix work for you? :)

#18 lore raw

On Wednesday 19 May 2004 13:59, Marcelo Tosatti wrote:

Hi Marcelo, Carson ...

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

[1.] Kernel panic while using distcc
[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.
[3.] Keywords: distcc BUG page_alloc.c

Quoted 1 line

So did Andrea's fix work for you? :)

sorry if I did not follow this thread from the beginning, but why is distcc causing a BUG() in page_alloc.c? I use distcc since I don't know when and never had any BUG() in page_alloc with distcc, nor the specific bug at :98.

I have 7 machines in my distcc farm, and all are "remote controlled".

Could someone please clarify me? Thank you.

ciao, Marc

#19 lore raw
Quoted 18 lines · Marc-Christian Petersen

On Wednesday 19 May 2004 13:59, Marcelo Tosatti wrote:

Hi Marcelo, Carson ...

Quoted 12 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

[1.] Kernel panic while using distcc
[2.] I have 5-6 development linux systems that we use without problem under a normal development workload. Trying distcc for speeding up compilation, we have a fully reproducible kernel panic in a very short time (seconds after compilation start). The kernel panic happens *only* when the systems are "remotely controlled" (the distcc daemon is receiving source files from remote systems, compile and send back compiled objects). When compiling with distcc the local system doesn't show any kernel panic, while the same system used as a "remote compiler system" dies very quickly.
[3.] Keywords: distcc BUG page_alloc.c

Quoted 1 line

So did Andrea's fix work for you? :)

Quoted 7 lines

sorry if I did not follow this thread from the beginning, but why is distcc causing a BUG() in page_alloc.c? I use distcc since I don't know when and never had any BUG() in page_alloc with distcc, nor the specific bug at :98.

I have 7 machines in my distcc farm, and all are "remote controlled".

Could someone please clarify me? Thank you.

We try to free a page which has been sent over the network
in IRQ context, because with sendfile() its possible that such IRQ context reference is the last one on the page.

Quoting David Miller:

When vmscan.c shrinks a cache, it never tosses a page which is LRU if the page count is not 1 (after trying to toss attached buffers if any).

I think LRU used to contribute a page count, which prevented this problem, or something like that.

In fact it seems trivial to trigger the bug in question:

1) open file
2) sendfile() it over a socket
3) quickly close the file, no existing user references, only the sendfile() packet references remain to the page

When the TCP packet gets ACK'd, we explode in __free_pages_ok() as per the report.