Skip to contentSkip to conversation

Strange 'zombie' problem both in 2.4 and 2.6

DISCUSSION Nikita V. Youshchenko 2 participants 6 messages Apr 1, 2004

lore
Share & tools
lore
#0 lore raw

Hello.

Some time ago I was faced with a strange problem in 2.4 kernel. I could reproduce in only on one system - a production 2-CPU server that is used as LTSP server here and also runs tons of services and MUST be always up.

The problem is the following.
Server runs normally (and uptime may be already several weeks, but may be only several hours).
Suddenly something happens.
And process table becomes full of zombies.
Looks like any thread created by any program becomes a zombie when finished. Same programs (actually, same running processes) join()ed finished threads correctly before Something Happened. So it looks very much that Something happens inside the kernel.
Affected programs include mozilla, clamav, mysqld, licq and anything else that creates short-living threads, or at least threads that live shorter than program itself.

It looks like at some moment kernel looses the abitily to inform process that their threads are over. AKAIK, this is done by SIGCHLD? Anyway, manual sending SIGCHLD to the parent of zombies does not help.

After the problem happens, server becomes unusable (because of process table overflow) in several minutes. One time Something Else happened, and all those zombies disappeared. In all other cases a reboot was required.

If the process that created those "zombie thread" is terminated (i.e. sevice stopped), all his zombies disappear. However, after service is restarted, zombies become to appear again.

Athough I tried, I could not find any correlation between making system to this "zombie-keeping" state and anything else happenning with the system. Looks like that running java apps (with blackdown jdk) makes this happen more often, bot still no direct correlation.

The problem happened with official 2.4.23, 2.4.24 and 2.4.25 kernels, compiled from kernel.org sources.

Yestedray I was tired with this zombie problem (it arised twice during this week), and decided to upgrade server to kernel 2.6.
I installed 2.6.4 kernel from the Debain kernel-image-2.6.4-1-k7-smp package.

Unfortunately, this did not eliminate the problem: it happened today again. The difference is that when running in 2.6, most binaries use NPTL libs from /lib/i686/cmov/, and seem not to be affected by the problem (i.e. no zombies from them). However, users need to run some statically-linked binaries (without source available) that have non-NPTL libs statically linked and so still use linuxthreads; those are affected (i.e. do create zombies). So problem is not rendering server unusable (so it no longer that critical), but it still exists in the 2.6 kernel.

I can't reproduce the problem on any other host. And the affected system is a production server that is somewhat difficult to use for debugging :( It is a dual-K7 server with Tyan Tiger MPX S2466 motherboard and 2 Gb of ram. Output of 'lspci -vv' and 'cat /proc/cpuinfo' is attached. I may provide any other technical information.

I'm a seasoned unix developer and sysadmin, and have some kernel hacking experience. However, I don't work with the kernel currently, so I am not "in context of" kernel internals.
So I'm looking either for a fix :), or for some advice on what to do with this (i.e. where to look in the kernel code and what to look for).

Nikita Youshchenko,
sysadmin at lvk.cs.msu.su

Quoted 1 line

uname -a

Linux zigzag 2.6.4-1-k7-smp #1 SMP Sun Mar 14 00:19:02 EST 2004 i686 GNU/Linux

Quoted 1 line

cat /proc/cpuinfo

processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : AMD Athlon(tm) MP 2800+
stepping : 0
cpu MHz : 2133.583
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 4210.68

processor : 1
vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : AMD Athlon(tm) Processor
stepping : 0
cpu MHz : 2133.583
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mp mmxext 3dnowext 3dnow
bogomips : 4259.84

Quoted 1 line

lspci -vv

00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller (rev 20)

	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 32
	Region 0: Memory at f4000000 (32-bit, prefetchable) [size=64M]
	Region 1: Memory at f0600000 (32-bit, prefetchable) [size=4K]
	Region 2: I/O ports at 1020 [disabled] [size=4]
	Capabilities: <available only to root>

00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] AGP Bridge (prog-if 00 [Normal decode])

	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 99
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: f0100000-f01fffff
	Prefetchable memory behind bridge: f8000000-fbffffff
	BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-

00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ISA (rev 05)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0

00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE (rev 04) (prog-if 8a [Master SecP PriP])
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Region 4: I/O ports at f000 [size=16]

00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI (rev 03)
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-

00:08.0 RAID bus controller: Promise Technology, Inc. PDC20271 (rev 02) (prog-if 85)

	Subsystem: Promise Technology, Inc.: Unknown device 4d68
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 64 (1000ns min, 4500ns max), Cache Line Size: 0x10 (64 bytes)
	Interrupt: pin A routed to IRQ 20
	Region 0: I/O ports at 1038 [size=8]
	Region 1: I/O ports at 1030 [size=4]
	Region 2: I/O ports at 1028 [size=8]
	Region 3: I/O ports at 1024 [size=4]
	Region 4: I/O ports at 1010 [size=16]
	Region 5: Memory at f0000000 (32-bit, non-prefetchable) [size=64K]
	Expansion ROM at <unassigned> [disabled] [size=64K]
	Capabilities: <available only to root>

00:10.0 PCI bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] PCI (rev 05) (prog-if 00 [Normal decode])

	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 64
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=168
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: f0200000-f03fffff
	BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-

01:05.0 VGA compatible controller: ATI Technologies Inc Rage 128 RF/SG AGP (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Magnum/Xpert128/X99/Xpert2000
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B+
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 66 (2000ns min), Cache Line Size: 0x10 (64 bytes)
Interrupt: pin A routed to IRQ 18
Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
Region 1: I/O ports at 2000 [size=256]
Region 2: Memory at f0100000 (32-bit, non-prefetchable) [size=16K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: <available only to root>

02:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-768 [Opus] USB (rev 07) (prog-if 10 [OHCI])
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] USB
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR+
Latency: 64 (20000ns max)
Interrupt: pin D routed to IRQ 19
Region 0: Memory at f0300000 (32-bit, non-prefetchable) [size=4K]

02:04.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
Subsystem: Intel Corp. EtherExpress PRO/100+ Management Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (2000ns min, 14000ns max), Cache Line Size: 0x10 (64 bytes)
Interrupt: pin A routed to IRQ 16
Region 0: Memory at f0301000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 3480 [size=64]
Region 2: Memory at f0200000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at <unassigned> [disabled] [size=1M]
Capabilities: <available only to root>

02:06.0 SCSI storage controller: Tekram Technology Co.,Ltd. TRM-S1040 (rev 01)
Subsystem: Tekram Technology Co.,Ltd. TRM-S1040
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64, Cache Line Size: 0x10 (64 bytes)
Interrupt: pin A routed to IRQ 18
Region 0: I/O ports at 3000 [size=256]
Region 1: Memory at f0302000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: <available only to root>

02:07.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02)
Subsystem: Ensoniq Creative Sound Blaster AudioPCI128
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort+ <MAbort- >SERR- <PERR-
Latency: 64 (3000ns min, 32000ns max)
Interrupt: pin A routed to IRQ 19
Region 0: I/O ports at 34c0 [size=64]
Capabilities: <available only to root>

02:08.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado] (rev 78)
Subsystem: Tyan Computer: Unknown device 2466
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 80 (2500ns min, 2500ns max), Cache Line Size: 0x10 (64 bytes)
Interrupt: pin A routed to IRQ 19
Region 0: I/O ports at 3400 [size=128]
Region 1: Memory at f0303000 (32-bit, non-prefetchable) [size=128]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: <available only to root>

#1 lore raw
Quoted 7 lines · Nikita V. Youshchenko

Hello.

Some time ago I was faced with a strange problem in 2.4 kernel. I could reproduce in only on one system - a production 2-CPU server that is used as LTSP server here and also runs tons of services and MUST be always up.

Quoted 12 lines

The problem is the following.
Server runs normally (and uptime may be already several weeks, but may be only several hours).
Suddenly something happens.
And process table becomes full of zombies.
Looks like any thread created by any program becomes a zombie when finished. Same programs (actually, same running processes) join()ed finished threads correctly before Something Happened. So it looks very much that Something happens inside the kernel.
Affected programs include mozilla, clamav, mysqld, licq and anything else that creates short-living threads, or at least threads that live shorter than program itself.

How does ps -AH e looks like?

Quoted 3 lines

It looks like at some moment kernel looses the abitily to inform process that their threads are over. AKAIK, this is done by SIGCHLD? Anyway, manual sending SIGCHLD to the parent of zombies does not help.

Did you try stracing parent process? It can receive SIGCHLD but ignore/mishandle it.

Quoted 7 lines

After the problem happens, server becomes unusable (because of process table overflow) in several minutes. One time Something Else happened, and all those zombies disappeared. In all other cases a reboot was required.

If the process that created those "zombie thread" is terminated (i.e. sevice stopped), all his zombies disappear. However, after service is restarted, zombies become to appear again.

Probably they get reparented to init and it wait()'s for them, ending their afterlife. So SIGCHLD works (at least in this case).

Quoted 13 lines

Athough I tried, I could not find any correlation between making system to this "zombie-keeping" state and anything else happenning with the system. Looks like that running java apps (with blackdown jdk) makes this happen more often, bot still no direct correlation.

The problem happened with official 2.4.23, 2.4.24 and 2.4.25 kernels, compiled from kernel.org sources.

Yestedray I was tired with this zombie problem (it arised twice during this week), and decided to upgrade server to kernel 2.6.
I installed 2.6.4 kernel from the Debain kernel-image-2.6.4-1-k7-smp package.

Quoted 8 lines

Unfortunately, this did not eliminate the problem: it happened today again. The difference is that when running in 2.6, most binaries use NPTL libs from /lib/i686/cmov/, and seem not to be affected by the problem (i.e. no zombies from them). However, users need to run some statically-linked binaries (without source available) that have non-NPTL libs statically linked and so still use linuxthreads; those are affected (i.e. do create zombies). So problem is not rendering server unusable (so it no longer that critical), but it still exists in the 2.6 kernel.

Sounds like userspace problem in threading libraries. What version of glibc/linuxthreads was in use before? Maybe post your report on linuxthreads mailing list.

Quoted 11 lines

I can't reproduce the problem on any other host. And the affected system is a production server that is somewhat difficult to use for debugging :( It is a dual-K7 server with Tyan Tiger MPX S2466 motherboard and 2 Gb of ram. Output of 'lspci -vv' and 'cat /proc/cpuinfo' is attached. I may provide any other technical information.

I'm a seasoned unix developer and sysadmin, and have some kernel hacking experience. However, I don't work with the kernel currently, so I am not "in context of" kernel internals.
So I'm looking either for a fix :), or for some advice on what to do with this (i.e. where to look in the kernel code and what to look for).

Signature · 1 line
vda
#2 lore raw
Quoted 18 lines
Quoted 5 lines

Some time ago I was faced with a strange problem in 2.4 kernel. I could reproduce in only on one system - a production 2-CPU server that is used as LTSP server here and also runs tons of services and MUST be always up.

Quoted 12 lines

The problem is the following.
Server runs normally (and uptime may be already several weeks, but may be only several hours).
Suddenly something happens.
And process table becomes full of zombies.
Looks like any thread created by any program becomes a zombie when finished. Same programs (actually, same running processes) join()ed finished threads correctly before Something Happened. So it looks very much that Something happens inside the kernel.
Affected programs include mozilla, clamav, mysqld, licq and anything else that creates short-living threads, or at least threads that live shorter than program itself.

Quoted 1 line

How does ps -AH e looks like?

See output of "ps -lax" in attachment.

Quoted 6 lines
Quoted 3 lines

It looks like at some moment kernel looses the abitily to inform process that their threads are over. AKAIK, this is done by SIGCHLD? Anyway, manual sending SIGCHLD to the parent of zombies does not help.

Did you try stracing parent process? It can receive SIGCHLD but ignore/mishandle it.

I tried to use strace -f, so all threads exist in the output. No signals arrive, expect those send manually by kill().
Stracing same binary on another host shows that SIGRT_1 arrives to the parent.
I may send the strace logs, but they are somewhat large.
So kernel really stops devivering signals.

As far as I understand, in case of threads SIGRT_1 is used instead of SIGCHLD.
So I tried to send SIGRT_1 to the parent manually. And zombies disappeared! However, new zombies appear soon. They may still be removed by manual SIGRT_1, but it is not a solution for a kernel bug :).

Quoted 11 lines
Quoted 8 lines

After the problem happens, server becomes unusable (because of process table overflow) in several minutes. One time Something Else happened, and all those zombies disappeared. In all other cases a reboot was required.

If the process that created those "zombie thread" is terminated (i.e. sevice stopped), all his zombies disappear. However, after service is restarted, zombies become to appear again.

Probably they get reparented to init and it wait()'s for them, ending their afterlife. So SIGCHLD works (at least in this case).

Seems that signal passing works only after reparenting zombies.

Quoted 23 lines
Quoted 13 lines

Athough I tried, I could not find any correlation between making system to this "zombie-keeping" state and anything else happenning with the system. Looks like that running java apps (with blackdown jdk) makes this happen more often, bot still no direct correlation.

The problem happened with official 2.4.23, 2.4.24 and 2.4.25 kernels, compiled from kernel.org sources.

Yestedray I was tired with this zombie problem (it arised twice during this week), and decided to upgrade server to kernel 2.6.
I installed 2.6.4 kernel from the Debain kernel-image-2.6.4-1-k7-smp package.

Quoted 9 lines

Unfortunately, this did not eliminate the problem: it happened today again. The difference is that when running in 2.6, most binaries use NPTL libs from /lib/i686/cmov/, and seem not to be affected by the problem (i.e. no zombies from them). However, users need to run some statically-linked binaries (without source available) that have non-NPTL libs statically linked and so still use linuxthreads; those are affected (i.e. do create zombies). So problem is not rendering server unusable (so it no longer that critical), but it still exists in the 2.6 kernel.

Quoted 3 lines

Sounds like userspace problem in threading libraries. What version of glibc/linuxthreads was in use before? Maybe post your report on linuxthreads mailing list.

I doubt it is a userspace problem.
It happens with the same userspace libs and binaries (or even same running processes) with which it did not happen sometime ago.
It happens at the same moment with different processes running from different accounts.
Restarting processes doesn't help.
It is not reprodusable on other hosts.
Manual signal send (kill -33 <parentpid>) removes already existing zombies. I can hardly imagine a userspace problem that behaves like this.

Nikita

0   206 18992     1  16   0 50512 1600 schedu S    ?          0:00 /space/p2/n/donkey/donkey0.50.1 - ! -g -l
1   206 18993 18992  16   0 50512 1600 schedu S    ?          0:00 /space/p2/n/donkey/donkey0.50.1 - ! -g -l
1   206 18994 18993  15   0 50512 1600 schedu S    ?          0:00 /space/p2/n/donkey/donkey0.50.1 - ! -g -l
1   206 18995 18993  18   0 50512 1600 io_sch D    ?          0:31 /space/p2/n/donkey/donkey0.50.1 - ! -g -l
1   206 18996 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19021 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19068 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19069 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19070 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19071 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19072 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19073 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19074 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19077 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19078 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19090 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19091 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19092 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19093 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19094 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19095 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19096 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19107 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19108 18993  15   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
1   206 19111 18993  16   0     0    0 exit   Z    ?          0:00 [donkey0.50.1] <defunct>
0   206 19123 19080  15   0  2940  748 pipe_w S    pts/254    0:00 grep donkey
#3 lore raw
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

It looks like at some moment kernel looses the abitily to inform process that their threads are over. AKAIK, this is done by SIGCHLD? Anyway, manual sending SIGCHLD to the parent of zombies does not help.

Did you try stracing parent process? It can receive SIGCHLD but ignore/mishandle it.

Quoted 6 lines

I tried to use strace -f, so all threads exist in the output. No signals arrive, expect those send manually by kill().
Stracing same binary on another host shows that SIGRT_1 arrives to the parent.
I may send the strace logs, but they are somewhat large.
So kernel really stops devivering signals.

Post reasonably small pieces of them.

Quoted 5 lines

As far as I understand, in case of threads SIGRT_1 is used instead of SIGCHLD.
So I tried to send SIGRT_1 to the parent manually. And zombies disappeared! However, new zombies appear soon. They may still be removed by manual SIGRT_1, but it is not a solution for a kernel bug :).

Maybe. Maybe not. I am no expert, I'd try to learn out how SIGRT_1 is generated in normal case (I suppose kernel does not distinguish between threads and processes, maybe it's done by threading libs?)

Quoted 19 lines
Quoted 2 lines

Probably they get reparented to init and it wait()'s for them, ending their afterlife. So SIGCHLD works (at least in this case).

Seems that signal passing works only after reparenting zombies.

Quoted 10 lines
Quoted 9 lines

Unfortunately, this did not eliminate the problem: it happened today again. The difference is that when running in 2.6, most binaries use NPTL libs from /lib/i686/cmov/, and seem not to be affected by the problem (i.e. no zombies from them). However, users need to run some statically-linked binaries (without source available) that have non-NPTL libs statically linked and so still use linuxthreads; those are affected (i.e. do create zombies). So problem is not rendering server unusable (so it no longer that critical), but it still exists in the 2.6 kernel.

Quoted 3 lines

Sounds like userspace problem in threading libraries. What version of glibc/linuxthreads was in use before? Maybe post your report on linuxthreads mailing list.

Quoted 9 lines

I doubt it is a userspace problem.
It happens with the same userspace libs and binaries (or even same running processes) with which it did not happen sometime ago.
It happens at the same moment with different processes running from different accounts.
Restarting processes doesn't help.
It is not reprodusable on other hosts.
Manual signal send (kill -33 <parentpid>) removes already existing zombies. I can hardly imagine a userspace problem that behaves like this.

I won't argue. One thing is clear: not enough info at this time :(

Try to instrument (printk("...")) parts of kernel responsible for handling exit() etc.
--
vda

Quoted 2 lines

Nikita

#4 lore raw
Quoted 10 lines
Quoted 6 lines

As far as I understand, in case of threads SIGRT_1 is used instead of SIGCHLD.
So I tried to send SIGRT_1 to the parent manually. And zombies disappeared! However, new zombies appear soon. They may still be removed by manual SIGRT_1, but it is not a solution for a kernel bug :).

Maybe. Maybe not. I am no expert, I'd try to learn out how SIGRT_1 is generated in normal case (I suppose kernel does not distinguish between threads and processes, maybe it's done by threading libs?)

I've looked at the kernel source.
This is what I found.

- looks like do_notify_parent() from kernel/signal.c is called to notify parent about child termination.

- do_notify_parent() calls __group_send_sig_info() to send the signal, and does not check the return code. However, __group_send_sig_info() may fail.

- __group_send_sig_info() calls send_signal()

- send_signal() contains the following code:

	struct sigqueue * q = NULL;
...
	if (atomic_read(&nr_queued_signals) < max_queued_signals)
		q = kmem_cache_alloc(sigqueue_cachep, GFP_ATOMIC);
	if (q) {
...
	} else {
		if (sig >= SIGRTMIN && info && (unsigned long)info != 1
		   && info->si_code != SI_USER)
			return -EAGAIN;

...

SIGRT_1 = 33, 33 is greater than SIGRTMIN, info is definitely not 0 or 1, and info->si_code is definitly not SI_USER on the path related to parent process notification.

nr_queued_signals and sigqueue_cachep seem to be local for kernel/signal.c file, and code is organized such that nr_queued_signals shows exactly how many elements are allocated in sigqueue_cachep. max_queued_signals equals to 1024, so it is not allowed to allocate more than 1024 elements from sigqueue_cachep.

sigqueue_cachep is initialized in signals_init():

	sigqueue_cachep =
		kmem_cache_create("sigqueue",
				  sizeof(struct sigqueue),
				  __alignof__(struct sigqueue),
				  0, NULL, NULL);

So I looked into /proc/slabinfo on the server running "zombie-loving" kernel, and found the following line:
nikita@zigzag:/proc> grep sigqueue slabinfo
sigqueue 1024 1107 144 27 1 : tunables 120 60 8 : slabdata 41 41 0

As far as I understand, the first number in this output is the number of elements allocated from "sigqueue" cache. That is, all 1024 elements are allocated!

So looks like 'atomic_read(&nr_queued_signals) < max_queued_signals' is false, so 'q' is not allocated, and send_signal() returns -EAGAIN while trying to send SIGRT_1 to the parent process. This error code is passed from __group_send_sig_info() to do_notify_parent(), and just ignored there. So signal is not delivered, and dying process is left in zombie state.

So "something" that happens in the kernel that makes it "zombie-lover" is sigqueue overflow.

Another question is why this ever happens on my server ...

#5 lore raw
Quoted 6 lines · Nikita V. Youshchenko
Quoted 6 lines
Quoted 6 lines

As far as I understand, in case of threads SIGRT_1 is used instead of SIGCHLD.
So I tried to send SIGRT_1 to the parent manually. And zombies disappeared! However, new zombies appear soon. They may still be removed by manual SIGRT_1, but it is not a solution for a kernel bug

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

:).

Maybe. Maybe not. I am no expert, I'd try to learn out how SIGRT_1 is generated in normal case (I suppose kernel does not distinguish between threads and processes, maybe it's done by threading libs?)

I've looked at the kernel source.
This is what I found.

Good! :)

Quoted 48 lines

- looks like do_notify_parent() from kernel/signal.c is called to notify parent about child termination.

- do_notify_parent() calls __group_send_sig_info() to send the signal, and does not check the return code. However, __group_send_sig_info() may fail.

- __group_send_sig_info() calls send_signal()

- send_signal() contains the following code:

	struct sigqueue * q = NULL;
...
	if (atomic_read(&nr_queued_signals) < max_queued_signals)
		q = kmem_cache_alloc(sigqueue_cachep, GFP_ATOMIC);
	if (q) {
...
	} else {
		if (sig >= SIGRTMIN && info && (unsigned long)info != 1
		   && info->si_code != SI_USER)
			return -EAGAIN;

...

SIGRT_1 = 33, 33 is greater than SIGRTMIN, info is definitely not 0 or 1, and info->si_code is definitly not SI_USER on the path related to parent process notification.

nr_queued_signals and sigqueue_cachep seem to be local for kernel/signal.c file, and code is organized such that nr_queued_signals shows exactly how many elements are allocated in sigqueue_cachep. max_queued_signals equals to 1024, so it is not allowed to allocate more than 1024 elements from sigqueue_cachep.

sigqueue_cachep is initialized in signals_init():

	sigqueue_cachep =
		kmem_cache_create("sigqueue",
				  sizeof(struct sigqueue),
				  __alignof__(struct sigqueue),
				  0, NULL, NULL);

So I looked into /proc/slabinfo on the server running "zombie-loving" kernel, and found the following line:
nikita@zigzag:/proc> grep sigqueue slabinfo
sigqueue 1024 1107 144 27 1 : tunables 120 60 8 : slabdata 41 41 0

As far as I understand, the first number in this output is the number of elements allocated from "sigqueue" cache. That is, all 1024 elements are allocated!

Quoted 5 lines

So looks like 'atomic_read(&nr_queued_signals) < max_queued_signals' is false, so 'q' is not allocated, and send_signal() returns -EAGAIN while trying to send SIGRT_1 to the parent process. This error code is passed from __group_send_sig_info() to do_notify_parent(), and just ignored there.

Hmmm what it can do there? Maybe only printk(). The question is why sigqueue gets so big and does not shrink.

Quoted 5 lines

So signal is not delivered, and dying process is left in zombie state.

So "something" that happens in the kernel that makes it "zombie-lover" is sigqueue overflow.

You found an explanation why there are zombies. Now, why it starts to happen? Why does it persists? There must be some code which shrinks sigqueue. It does not seem to work right.
--
vda