Skip to contentSkip to conversation

process stuck in release_task() call

2.4.28-rc1 Andrey J. Melnikoff (TEMHOTA) 4 participants 6 messages Nov 9, 2004 – Nov 30, 2004 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Hello!

With 2.4.28-pre3 and 2.4.28-rc1 i see strange situation - sendmail some times get stuck into release_task() call.

System - Tyan Tiger MPX, dual Athlon MP 2800+ with 1Gb memory.

--- SysRq-T output ---
ksymoops 2.4.9 on i686 2.4.28-rc1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.28-rc1/ (default)
-m /boot/System.map-2.4.28-rc1 (default)

Reading Oops report from the terminal
sendmail S C012073D 0 15814 1 32701 14365 (NOTLB)
Using defaults from ksymoops -t elf32-i386 -a i386

Call Trace:    [<c012073d>] [<c0106582>] [<c0107717>]
sendmail      Z 00000000     4 30459  15814         30669       (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c011547d>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 30669  15814         30707 30459 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     4 30707  15814         31549 30669 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000  2624 31549  15814         31708 30707 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 31708  15814         32269 31549 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32269  15814         32352 31708 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000    20 32352  15814         32403 32269 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32403  15814         32413 32352 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000   624 32413  15814         32468 32403 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32468  15814         32473 32413 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32473  15814         32482 32468 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32482  15814         32499 32473 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]

..... many sendmail zombies ......

Warning (Oops_read): Code line not seen, dumping what data is available

Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; c012073d <release_task+1fd/230> <=====

Trace; c012073d <release_task+1fd/230>
Trace; c0106582 <sys_rt_sigsuspend+122/160>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; 00000000 Before first symbol

Trace; c0120f53 <exit_notify+103/3c0> Trace; c0121600 <do_exit+3f0/4e0>
Trace; c011547d <smp_apic_timer_interrupt+12d/130> Trace; c0121725 <sys_exit+15/20>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; 00000000 Before first symbol

Trace; c0120f53 <exit_notify+103/3c0>
Trace; c0121600 <do_exit+3f0/4e0>
Trace; c0121725 <sys_exit+15/20>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

.... same trace with other zombies ......

disassemble show other result - process stuck into free_pages() call:

c0120540 <release_task>:
c0120540: 55 push %ebp
....

c0120736:       89 d8                   mov    %ebx,%eax
c0120738:       e8 73 dd 01 00          call   c013e4b0 <free_pages> <= here
c012073d:       83 c4 10                add    $0x10,%esp
c0120740:       5b                      pop    %ebx
c0120741:       5e                      pop    %esi
c0120742:       c9                      leave

....

Any hints ?

Signature · 2 lines
 Best regards, TEMHOTA-RIPN aka MJA13-RIPE
 System Administrator. mailto:[email protected]
#1 lore raw

Hi Andrey,

Quoted 76 lines · Andrey J. Melnikoff (TEMHOTA)

Hello!

With 2.4.28-pre3 and 2.4.28-rc1 i see strange situation - sendmail some times get stuck into release_task() call.

System - Tyan Tiger MPX, dual Athlon MP 2800+ with 1Gb memory.

--- SysRq-T output ---
ksymoops 2.4.9 on i686 2.4.28-rc1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.28-rc1/ (default)
-m /boot/System.map-2.4.28-rc1 (default)

Reading Oops report from the terminal
sendmail S C012073D 0 15814 1 32701 14365 (NOTLB)
Using defaults from ksymoops -t elf32-i386 -a i386

Call Trace:    [<c012073d>] [<c0106582>] [<c0107717>]
sendmail      Z 00000000     4 30459  15814         30669       (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c011547d>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 30669  15814         30707 30459 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     4 30707  15814         31549 30669 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000  2624 31549  15814         31708 30707 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 31708  15814         32269 31549 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32269  15814         32352 31708 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000    20 32352  15814         32403 32269 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32403  15814         32413 32352 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000   624 32413  15814         32468 32403 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32468  15814         32473 32413 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32473  15814         32482 32468 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]
sendmail      Z 00000000     0 32482  15814         32499 32473 (L-TLB)
Call Trace:    [<c0120f53>] [<c0121600>] [<c0121725>] [<c0107717>]

..... many sendmail zombies ......

Warning (Oops_read): Code line not seen, dumping what data is available

Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; c012073d <release_task+1fd/230> <=====

Trace; c012073d <release_task+1fd/230>
Trace; c0106582 <sys_rt_sigsuspend+122/160>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; 00000000 Before first symbol

Trace; c0120f53 <exit_notify+103/3c0> Trace; c0121600 <do_exit+3f0/4e0>
Trace; c011547d <smp_apic_timer_interrupt+12d/130> Trace; c0121725 <sys_exit+15/20>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

Quoted 1 line
Quoted 1 line

EIP; 00000000 Before first symbol

Trace; c0120f53 <exit_notify+103/3c0>
Trace; c0121600 <do_exit+3f0/4e0>
Trace; c0121725 <sys_exit+15/20>
Trace; c0107717 <system_call+33/38>
Proc; sendmail

.... same trace with other zombies ......

Quoted 7 lines

disassemble show other result - process stuck into free_pages() call:

c0120540 <release_task>:
c0120540: 55 push %ebp
....

c0120736:       89 d8                   mov    %ebx,%eax
c0120738:       e8 73 dd 01 00          call   c013e4b0 <free_pages> <= here

is this release_task+1fd? Can you send me the full disassemble of release_task?

It can't be blocked here, its a "call" instruction.

free_pages can't block either. Odd.

It is reproducible?

First wild guess (because I haven't got much of a clue really) would be to revert the mm/page_alloc.c __free_pages() "fastcall" gcc3.4 change.

#2 lore raw

Hi Marcelo,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

EIP; c012073d <release_task+1fd/230> <=====

(...)

Quoted 7 lines
Quoted 5 lines

c0120540 <release_task>:
c0120540: 55 push %ebp
....

c0120736:       89 d8                   mov    %ebx,%eax
c0120738:       e8 73 dd 01 00          call   c013e4b0 <free_pages> <= here

is this release_task+1fd? Can you send me the full disassemble of release_task?

Yes it is because the next instruction after call will be at c0120738+5 = c012073d = release_task+1fd. (the return address on the stack is the address of the next instruction after the call).

Quoted 1 line

It can't be blocked here, its a "call" instruction.

Seems rather strange indeed ! Perhaps this is not the disassembled function of the *running* kernel ? it would be good to disassemble vmlinux and ensure that it is exactly the one currently running. I too have already lost lots of time searching a wrong bug because I disassembled the wrong kernel, so I'm certain it can happen even when we're very careful :-(

Quoted 1 line

free_pages can't block either. Odd.

Marcelo, I have two questions for my own understanding :
- free_pages does spin_lock(&zone->lock) around the while() loop. Considering that someone else could hold the lock (bug, etc...), it could block here. But my feeling is that if such a lock were kept held, the system would be totally frozen because everything which would want to free memory would get stuck (even a process exit). Am I right ?

- would it enhance performance a bit to put a bunch of 'unlikely()' in all the ifs which end in BUG(), especially inside the loop ?

Regards,
Willy

#3 lore raw
Quoted 10 lines · Willy Tarreau

Hi Marcelo,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

EIP; c012073d <release_task+1fd/230> <=====

(...)

Quoted 6 lines
Quoted 5 lines

c0120540 <release_task>:
c0120540: 55 push %ebp
....

c0120736:       89 d8                   mov    %ebx,%eax
c0120738:       e8 73 dd 01 00          call   c013e4b0 <free_pages> <= here
Quoted 5 lines
Quoted 1 line

is this release_task+1fd? Can you send me the full disassemble of release_task?

Yes it is because the next instruction after call will be at c0120738+5 = c012073d = release_task+1fd. (the return address on the stack is the address of the next instruction after the call).

OK.

Quoted 8 lines
Quoted 1 line

It can't be blocked here, its a "call" instruction.

Seems rather strange indeed ! Perhaps this is not the disassembled function of the *running* kernel ? it would be good to disassemble vmlinux and ensure that it is exactly the one currently running. I too have already lost lots of time searching a wrong bug because I disassembled the wrong kernel, so I'm certain it can happen even when we're very careful :-(

Quoted 8 lines
Quoted 1 line

free_pages can't block either. Odd.

Marcelo, I have two questions for my own understanding :
- free_pages does spin_lock(&zone->lock) around the while() loop. Considering that someone else could hold the lock (bug, etc...), it could block here. But my feeling is that if such a lock were kept held, the system would be totally frozen because everything which would want to free memory would get stuck (even a process exit). Am I right ?

Right, the system will be totally frozen spinning on the lock.

Quoted 2 lines

- would it enhance performance a bit to put a bunch of 'unlikely()' in all the ifs which end in BUG(), especially inside the loop ?

Yes, it should generate better code.

Try it and see how the generated code differs from the original without unlikely.

I'm not aware of the internals of unlikely however, so I can't explain how it works in details... the GCC documentation should do it. :)

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Hello Marcelo, Willy!
On Tue, Nov 16, 2004 at 08:06:42AM -0200, Marcelo Tosatti wrote next:

Quoted 4 lines · Andrey J. Melnikoff (TEMHOTA)

Andrey,

I do not have much of a clue of what is going on here.

show_trace() has made a fool of me and I started to ask silly questions :)

Quoted 1 line

Can you try 2.4.27 please?

Ok, i'm tested 2.4.25 - same result. But this is complete userland problem.

There two problem:

First - show_trace() give incorrect traces. it strat unwind stack from address in tsk->thread.esp', but it should use address saved in regs->ebp'
- this make more accuracy stack trace.

Second - strange libpthreads problem.
libpthreads always install own sa_restorer helper, and when first signal arrived - call signal handler and if (when process in signal handler) arrived new signal - lipthreads start play with rt_sigprocmask() and rt_sigsuspend() syscalls inside own sa_restorer helper. woops - infinity loop inside libpthreads.

Signature · 2 lines
 Best regards, TEMHOTA-RIPN aka MJA13-RIPE
 System Administrator. mailto:[email protected]
#7 lore raw
In reply to Willy Tarreau

Hello Willy Tarreau!

Quoted 10 lines · In article <[email protected]> you wrote:
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

EIP; c012073d <release_task+1fd/230> <=====

(...)

Quoted 7 lines
Quoted 5 lines

c0120540 <release_task>:
c0120540: 55 push %ebp
....

c0120736:       89 d8                   mov    %ebx,%eax
c0120738:       e8 73 dd 01 00          call   c013e4b0 <free_pages> <= here

is this release_task+1fd? Can you send me the full disassemble of release_task?

Quoted 8 lines

Yes it is because the next instruction after call will be at c0120738+5 = c012073d = release_task+1fd. (the return address on the stack is the address of the next instruction after the call).

Quoted 1 line

It can't be blocked here, its a "call" instruction.

Seems rather strange indeed ! Perhaps this is not the disassembled function of the *running* kernel ?

This is also strange for me. Stack trace should point into __free_pages_ok() address space. Only this function work with lock's.

Quoted 2 lines

it would be good to disassemble vmlinux and ensure that it is exactly the one currently running.

This is code from vmlinux-2.4.28-rc1.

Quoted 3 lines

I too have already lost lots of time searching a wrong bug because I disassembled the wrong kernel, so I'm certain it can happen even when we're very careful :-(

[skipp]

PS: Please, keep CC: to me.