<=2.4.17 deadlock (RedHat 7.2, SMP, ext3 related?) (2)
Frank,
could you open your vmlinux file with gdb, and figure out where the 2 stext_lock references lead to?
stext_lock+1c82 just means that it spins trying to acquire a spinlock, but which one?
IIRC
gdb vmlinux
$x/30i 0xc03fe0a0
$x/30i 0xc0403050should be enough.
Quoted 1 line
Quoted 1 line
EIP;
c03fe0ba<stext_lock+1c82/e9ac> <=====
Trace; c0140134 <chrdev_open+28/124>
Trace; c013eba2 <dentry_open+e6/194>
Trace; c013eab2 <filp_open+52/5c>
Trace; c013ee23 <sys_open+4b/140>
Trace; c01076cb <system_call+33/38>
Probably this cpu is spinning in get_chrfops(), trying to acquire the big kernel lock.
Quoted 1 line
Quoted 1 line
EIP;
c0403068<stext_lock+6c30/e9ac> <=====
Trace; c016d320 <ext3_delete_inode+0/270>
Trace; c028ee04 <ppp_ioctl+33c/d84>
Trace; c0159e01 <iput+2c5/2f0>
Trace; c0156e68 <d_delete+dc/20c>
Trace; c013f063 <filp_close+133/140>
Trace; c0150bc6 <sys_ioctl+24a/2e4>
Trace; c028eac8 <ppp_ioctl+0/d84>Trace; c01076cb <system_call+33/38>
I think the call chain issystem_call
-> calls sys_ioctl.
acquires the big kernel lock.
puts ppp_ioctl on the stack (+0: it's a function pointer, not a return value.)
-> calls ppp_ioctl
and that one locks up.
The references to close/iput/delete_inode are just stale stack values from a previous syscall.
Please check where stext_lock+6c30 leads: I think the deadlock is somewhere within ppp_ioctl, and then the system locks up because the big kernel lock is blocked.
--
Manfred
Quoted 10 lines · Manfred Spraul
Frank,
could you open your vmlinux file with gdb, and figure out where the 2stext_lockreferences lead to?
stext_lock+1c82 just means that it spins trying to acquire a spinlock, but which one?
IIRC
gdb vmlinux$x/30i 0xc03fe0a0 $x/30i 0xc0403050should be enough.
(gdb) x/30i 0xc03fe0a0
0xc03fe0a0 <stext_lock+7272>: lea 0xc04e79c0,%eax 0xc03fe0a6 <stext_lock+7278>: call 0xc01062f4 <__read_lock_failed> 0xc03fe0ab <stext_lock+7283>: pop %eax 0xc03fe0ac <stext_lock+7284>: jmp 0xc013fd3e <get_chrfops+78> 0xc03fe0b1 <stext_lock+7289>: cmpb $0x0,0xc05dd400 0xc03fe0b8 <stext_lock+7296>: repz nop 0xc03fe0ba <stext_lock+7298>: jle 0xc03fe0b1 <stext_lock+7289> 0xc03fe0bc <stext_lock+7300>: jmp 0xc013fde0 <get_chrfops+240> 0xc03fe0c1 <stext_lock+7305>: push %eax 0xc03fe0c2 <stext_lock+7306>: lea 0xc04e79c0,%eax 0xc03fe0c8 <stext_lock+7312>: call 0xc01062f4 <__read_lock_failed> 0xc03fe0cd <stext_lock+7317>: pop %eax 0xc03fe0ce <stext_lock+7318>: jmp 0xc013feee <get_chrfops+510> 0xc03fe0d3 <stext_lock+7323>: push %eax 0xc03fe0d4 <stext_lock+7324>: lea 0xc04e79c0,%eax 0xc03fe0da <stext_lock+7330>: call 0xc01062d4 <__write_lock_failed> 0xc03fe0df <stext_lock+7335>: pop %eax 0xc03fe0e0 <stext_lock+7336>: jmp 0xc013ff78 <register_chrdev+68> 0xc03fe0e5 <stext_lock+7341>: push %eax 0xc03fe0e6 <stext_lock+7342>: lea 0xc04e79c0,%eax 0xc03fe0ec <stext_lock+7348>: call 0xc01062d4 <__write_lock_failed> 0xc03fe0f1 <stext_lock+7353>: pop %eax 0xc03fe0f2 <stext_lock+7354>: jmp 0xc013fff3 <register_chrdev+191> 0xc03fe0f7 <stext_lock+7359>: push %eax 0xc03fe0f8 <stext_lock+7360>: lea 0xc04e79c0,%eax 0xc03fe0fe <stext_lock+7366>: call 0xc01062d4 <__write_lock_failed> 0xc03fe103 <stext_lock+7371>: pop %eax 0xc03fe104 <stext_lock+7372>: jmp 0xc01400a3 <unregister_chrdev+79> 0xc03fe109 <stext_lock+7377>: cmpb $0x0,0xc05dd400 0xc03fe110 <stext_lock+7384>: repz nop (gdb) x/30i 0xc0403050 0xc0403050 <stext_lock+27672>: nop 0xc0403051 <stext_lock+27673>: jle 0xc0403048 <stext_lock+27664> 0xc0403053 <stext_lock+27675>: jmp 0xc0292971 <find_compressor+53> 0xc0403058 <stext_lock+27680>: cmpb $0x0,0xc057d540 0xc040305f <stext_lock+27687>: repz nop 0xc0403061 <stext_lock+27689>: jle 0xc0403058 <stext_lock+27680> 0xc0403063 <stext_lock+27691>: jmp 0xc0292ac0 <ppp_create_interface+72> 0xc0403068 <stext_lock+27696>: cmpb $0x0,0xc057d540 0xc040306f <stext_lock+27703>: repz nop 0xc0403071 <stext_lock+27705>: jle 0xc0403068 <stext_lock+27696> 0xc0403073 <stext_lock+27707>: jmp 0xc0292df0 <ppp_destroy_interface+68> 0xc0403078 <stext_lock+27712>: cmpb $0x0,(%ebx) 0xc040307b <stext_lock+27715>: repz nop 0xc040307d <stext_lock+27717>: jle 0xc0403078 <stext_lock+27712> 0xc040307f <stext_lock+27719>: jmp 0xc0292e60 <ppp_destroy_interface+180> 0xc0403084 <stext_lock+27724>: cmpb $0x0,(%ebx) 0xc0403087 <stext_lock+27727>: repz nop 0xc0403089 <stext_lock+27729>: jle 0xc0403084 <stext_lock+27724> 0xc040308b <stext_lock+27731>: jmp 0xc0292eb0 <ppp_destroy_interface+260> 0xc0403090 <stext_lock+27736>: cmpb $0x0,(%ebx) 0xc0403093 <stext_lock+27739>: repz nop 0xc0403095 <stext_lock+27741>: jle 0xc0403090 <stext_lock+27736> 0xc0403097 <stext_lock+27743>: jmp 0xc0292f40 <ppp_destroy_interface+404> 0xc040309c <stext_lock+27748>: cmpb $0x0,(%ebx) 0xc040309f <stext_lock+27751>: repz nop 0xc04030a1 <stext_lock+27753>: jle 0xc040309c <stext_lock+27748> 0xc04030a3 <stext_lock+27755>: jmp 0xc0293014 <ppp_destroy_interface+616> 0xc04030a8 <stext_lock+27760>: push %eax 0xc04030a9 <stext_lock+27761>: push %ecx 0xc04030aa <stext_lock+27762>: push %edx
(gdb) p stext_lock
$1 = {<text variable, no debug info>} 0xc03fc438 <stext_lock> (gdb) p stext_lock+0x1c82
$1 = (<text variable, no debug info> *) 0xc03fe0ba <stext_lock+7298> (gdb) p stext_lock+0x6c30
$2 = (<text variable, no debug info> *) 0xc0403068 <stext_lock+27696>
Signature · 1 line
Frank
From: "Frank van Maarseveen" <[email protected]>
Quoted 5 lines
0xc03fe0b1<stext_lock+7289>: cmpb $0x0,0xc05dd4000xc03fe0b8<stext_lock+7296>: repz nop0xc03fe0ba<stext_lock+7298>: jle0xc03fe0b1<stext_lock+7289>0xc03fe0bc<stext_lock+7300>: jmp0xc013fde0<get_chrfops+240>
I assume 0xd05dd400 is kernel_flag.
Quoted 4 lines
0xc0403068<stext_lock+27696>: cmpb $0x0,0xc057d5400xc040306f<stext_lock+27703>: repz nop0xc0403071<stext_lock+27705>: jle0xc0403068<stext_lock+27696>0xc0403073<stext_lock+27707>: jmp0xc0292df0<ppp_destroy_interface+68>
And 0xc057d540 is all_ppp_lock.
The only obvious abuse is that both ppp_destroy_interface and ppp_create_interface call rtnl_lock (that's a semaphore) with the spinlock acquired.
--
Manfred
Quoted 1 line · Manfred Spraul
If you are brave, you can try the attached patch.
Your assuptions about the address of kernel_flag and all_ppp_lock were correct (except for an obvious typo).
The patch seems to work, no deadlocks so far but I'll keep an eye on it.
Thanks,
Signature · 1 line
Frank