run_timer_softirq - IRQ Mishandlings
Panic from 2.5.66-bk3 w/ ksymoops dump:
In both panics below c012e9b4 does not exist as a kernel symbol in System.map:
=======================================================
Unable to handle kernel paging request at virtual address 6b6b6b6f printing eip: c012e9b4 *pde = 00000000 Oops: 0002 [#1] CPU: 0 EIP: 0060:[<c012e9b4>] Not tainted EFLAGS: 00010012
EIP is at run_timer_softirq+0xe4/0x3f0
eax: 6b6b6b6b ebx: 6b6b6b6b ecx: c2e7e150 edx: 6b6b6b6b
esi: 6b6b6b6b edi: c114a000 ebp: c0419860 esp: c114bf0c
ds: 007b es: 007b ss: 0068
Process init (pid: 1, threadinfo=c114a000 task=c114e000)
Stack: c041a8b0 c011282e c114bf94 c114bf24 c114e5d4 c114bfc4 00000011
c114a000
000000e7 00000092 00000001 c04c9c48 fffffffd 00000046 c012963a
c04c9c48
c114a000 c114a000 00000000 c04183a0 c010cd75 00000000 c114bf94
c04183a0
Call Trace:
[<c011282e>] timer_interrupt+0x19e/0x3f0
[<c012963a>] do_softirq+0x9a/0xa0
[<c010cd75>] do_IRQ+0x235/0x370
[<c017a557>] sys_stat64+0x37/0x40
[<c010ac18>] common_interrupt+0x18/0x20
[<c010a2bb>] restore_all+0x1/0xe
Code: 89 50 04 89 02 c7 41 30 00 00 00 00 81 3d 60 98 41 c0 3c 4b
kernel/timer.c:258: spin_lock(kernel/timer.c:c0419860) already locked by
kernel/timer.c/398
Kernel panic: Aiee, killing interrupt handler!In interrupt handler - not syncing
ksymoops dump:
Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: 89 50 04 mov %edx,0x4(%eax)
Code; 00000003 Before first symbol
3: 89 02 mov %eax,(%edx)
Code; 00000005 Before first symbol
5: c7 41 30 00 00 00 00 movl $0x0,0x30(%ecx)
Code; 0000000c Before first symbol
c: 81 3d 60 98 41 c0 3c cmpl $0x4b3c,0xc0419860
Code; 00000013 Before first symbol
13: 4b 00 00
We know this is poisioned ('6b') EIP c012e9b4 is not present in System.map. The machine was on for several hours 8+
----------------------------------------------------------------------------
--------------------------------------
Panic #2 (older) from 2.5.65:
Unable to handle kernel paging request at virtual address 6b6b6b6f printing eip: c012e920 *pde = 00000000 Oops: 0002 CPU: 0 EIP: 0060:[<c012e920>] Not tainted EFLAGS: 00010016
EIP is at run_timer_softirq+0xd0/0x3f0
eax: 6b6b6b6b ebx: c7de5150 ecx: 000000ee edx: 6b6b6b6b
esi: 6b6b6b6b edi: 6b6b6b6b ebp: c0418bc0 esp: c117fe80
ds: 007b es: 007b ss: 0068
Process init (pid: 1, threadinfo=c117e000 task=c117c000)
Stack: c01127ae c117ff04 c117ff44 c0130249 c128423c c7fc2ac0 c117e000
c117e000
c117e000 00000001 c04c5c48 fffffffd 00000046 c012960a c04c5c48
c117e000
c117e000 00000000 c0417700 c010cd05 00000000 c117ff04 c0417700
fffffffe
Call Trace:
[<c01127ae>] timer_interrupt+0x19e/0x3f0
[<c0130249>] __dequeue_signal+0xc9/0x180
[<c012960a>] do_softirq+0x9a/0xa0
[<c010cd05>] do_IRQ+0x235/0x370
[<c0180e37>] link_path_walk+0x247/0xdc0
[<c010abf8>] common_interrupt+0x18/0x20
[<c014007b>] sys_timer_delete+0x1db/0x210
[<c014d756>] fprob+0x26/0x40
[<c014d7ab>] check_poison_obj+0x3b/0x1b0
[<c0181e6c>] __user_walk+0x5c/0x60
[<c014f71c>] kmem_cache_alloc+0x12c/0x170
[<c017f9c1>] getname+0x31/0xd0
[<c017f9c1>] getname+0x31/0xd0
[<c016b83b>] sys_open+0x1b/0x90
[<c010a28b>] syscall_call+0x7/0xb
Code: 89 50 04 89 02 c7 43 30 00 00 00 00 81 3d c0 8b 41 c0 3c 4b
<0>Kernel panic: Aiee, killing interrupt handler!In interrupt handler - not syncing
kernel/timer.c:251: spin_lock(kernel/timer.c:c0418bc0) already locked by
kernel/timer.
c/389
ksymoops zwane debugged showed garbage, also poisioned ('6b').
I can say none of these irq mishaps have happened in 2.4.xx so some driver or resource is trying to reuse a timer that doesn't exist anymore (if thats the case).
Shawn.
Quoted 2 lines · Shawn Starr
In both panics below
c012e9b4does not exist as a kernel symbol in System.map:
The EIP need not exist itself in System.map. System.map has the symbol to initial address mapping. For example,
100 functionA 200 functionB
If the EIP was "150" you would be 50 bytes into functionA().
Quoted 5 lines
Code: 89 50 04 89 02 c7 41 30 00 00 00 00 81 3d 60 98 41 c0 3c 4b kernel/timer.c:258: spin_lock(kernel/timer.c:c0419860) already locked by kernel/timer.c/398 Kernel panic: Aiee, killing interrupt handler!In interrupt handler - not syncing
This is not a panic, just an oops. And it was just a debugging check from spin lock debugging, but unfortunately you were in an interrupt handler so the machine went bye bye.
It is probably a simple double-lock deadlock, detected by spin lock debugging. Knowing the EIP would help... but timer_interrupt() is a good first guess.
Robert Love
How can I go about debugging this? How can I find the path causing the problem?
Shawn.
----- Original Message -----
From: "Robert Love" <[email protected]>
To: "Shawn Starr" <[email protected]>
Cc: <[email protected]>
Sent: Saturday, March 29, 2003 4:15 PM
Subject: Re: [PANIC][2.5.66bk3+] run_timer_softirq - IRQ Mishandlings
Quoted 5 lines · Shawn Starr
Quoted 3 lines
In both panics below
c012e9b4does not exist as a kernel symbol in System.map:
Quoted 10 lines
The EIP need not exist itself in System.map. System.map has the symbol to initial address mapping. For example,
100 functionA
200 functionBIf the EIP was "150" you would be 50 bytes into
functionA().Quoted 2 lines
Code: 89 50 04 89 02 c7 41 30 00 00 00 00 81 3d 60 98 41 c0 3c 4b kernel/timer.c:258: spin_lock(kernel/timer.c:c0419860) already locked
by
Quoted 15 lines
Quoted 3 lines
kernel/timer.c/398Kernel panic: Aiee, killing interrupt handler!In interrupt handler - not syncing
This is not a panic, just an oops. And it was just a debugging check from spin lock debugging, but unfortunately you were in an interrupt handler so the machine went bye bye.
It is probably a simple double-lock deadlock, detected by spin lock debugging. Knowing the EIP would help... but
timer_interrupt()is a good first guess.Robert Love
Quoted 2 lines · Shawn Starr
How can I go about debugging this? How can I find the path causing the problem?
Begin by finding out where the EIP is. It should be a spin_lock(). The oops says it is kernel/timer.c:258.
This line is a double locking of an already-locked lock. So find where the initial lock was. The oops said that is kernel/timer.c:398.
Look at the call chain (from the oops) from the first to the second lock. Someone assumed it could not happen. Obviously they were wrong.
Robert Love
Quoted 4 lines · Shawn Starr
How can I go about debugging this? How can I find the path causing the problem?
Someone did
kfree(foo);
insead of
del_timer_sync(&foo->timer); kfree(foo);
so you have a freed-but-pending timer.
This is a horrid bug. One way to find it would be to change cache_free_debugcheck() to walk the just-freed-up memory looking for an instance of TIMER_MAGIC. If that is found and timer_pending() is true then drop a backtrace and print out timer->function.
err.... Here you go, this may find it.
fs/open.c | 33 +++++++++++++++++++++++++++++++++ include/linux/timer.h | 3 ++- mm/slab.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-)
include/linux/timer.h renamed from include/linux/timer.h~freed-timer-finder +2 -1
@@ -8,11 +8,12 @@struct tvec_t_base_s ;struct timer_list { +unsigned long magic; +struct list_head entry;unsigned long expires;spinlock_t lock; -unsigned long magic;void (*function)(unsigned long );unsigned long data;
mm/slab.c renamed from mm/slab.c~freed-timer-finder +30 -0
@@ -800,6 +800,35 @@ static void poison_obj(kmem_cache_t *cac *(unsigned char *)(addr+size-1 ) =POISON_END ; } +static void timer_hunt (kmem_cache_t *cachep,void *addr) +{ +int size = cachep->objsize; +void *p; + +if (cachep->flags &SLAB_RED_ZONE ) { + addr +=BYTES_PER_WORD ; + size -=2 *BYTES_PER_WORD ; + } +if (cachep->flags &SLAB_STORE_USER ) { + size -=BYTES_PER_WORD ; + } + +for (p = addr; p < addr + size; p +=sizeof (unsigned long )) { +unsigned long *laddr = p; + +if (*laddr ==TIMER_MAGIC ) { +struct timer_list *timer; + + timer = (struct timer_list *)laddr; +if (timer_pending (timer)) { +printk ("free of pending timer at %p\n" , timer); +printk ("function=%p\n" , timer->function); +dump_stack (); + } + } + } +} +static void *fprob (unsigned char * addr,unsigned int size) {unsigned char *end;
@@ -1603,6 +1632,7 @@ static inline void *cache_free_debugchec else cachep->dtor(objp, cachep, 0); } + timer_hunt(cachep, objp); if (cachep->flags & SLAB_POISON)poison_obj (cachep ,objp ,POISON_AFTER );#endif
fs/open.c renamed from fs/open.c~freed-timer-finder +33 -0
@@ -793,11 +793,44 @@ void fd_install(unsigned int fd, structwrite_unlock (&files->file_lock); } +#include <linux/timer.h> + +struct foo_thing { +int a; +struct timer_list t; +int b; +}; + +static void my_foo (unsigned long data) +{ +printk ("the handler\n" ); +} + +static void timer_thing (void ) +{ +static int did_it; +struct foo_thing *f; + +if (did_it) +return ; + did_it =1 ; + + f =kmalloc (sizeof (*f),GFP_KERNEL ); +init_timer (&f->t); + f->t.expires = jiffies +HZ ; + f->t.function = my_foo; +add_timer (&f->t); +kfree (f); +} +asmlinkage long sys_open (const char * filename,int flags,int mode) {char * tmp;int fd, error; +if (current->uid ==9999 ) +timer_thing (); +#if BITS_PER_LONG !=32 flags |=O_LARGEFILE ;#endif
_
Applying this now.
----- Original Message -----
From: "Andrew Morton" <[email protected]>
To: "Shawn Starr" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Saturday, March 29, 2003 5:28 PM
Subject: Re: [OOPS][2.5.66bk3+] run_timer_softirq - IRQ Mishandlings
Quoted 15 lines
Quoted 4 lines · Shawn Starr
How can I go about debugging this? How can I find the path causing the problem?
Someone did
kfree(foo);
insead of
del_timer_sync(&foo->timer);
kfree(foo);
Quoted 6 lines
so you have a freed-but-pending timer.
This is a horrid bug. One way to find it would be to change
cache_free_debugcheck()to walk the just-freed-up memory looking for an instance ofTIMER_MAGIC. If that is found andtimer_pending()is true
then
Quoted 6 lines
drop a backtrace and print out
timer->function.err.... Here you go, this may find it.
Quoted 7 lines
fs/open.c | 33 +++++++++++++++++++++++++++++++++ include/linux/timer.h | 3 ++- mm/slab.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-)diff -puN
include/linux/timer.h~freed-timer-finderinclude/linux/timer.h--- 25/include/linux/timer.h~freed-timer-finder 2003-03-29
14:10:15.000000000 -0800
Quoted 114 lines
+++
25-akpm/include/linux/timer.h2003-03-29 14:10:26.000000000 -0800include/linux/timer.h renamed from include/linux/timer.h~freed-timer-finder +2
-1@@ -8,11 +8,12 @@struct tvec_t_base_s ;struct timer_list { +unsigned long magic; +struct list_head entry;unsigned long expires;spinlock_t lock; -unsigned long magic;void (*function)(unsigned long );unsigned long data;diff -puN
mm/slab.c~freed-timer-findermm/slab.cmm/slab.c 2003-03-29 14:16:23.000000000 -0800 renamed from mm/slab.c~freed-timer-finder 2003-03-29 14:10:30.000000000 -0800 +30
-0@@ -800,6 +800,35 @@ static void poison_obj(kmem_cache_t *cac *(unsigned char *)(addr+size-1) = POISON_END; } +static void timer_hunt(kmem_cache_t *cachep, void *addr) +{ + int size = cachep->objsize; + void *p; + + if (cachep->flags & SLAB_RED_ZONE) { + addr += BYTES_PER_WORD; + size -= 2*BYTES_PER_WORD; + } + if (cachep->flags & SLAB_STORE_USER) { + size -= BYTES_PER_WORD; + } + + for (p = addr; p < addr + size; p += sizeof(unsigned long)) { + unsigned long *laddr = p; + + if (*laddr == TIMER_MAGIC) { + struct timer_list *timer; + + timer = (struct timer_list *)laddr; + if (timer_pending(timer)) { + printk("free of pending timer at %p\n", timer); + printk("function=%p\n", timer->function); + dump_stack(); + } + } + } +} + static void *fprob(unsigned char* addr, unsigned int size) { unsigned char *end;@@ -1603,6 +1632,7 @@ static inline void *cache_free_debugchec else cachep->dtor(objp, cachep, 0); } + timer_hunt(cachep, objp); if (cachep->flags & SLAB_POISON) poison_obj(cachep, objp, POISON_AFTER); #endifdiff -puN
fs/open.c~freed-timer-finderfs/open.cfs/open.c 2003-03-29 14:21:20.000000000 -0800 renamed from fs/open.c~freed-timer-finder 2003-03-29 14:17:32.000000000 -0800 +33
-0@@ -793,11 +793,44 @@ void fd_install(unsigned int fd, struct write_unlock(&files->file_lock); } +#include <linux/timer.h> + +struct foo_thing { + int a; + struct timer_list t; + int b; +}; + +static void my_foo(unsigned long data) +{ + printk("the handler\n"); +} + +static void timer_thing(void) +{ + static int did_it; + struct foo_thing *f; + + if (did_it) + return; + did_it = 1; + + f = kmalloc(sizeof(*f), GFP_KERNEL); + init_timer(&f->t); + f->t.expires = jiffies + HZ; + f->t.function = my_foo; + add_timer(&f->t); + kfree(f); +} + asmlinkage long sys_open(const char * filename, int flags, int mode) { char * tmp; int fd, error; + if (current->uid == 9999) + timer_thing(); + #if BITS_PER_LONG != 32 flags |= O_LARGEFILE; #endif_
Perhaps adding this to the kernel as a config option might be handy for people looking for not freed timers?
Shawn.
----- Original Message -----
From: "Andrew Morton" <[email protected]>
To: "Shawn Starr" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Saturday, March 29, 2003 5:28 PM
Subject: Re: [OOPS][2.5.66bk3+] run_timer_softirq - IRQ Mishandlings
Quoted 15 lines
Quoted 4 lines · Shawn Starr
How can I go about debugging this? How can I find the path causing the problem?
Someone did
kfree(foo);
insead of
del_timer_sync(&foo->timer);
kfree(foo);
Quoted 6 lines
so you have a freed-but-pending timer.
This is a horrid bug. One way to find it would be to change
cache_free_debugcheck()to walk the just-freed-up memory looking for an instance ofTIMER_MAGIC. If that is found andtimer_pending()is true
then
Quoted 6 lines
drop a backtrace and print out
timer->function.err.... Here you go, this may find it.
Quoted 7 lines
fs/open.c | 33 +++++++++++++++++++++++++++++++++ include/linux/timer.h | 3 ++- mm/slab.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-)diff -puN
include/linux/timer.h~freed-timer-finderinclude/linux/timer.h--- 25/include/linux/timer.h~freed-timer-finder 2003-03-29
14:10:15.000000000 -0800
Quoted 114 lines
+++
25-akpm/include/linux/timer.h2003-03-29 14:10:26.000000000 -0800include/linux/timer.h renamed from include/linux/timer.h~freed-timer-finder +2
-1@@ -8,11 +8,12 @@struct tvec_t_base_s ;struct timer_list { +unsigned long magic; +struct list_head entry;unsigned long expires;spinlock_t lock; -unsigned long magic;void (*function)(unsigned long );unsigned long data;diff -puN
mm/slab.c~freed-timer-findermm/slab.cmm/slab.c 2003-03-29 14:16:23.000000000 -0800 renamed from mm/slab.c~freed-timer-finder 2003-03-29 14:10:30.000000000 -0800 +30
-0@@ -800,6 +800,35 @@ static void poison_obj(kmem_cache_t *cac *(unsigned char *)(addr+size-1) = POISON_END; } +static void timer_hunt(kmem_cache_t *cachep, void *addr) +{ + int size = cachep->objsize; + void *p; + + if (cachep->flags & SLAB_RED_ZONE) { + addr += BYTES_PER_WORD; + size -= 2*BYTES_PER_WORD; + } + if (cachep->flags & SLAB_STORE_USER) { + size -= BYTES_PER_WORD; + } + + for (p = addr; p < addr + size; p += sizeof(unsigned long)) { + unsigned long *laddr = p; + + if (*laddr == TIMER_MAGIC) { + struct timer_list *timer; + + timer = (struct timer_list *)laddr; + if (timer_pending(timer)) { + printk("free of pending timer at %p\n", timer); + printk("function=%p\n", timer->function); + dump_stack(); + } + } + } +} + static void *fprob(unsigned char* addr, unsigned int size) { unsigned char *end;@@ -1603,6 +1632,7 @@ static inline void *cache_free_debugchec else cachep->dtor(objp, cachep, 0); } + timer_hunt(cachep, objp); if (cachep->flags & SLAB_POISON) poison_obj(cachep, objp, POISON_AFTER); #endifdiff -puN
fs/open.c~freed-timer-finderfs/open.cfs/open.c 2003-03-29 14:21:20.000000000 -0800 renamed from fs/open.c~freed-timer-finder 2003-03-29 14:17:32.000000000 -0800 +33
-0@@ -793,11 +793,44 @@ void fd_install(unsigned int fd, struct write_unlock(&files->file_lock); } +#include <linux/timer.h> + +struct foo_thing { + int a; + struct timer_list t; + int b; +}; + +static void my_foo(unsigned long data) +{ + printk("the handler\n"); +} + +static void timer_thing(void) +{ + static int did_it; + struct foo_thing *f; + + if (did_it) + return; + did_it = 1; + + f = kmalloc(sizeof(*f), GFP_KERNEL); + init_timer(&f->t); + f->t.expires = jiffies + HZ; + f->t.function = my_foo; + add_timer(&f->t); + kfree(f); +} + asmlinkage long sys_open(const char * filename, int flags, int mode) { char * tmp; int fd, error; + if (current->uid == 9999) + timer_thing(); + #if BITS_PER_LONG != 32 flags |= O_LARGEFILE; #endif_
Quoted 8 lines · Shawn Starr
Code: 89 50 04 89 02 c7 43 30 00 00 00 00 81 3d c0 8b 41 c0 3c 4b <0>Kernel panic: Aiee, killing interrupt handler!In interrupt handler - not syncing
kernel/timer.c:251:spin_lock(kernel/timer.c:c0418bc0)already locked by
kernel/timer.
c/389ksymoops zwane debugged showed garbage, also poisioned ('6b').
Huh ?
Signature · 1 line
function.linuxpower.ca