Skip to contentSkip to conversation

Structure clobbering causes timer oopses

DISCUSSION Dave Hansen 3 participants 4 messages Oct 13, 2002

lore
Share & tools
lore

Dave Hansen

author
#0 lore raw

It put some extrace checks in timer_t, including a tripwire at the beginning and end, just in case the timer was just trampled by something. It was.

I added begin, and end:

Quoted 17 lines

struct timer_list {

(quoted diff) +2 -0
+        unsigned int begin;
        struct list_head entry;
        unsigned long expires;
 
        void (*function)(unsigned long);
        unsigned long data;
 
        struct tvec_t_base_s *base;
+        unsigned int end;

};

static inline void init_timer(struct timer_list * timer) {

(quoted diff) +2 -0
+        timer->begin = TIMER_BEG_MAGIC;
+        timer->end = TIMER_END_MAGIC;
        timer->base = NULL;
Quoted 1 line

}

then this beast:
(yeah, yeah, it ain't pretty, but it worked)

Quoted 9 lines
#define CHECK_TIMER(timer) do {\
                if (((timer)->begin!=TIMER_BEG_MAGIC) || \
                    ((timer)->end!=TIMER_END_MAGIC)) {\
                        printk("timer magic check failed %s:%s():%d\n",
               __stringify(KBUILD_BASENAME),__FUNCTION__,__LINE__);\
                        printk("begin: 0x%x end:0x%x\n", (timer)->begin, 
                        (timer)->end);\
                        dump_stack();\
                }} while (0)

Just before a crash, I got:

timer magic check failed timer:__run_timers():351
begin: 0xc035fbc8 end:0xc035fbe8

Call Trace:
 [<c0120d53>] run_timer_tasklet+0xf7/0x188
 [<c011d945>] tasklet_hi_action+0x85/0xe0
 [<c011d64a>] do_softirq+0x5a/0xac
 [<c01117ed>] smp_apic_timer_interrupt+0x111/0x118
 [<c0105334>] poll_idle+0x0/0x48
 [<c0107a7a>] apic_timer_interrupt+0x1a/0x20
 [<c0105334>] poll_idle+0x0/0x48
 [<c010535d>] poll_idle+0x29/0x48
 [<c01053b3>] cpu_idle+0x37/0x48
 [<c011898d>] printk+0x125/0x140

Then, the full crash:

general protection fault: fbe0
CPU:    4
EIP:    0060:[<c035fbe9>]    Not tainted
EFLAGS: 00010287

EIP is at tvec_bases+0x169/0x20400
eax: d18deac0 ebx: c035dbcc ecx: c035fbe0 edx: c0363f70
esi: c035fbd8 edi: c0363b00 ebp: 00000001 esp: f77c7f1c
ds: 0068 es: 0068 ss: 0068
Process swapper (pid: 0, threadinfo=f77c6000 task=f77c5060)

Stack: c0120d9b c035fbe0 cb1101c8 00000000 f77c6000 c011d945 00000000
       00000001 c035f960 fffffffe 00000080 c03443e4 c03443e4 c011d64a
       c035f960 00000010 00000004 00000000 00000000 00000046 c01117ed
       f77c6000 c0105334 00000000
Call Trace:
 [<c0120d9b>] run_timer_tasklet+0x13f/0x188
 [<c011d945>] tasklet_hi_action+0x85/0xe0
 [<c011d64a>] do_softirq+0x5a/0xac
 [<c01117ed>] smp_apic_timer_interrupt+0x111/0x118
 [<c0105334>] poll_idle+0x0/0x483
 [<c0107a7a>] apic_timer_interrupt+0x1a/0x20
 [<c0105334>] poll_idle+0x0/0x48
 [<c010535d>] poll_idle+0x29/0x48
 [<c01053b3>] cpu_idle+0x37/0x48
 [<c011898d>] printk+0x125/0x140

APIC error on CPU4: 08(08)
APIC error on CPU4: 08(08)
APIC error on CPU4: 08(08)
APIC error on CPU4: 08(08)
...

Notice that the junk that got put in begin, end, and function, are fairly close values, like something was trying to fill out an array.

Can anyone think of clever ways to figure out what is doing the trampling?

BTW, I found lots of users who aren't using init_timer(). Should I publicly humiliate them?

Signature · 2 lines
Dave Hansen
[email protected]
#1 lore raw
Quoted 4 lines · Dave Hansen

...
timer magic check failed timer:__run_timers():351
begin: 0xc035fbc8 end:0xc035fbe8

Can you look these up in System.map?

Quoted 4 lines

..

BTW, I found lots of users who aren't using init_timer(). Should I publicly humiliate them?

If they're initially using add_timer(), that works out OK. It they start out using mod_timer() (or del_timer) then bug.

I assume you tried all the memory debugging options?

author
#2 lore raw
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Dave Hansen

...
timer magic check failed timer:__run_timers():351
begin: 0xc035fbc8 end:0xc035fbe8

Can you look these up in System.map?

Inside tvec_bases, just like eip, because of timer_t->function. c035fa80 d tvec_bases
c037fe80 d pidmap_lock
c037fea0 D page_states

Quoted 5 lines
Quoted 2 lines

BTW, I found lots of users who aren't using init_timer(). Should I publicly humiliate them?

If they're initially using add_timer(), that works out OK. It they start out using mod_timer() (or del_timer) then bug.

The init_timer() comment says otherwise, but I imagine that not using it shouldn't _cause_ any bugs.

* init_timer() must be done to a timer prior calling *any* of the
* other timer functions.

Quoted 1 line

I assume you tried all the memory debugging options?

No luck there. I can't even get the oops to trigger with all the debugging on.

Signature · 2 lines
Dave Hansen
[email protected]
#3 lore raw
Quoted 8 lines · Dave Hansen
Quoted 2 lines

If they're initially using add_timer(), that works out OK. It they start out using mod_timer() (or del_timer) then bug.

The init_timer() comment says otherwise, but I imagine that not using it shouldn't _cause_ any bugs.

* init_timer() must be done to a timer prior calling *any* of the
* other timer functions.

I am not sure about that. init_timer() initializes timer->base and timer_pending() checks for base == NULL. So, it is illegal to do timer_pending(), mod_timer() and del_timer*() without an init_timer() or an add_timer() earlier. But then, I presume this was a requirement in the earlier timer interfaces too. No ?

Thanks

Signature · 2 lines
Dipankar Sarma  <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.