problem with the timers ?!? (was: No one wants to help me)

ok, here are some more info on this issue. BTW, I am the other guy Mircea is talking about, and I wrote the paranoia timer.c that call debug_timer_list() all the time to check the integrity of the timer lists. That file was a test of mine and it wasn't initialy intended for public release, so SORRY for // comments and so. it happens that I like them and the problem is with the timers not with the comments. let's stay focused. the final patch will look nice. I promise.

so,

1. first of all this is not a hardware problem. the memory is OK. CPU is OK. BTW it is a dual PIII 733, 512 RAM. IDE ATA 100.

2. this problem happens only in internal_add_timer() especialy it happens when it calls "list_add(&timer->list, vec->prev);" at the end of internal_add_timer()

I analized a crash and the problem is that vec->prev is NULL. So someone is reseting the next/prev pointers of a timer that is in use.

3. there is chance that because of the patch the race condition that used to cause the crash does not happen any more. after 2 days, there is no crash. so SORRY, but there is no aditional info i can post, no pointers no functions, no nothing. it works just fine right now ... :-)

4. if you look at the debug_timer_list() function you will notice that when it finds a bad cell in the list, it will print the current cell and also the next/prev cell and the chain index that contains the bad cell. for each cell it will print the next/prev pointers, also the data and the function pointer.

Also it will try to find the other loose end and repair the list by connecting them back togher, bypassing the bad cell. this is a disperate measure I know, but it is better then just crashing.

5. I don't think there is an obvious problem in detach_timer() as Brian Gerst <[email protected]> is saying ...

static inline int detach_timer (struct timer_list *timer) {
if (!timer_pending(timer)) return 0; list_del(&timer->list);
return 1;
}

timer_pending(timer) check if the next/prev pointers are NULL. if they are NULL, then the timer is already REMOVED and IS NOT PENDING, so there is nothing to detach so detach_timer() just exit and does nothing. CORRECT. i don't see anything wrong with this.
the upper function can reset the next/prev pointers again. there is simply no problem, if the timer was already dleeted, they were already set to NULL. if the timer was just deleted, then is OK too to reset the pointers since it is deleted.

also it doesn't matter if that timer is running or not. if it is running then it is already removed, so again there is nothing to do. and again there is no problem with reseting the next/prev pointers there were already been NULL :-)

also I don't understand how a race condition can occour since all functions that play with the lists has a spin_lock() arround them. Maybe is not someting wrong in the timers, maybe some other part of the kernel is doing the bad thing.

6. Also I have a strange feeling that this is caused by the networking. OFC, it could be because networking has a lot of timers and the race condition just has a biger chance to be triggered by the networking ...

7. Could it be possible that some function add a timer, and BEFORE it expires, the function forgets to delete it, and just call init_timer again?

6. some history about this crashing thing ...
2.4.0 - I did not noticed it to crash. maybe because the system was
rebooted quite often because of other reasons.
2.4.1 - it worked for about 2 weeks, no problem. then i upgraded the
kernel.
2.4.2 - nightmare starts. first crash after 1 week.
- i upgrade to various pre something patches, no good. it keep crashing.
2.4.3 - crashed twice in the same day, last time after only 2h.

I downgraded to 2.4.1 - it worked fine for 1 week, and when i was finnaly happy it crashes. some problem.
This is when i wrote that modified timer.c
FYI, timer.c didn't changed a BIT since 2.4.0 till now. so you can apply the patch no matter your kernel version.

Now I am running the paranoia timer.c on a 2.4.4-pre2 kernel. and nothing happened after 2 days. If it will find a broken cell I will post the relevant info to LK ...

Best regards,
------
Doru Petrescu
KappaNet - Senior Software Engineer
E-mail: [email protected] LINUX - the choice of the GNU generation

Yes. All it takes is this:

handler(foo)

{
	stuff();
	mod_timer(&foo->timer, whenever);
}

mainline()

{
	del_timer(&foo->timer);
	kfree(foo);
}

If the handler and mainline run at the same time we will add a timer which is in freed memory. Later, someone reuses that memory and changes it. The timer list is corrupted.

Problem is, it seems that your machine is using IPV4, TCP, IDE, netfilter and nothing else. Those parts of the kernel don't have the above bug (well, they didn't mid last year).

One really, really useful piece of information would be the value of the `function' member of the corrupted timer. Your debug code prints this out. Do you still have the logs?

Was it ever non-zero?

If so, what function was it pointing at?

-

The matchine crashed several times, BEFORE I modified timer.c it did NOT CRASHED even once after that :(
I am waiting for it to chrash.
when it will crash, I will let you guys know.

But, as I stated before, I fear that because of the changes, the race condition does not happen any more ...

Best regards,
------
Doru Petrescu
KappaNet - Senior Software Engineer
E-mail: [email protected] LINUX - the choice of the GNU generation