Runaway cron task on 2.5.63/4 bk?
Second attempt to send this after not seeing it post after about a day. Anyone else have kernel posting problems?
I started seeing the cron task runaway, using 100% CPU continuously on a single CPU with
2.5.63+bk and now with 2.5.64 (about two weeks now.) No other apps/tasks seem to be affected, that I've noticed. It seems to take upwards of 8 hours running the kernel for this to occur.
top shows:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 594 root 25 0 1428 620 1364 R 49.9 0.1 195:23 cron
(This is a dual processor Athlon, so CPU0 is at 100% at the moment.) This is repeatable. Leaving the box running overnight, or all day, and the cron process is running 100% again after several hours. This does not occur in prior 2.5 kernels, or in 2.4.19.
Any idea what's causing this? What additional info on the process would be helpful? kernel .config file at
http://kevb.net/files/linux2564_config
Signature · 1 line
Kevin
Quoted 10 lines · Kevin Brosius
Second attempt to send this after not seeing it post after about a day. Anyone else have kernel posting problems?
I started seeing the cron task runaway, using 100% CPU continuously on a single CPU with
2.5.63+bk and now with 2.5.64 (about two weeks now.) No other apps/tasks seem to be affected, that I've noticed. It seems to take upwards of 8 hours running the kernel for this to occur.
Quoted 5 lines
top shows:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 594 root 25 0 1428 620 1364 R 49.9 0.1 195:23 cron
Yes I've seen this four times over maybe three weeks. Three times on dual CPU, once on a different UP machine.
In all cases, crond is stuck in a loop calling nanosleep with a tv_sec value of a bit over 4,000,000 and a tv_nsec value of zero. nanosleep keeps returning EINVAL immediately.
I'm not sure why crond is trying to sleep for so long. Maybe it has set an alarm.
errr, OK. This returns -EINVAL:
#include <time.h>
main()
{
struct timespec req;
struct timespec rem;
int ret;
req.tv_sec = 5000000;
req.tv_nsec = 0;
ret = nanosleep(&req, &rem);
if (ret)
perror("nanosleep");
}I shall take a look....
Quoted 6 lines · Andrew Morton
errr, OK. This returns
-EINVAL:#include <
time.h>
main()
Quoted 13 lines
{
struct timespec req;
struct timespec rem;
int ret;
req.tv_sec = 5000000;
req.tv_nsec = 0;
ret = nanosleep(&req, &rem);
if (ret)
perror("nanosleep");
}OK, I give up.
/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return -EINVAL;
George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.
I think we need to just sleep for as long as we can and return an appropriate partial result.
Quoted 33 lines · Andrew Morton
Quoted 6 lines · Andrew Morton
errr, OK. This returns
-EINVAL:#include <
time.h>
main()Quoted 13 lines
{ struct timespec req; struct timespec rem; int ret; req.tv_sec = 5000000; req.tv_nsec = 0; ret = nanosleep(&req, &rem); if (ret) perror("nanosleep"); }OK, I give up.
/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return-EINVAL;
Quoted 5 lines
George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.
I think we need to just sleep for as long as we can and return an appropriate partial result.
Cron really isn't at fault, I saw sleep(52) return 4500000, which it just passed into another sleep call.
The problem is a bug in do_clock_nanosleep. If it gets interrupted by a signal, when it calculates the amount of time left, it doesn't check if jiffies has advanced past the expire time, and can pass a negative value to jiffies_to_timespec, which results in values around 4,500,000 ((unsigned int)-1)/HZ, which ends up as sleep's return value. The following trivial patch appears to have fixed the problem on my system. Hopefully this isn't wrapped.
kernel/posix-timers.c +3 -0
@@ -1282,6 +1282,9 @@if (abs)return -ERESTARTNOHAND ; +if (time_after_eq (jiffies_f, new_timer.expires)) +return 0 ; +jiffies_to_timespec (new_timer.expires - jiffies_f, tsave);while (tsave->tv_nsec <0 ) {
Signature · 1 line
Todd Mokros <[email protected]>
Quoted 34 lines · Andrew Morton
Quoted 5 lines · Andrew Morton
errr, OK. This returns
-EINVAL:#include <
time.h>
main()Quoted 13 lines
{ struct timespec req; struct timespec rem; int ret; req.tv_sec = 5000000; req.tv_nsec = 0; ret = nanosleep(&req, &rem); if (ret) perror("nanosleep"); }OK, I give up.
/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return-EINVAL;
Quoted 7 lines
George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.
I think we need to just sleep for as long as we can and return an appropriate partial result.
Linus has fixed the problem cron showed up, so.
Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.
I suppose as HZ gets bigger, this argument will carry less weight,
but, still:
We have, I think, three choices:
1.) Error out as it does now,
2.) Sleep for MAX_INT and return ?????
3.) Sleep for MAX_INT and then sleep some more until the actual time is reached.
2.) Requires, if we are to return other than OK, some way to flag that the error happened.
3.) Likewise, requires more bits in the timer. If we went to a 64-bit expire count, we could do the "right" thing, however it adds an int to the size of the timer_struct.
So, folks, what is the _right_ thing to do here?
-g
Signature · 3 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Quoted 4 lines · george anzinger
Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for
MAX_INTnumber of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.
My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened
- sleep the max amount (old behaviour)
- consider it an error (new behaviour)
In this case the error case actually helped find the other unrelated bug, so in this case the error actually _helped_ us. However, that was only "help" from a kernel perspective, from a user perspective I definitely think that it makes no sense to have "sleep(largenum)" return -EINVAL.
And in the end it's the user that matters.
Linus
Quoted 7 lines · Linus Torvalds
Quoted 1 line · george anzinger
Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for
MAX_INTnumber of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened
I think this will require a 64-bit expire in the timer_struct (actually it would not be treated as such, but the struct would still need the added bits). Is this ok?
I will look at the problem in detail and see if there might be another way without the need of the added bits.
Quoted 7 lines
- sleep the max amount (old behavior)
- consider it an error (new behavior)In this case the error case actually helped find the other unrelated bug, so in this case the error actually _helped_ us. However, that was only "help" from a kernel perspective, from a user perspective I definitely think that it makes no sense to have "sleep(largenum)" return
-EINVAL.And in the end it's the user that matters.
Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.
Signature · 3 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
Quoted 12 lines · george anzinger
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · george anzinger
Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for
MAX_INTnumber of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened
Quoted 6 lines
I think this will require a 64-bit expire in the
timer_struct(actually it would not be treated as such, but the struct would still need the added bits). Is this ok?I will look at the problem in detail and see if there might be another way without the need of the added bits.
Is it not possible to just sit in a loop, sleeping for 0x7fffffff jiffies on each iteration? (Until the final partial bit of course)
Quoted 6 lines
Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.
If an app wants to sleep forever, calling
while (1) sleep(MAX_INT);
seems like a reasonable approach. I'd expect quite a lot of applications would be doing that.
Quoted 13 lines · Andrew Morton
Quoted 11 lines · george anzinger
Quoted 8 lines · Linus Torvalds
Quoted 1 line · george anzinger
Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for
MAX_INTnumber of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened
Quoted 7 lines
Quoted 3 lines
I think this will require a 64-bit expire in the
timer_struct(actually it would not be treated as such, but the struct would still need the added bits). Is this ok?I will look at the problem in detail and see if there might be another way without the need of the added bits.
Is it not possible to just sit in a loop, sleeping for
0x7fffffffjiffies on each iteration? (Until the final partial bit of course)
Seems reasonable. I will have a look.
-g
Quoted 19 lines
Quoted 3 lines
Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.
If an app wants to sleep forever, calling
while (1) sleep(MAX_INT);seems like a reasonable approach. I'd expect quite a lot of applications would be doing that.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 3 lines
George Anzinger [email protected] High-res-timers: http://sourceforge.net/projects/high-res-timers/ Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml