Skip to contentSkip to conversation

Runaway cron task on 2.5.63/4 bk?

DISCUSSION Kevin Brosius 5 participants 9 messages Mar 9, 2003 – Mar 10, 2003

lore
Share & tools
lore
#0 lore raw

Second attempt to send this after not seeing it post after about a day. Anyone else have kernel posting problems?

I started seeing the cron task runaway, using 100% CPU continuously on a single CPU with
2.5.63+bk and now with 2.5.64 (about two weeks now.) No other apps/tasks seem to be affected, that I've noticed. It seems to take upwards of 8 hours running the kernel for this to occur.

top shows:

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  594 root      25   0  1428  620  1364 R    49.9  0.1 195:23 cron

(This is a dual processor Athlon, so CPU0 is at 100% at the moment.) This is repeatable. Leaving the box running overnight, or all day, and the cron process is running 100% again after several hours. This does not occur in prior 2.5 kernels, or in 2.4.19.

Any idea what's causing this? What additional info on the process would be helpful? kernel .config file at
http://kevb.net/files/linux2564_config

Signature · 1 line
Kevin
#1 lore raw
Quoted 10 lines · Kevin Brosius

Second attempt to send this after not seeing it post after about a day. Anyone else have kernel posting problems?

I started seeing the cron task runaway, using 100% CPU continuously on a single CPU with
2.5.63+bk and now with 2.5.64 (about two weeks now.) No other apps/tasks seem to be affected, that I've noticed. It seems to take upwards of 8 hours running the kernel for this to occur.

Quoted 5 lines

top shows:

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  594 root      25   0  1428  620  1364 R    49.9  0.1 195:23 cron

Yes I've seen this four times over maybe three weeks. Three times on dual CPU, once on a different UP machine.

In all cases, crond is stuck in a loop calling nanosleep with a tv_sec value of a bit over 4,000,000 and a tv_nsec value of zero. nanosleep keeps returning EINVAL immediately.

I'm not sure why crond is trying to sleep for so long. Maybe it has set an alarm.

errr, OK. This returns -EINVAL:

#include <time.h>

main()

{
	struct timespec req;
	struct timespec rem;
	int ret;

	req.tv_sec = 5000000;
	req.tv_nsec = 0;

	ret = nanosleep(&req, &rem);
	if (ret)
		perror("nanosleep");
}

I shall take a look....

#2 lore raw
Quoted 6 lines · Andrew Morton

errr, OK. This returns -EINVAL:

#include <time.h>

main()

Quoted 13 lines
{
	struct timespec req;
	struct timespec rem;
	int ret;

	req.tv_sec = 5000000;
	req.tv_nsec = 0;

	ret = nanosleep(&req, &rem);
	if (ret)
		perror("nanosleep");
}

OK, I give up.

/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return -EINVAL;

George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.

I think we need to just sleep for as long as we can and return an appropriate partial result.

#3 lore raw
Quoted 33 lines · Andrew Morton
Quoted 6 lines · Andrew Morton

errr, OK. This returns -EINVAL:

#include <time.h>

main()

Quoted 13 lines
{
	struct timespec req;
	struct timespec rem;
	int ret;

	req.tv_sec = 5000000;
	req.tv_nsec = 0;

	ret = nanosleep(&req, &rem);
	if (ret)
		perror("nanosleep");
}

OK, I give up.

/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return -EINVAL;

Quoted 5 lines

George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.

I think we need to just sleep for as long as we can and return an appropriate partial result.

Cron really isn't at fault, I saw sleep(52) return 4500000, which it just passed into another sleep call.
The problem is a bug in do_clock_nanosleep. If it gets interrupted by a signal, when it calculates the amount of time left, it doesn't check if jiffies has advanced past the expire time, and can pass a negative value to jiffies_to_timespec, which results in values around 4,500,000 ((unsigned int)-1)/HZ, which ends up as sleep's return value. The following trivial patch appears to have fixed the problem on my system. Hopefully this isn't wrapped.

kernel/posix-timers.c +3 -0
@@ -1282,6 +1282,9 @@
 		if (abs)
 			return -ERESTARTNOHAND;
 
+		if (time_after_eq(jiffies_f, new_timer.expires))
+			return 0;
+
 		jiffies_to_timespec(new_timer.expires - jiffies_f, tsave);
 
 		while (tsave->tv_nsec < 0) {
Signature · 1 line
Todd Mokros <[email protected]>
#4 lore raw
In reply to Andrew Morton
Quoted 34 lines · Andrew Morton
Quoted 5 lines · Andrew Morton

errr, OK. This returns -EINVAL:

#include <time.h>

main()

Quoted 13 lines
{
	struct timespec req;
	struct timespec rem;
	int ret;

	req.tv_sec = 5000000;
	req.tv_nsec = 0;

	ret = nanosleep(&req, &rem);
	if (ret)
		perror("nanosleep");
}

OK, I give up.

/*
* This is a considered response, not exactly in
* line with the standard (in fact it is silent on
* possible overflows). We assume such a large
* value is ALMOST always a programming error and
* try not to compound it by setting a really dumb
* value.
*/
return -EINVAL;

Quoted 7 lines

George, RH7.3 and RH8.0 cron daemons are triggering this (trying to sleep for 4,500,000 seconds) and it causes them to go into a busy loop.

I think we need to just sleep for as long as we can and return an appropriate partial result.

Linus has fixed the problem cron showed up, so.

Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.
I suppose as HZ gets bigger, this argument will carry less weight,
but, still:

We have, I think, three choices:
1.) Error out as it does now,
2.) Sleep for MAX_INT and return ?????
3.) Sleep for MAX_INT and then sleep some more until the actual time is reached.

2.) Requires, if we are to return other than OK, some way to flag that the error happened.

3.) Likewise, requires more bits in the timer. If we went to a 64-bit expire count, we could do the "right" thing, however it adds an int to the size of the timer_struct.

So, folks, what is the _right_ thing to do here?

-g

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#5 lore raw
Quoted 4 lines · george anzinger

Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.

My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened
- sleep the max amount (old behaviour)
- consider it an error (new behaviour)

In this case the error case actually helped find the other unrelated bug, so in this case the error actually _helped_ us. However, that was only "help" from a kernel perspective, from a user perspective I definitely think that it makes no sense to have "sleep(largenum)" return -EINVAL.

And in the end it's the user that matters.

Linus

#6 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 1 line · george anzinger

Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.

My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened

I think this will require a 64-bit expire in the timer_struct (actually it would not be treated as such, but the struct would still need the added bits). Is this ok?

I will look at the problem in detail and see if there might be another way without the need of the added bits.

Quoted 7 lines

- sleep the max amount (old behavior)
- consider it an error (new behavior)

In this case the error case actually helped find the other unrelated bug, so in this case the error actually _helped_ us. However, that was only "help" from a kernel perspective, from a user perspective I definitely think that it makes no sense to have "sleep(largenum)" return -EINVAL.

And in the end it's the user that matters.

Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml
#7 lore raw
Quoted 12 lines · george anzinger
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · george anzinger

Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.

My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened

Quoted 6 lines

I think this will require a 64-bit expire in the timer_struct (actually it would not be treated as such, but the struct would still need the added bits). Is this ok?

I will look at the problem in detail and see if there might be another way without the need of the added bits.

Is it not possible to just sit in a loop, sleeping for 0x7fffffff jiffies on each iteration? (Until the final partial bit of course)

Quoted 6 lines

Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.

If an app wants to sleep forever, calling

	while (1)
		sleep(MAX_INT);

seems like a reasonable approach. I'd expect quite a lot of applications would be doing that.

#8 lore raw
Quoted 13 lines · Andrew Morton
Quoted 11 lines · george anzinger
Quoted 8 lines · Linus Torvalds
Quoted 1 line · george anzinger

Lets consider this one on its own merits. What SHOULD sleep do when asked to sleep for MAX_INT number of jiffies or more, i.e. when jiffies overflows? My notion, above, it that it is clearly an error.

My suggestion (in order of preference):
- sleep the max amount, and then restart as if a signal had happened

Quoted 7 lines
Quoted 3 lines

I think this will require a 64-bit expire in the timer_struct (actually it would not be treated as such, but the struct would still need the added bits). Is this ok?

I will look at the problem in detail and see if there might be another way without the need of the added bits.

Is it not possible to just sit in a loop, sleeping for 0x7fffffff jiffies on each iteration? (Until the final partial bit of course)

Seems reasonable. I will have a look.

-g

Quoted 19 lines
Quoted 3 lines

Hm... I changed it to what it is to make it easier to track down problems in the test code... and this was user code. My thinking was that such large values are clear errors, and having the code "hang" in the sleep just hides the problem. But then, I NEVER make a system call without checking for errors.... And, I was making a LOT of sleep calls and wanted to know which one(s) were wrong.

If an app wants to sleep forever, calling

	while (1)
		sleep(MAX_INT);

seems like a reasonable approach. I'd expect quite a lot of applications would be doing that.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml