Skip to contentSkip to conversation

scheduler starvation running irman with 2.5.64bk2

DISCUSSION [email protected] 3 participants 7 messages Mar 9, 2003 – Mar 10, 2003

lore
Share & tools
lore
#0 lore raw

irman triggers some odd behavior with 2.5.64bk2 on uniprocessor K6/2 475. "ps aux" hasn't returned for a couple hours, though irman appears to be doing it's thing. I haven't tried irman on smp.

Time to run irman 3x.

2.5.63 4066 seconds
2.5.63-mjb1 2993 seconds
2.5.63-mm2-dline 2856 seconds
2.5.64 3473 seconds
2.5.64bk2 ??

Time to complete irman is variable because the amount of work done in 3 runs is also variable.

vmstat looks fairly typical, and it prints every 60 seconds.

vmstat 60
procs memory swap io system cpu

 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 4  0  0   2564 376368    108   1916    0    0     0     0 1092 25978  2 98  0
 4  0  0   2564 376368    108   1920    0    0     0     0 1092 26102  2 98  0

This is vmstat 60 from 2.5.64

procs memory swap io system cpu

 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 4  0  0   2552 377024    116   1520    0    0     0     0 1092 32649  4 96  0
 3  0  0   2552 377024    116   1520    0    0     0     0 1092 32865  5 95  0

There are probably less than 50 processes running. "ps aux" doesn't hear <ctrl z> or <ctrl c>.

Machine doesn't accept new ssh connections.

It's as if after starting irman, new processes don't get any CPU time.

Signature · 2 lines
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html
#1 lore raw
Quoted 6 lines · On Sat, 8 Mar 2003 [email protected] wrote:

There are probably less than 50 processes running. "ps aux" doesn't hear <ctrl z> or <ctrl c>.

Machine doesn't accept new ssh connections.

It's as if after starting irman, new processes don't get any CPU time.

Try doing a sysrq-T to get task state for all the processes and even sysrq-P at any random point will give you an idea of where the cpu is.

Zwane

Signature · 1 line
function.linuxpower.ca
#2 lore raw
In reply to [email protected]
Quoted 11 lines · At 09:50 PM 3/8/2003 -0500, [email protected] wrote:

irman triggers some odd behavior with 2.5.64bk2 on uniprocessor K6/2 475. "ps aux" hasn't returned for a couple hours, though irman appears to be doing it's thing. I haven't tried irman on smp.

Time to run irman 3x.

2.5.63                  4066 seconds
2.5.63-mjb1             2993 seconds
2.5.63-mm2-dline        2856 seconds
2.5.64                  3473 seconds
2.5.64bk2               ??

It _appears_ to be a valid test-case. I can reproduce this with 2.5.64+combo and B2. I can also influence it to behave. (that doesn't mean that the scheduler changes are bust... could be that the test-case is doing something bad)

Q/indicator: do you ever see ksoftirqd _wanting_ to run? (set top SCHED_RR)

-Mike

#3 lore raw
Quoted 5 lines · Mike Galbraith
Quoted 4 lines · At 09:50 PM 3/8/2003 -0500, [email protected] wrote:

irman triggers some odd behavior with 2.5.64bk2 on uniprocessor K6/2 475. "ps aux" hasn't returned for a couple hours, though irman appears to be doing it's thing. I haven't tried irman on smp.

Quoted 7 lines
Quoted 7 lines

Time to run irman 3x.

2.5.63                  4066 seconds
2.5.63-mjb1             2993 seconds
2.5.63-mm2-dline        2856 seconds
2.5.64                  3473 seconds
2.5.64bk2               ??

(btw, it's only the process load that hangs)

#4 lore raw
In reply to [email protected]
Quoted 3 lines · At 09:50 PM 3/8/2003 -0500, [email protected] wrote:

irman triggers some odd behavior with 2.5.64bk2 on uniprocessor K6/2 475. "ps aux" hasn't returned for a couple hours, though irman appears to be doing it's thing. I haven't tried irman on smp.

OK, can you do the following to determine whether we're both seeing the _same_ problem?

1.)  build the attached rtnice utility (don't remember who wrote/posted this)
2.)  login on vt1 and set the shell SCHED_RR via rtnice -n 1 -p <pid_of_sh> -d RR
3.)  login on vt2 and renice that shell to -10
4.)  login on another vt as a normal user, and start irman
5.)  try login/out on another vt, or ps or _whatever_ (doesn't matter) until box is starving
6.)  on vt2, try to do ps (it should hang despite -10 priority)
7.)  on vt1, try to do ps (it should work just fine)

cc to Ingo since stock .64 doesn't show this problem, and .64-combo and .64-B2 do.

-Mike

#include <stdio.h>
#include <unistd.h>
#include <getopt.h>
#include <string.h>
#include <sys/types.h>
#include <sched.h>
#include <sys/time.h>

/*
* Gross hack, just in case glibc isn't up to date yet.
* (it wasn't when SCHED_IDLE was added to the kernel) */
#ifndef SCHED_IDLE
#define SCHED_IDLE 3
#endif

static char *usage = "Usage: %s -n <priority> -p <pid> -d FF|TS|RR|ID [command...]\n";

static char progname[128];

main(int argc, char *argv[])

	{

	int minpri, maxpri;
	int ch;
	char *p;
	struct timeval interval;
	static struct sched_param param;
	pid_t pid = 0;
	int do_priority = 0;
	int do_policy = 0;
	int policy = 0;
	int priority = 0;
	int args = 0;
	extern char *optarg;
	extern int optind;
	
	(void) strcpy(progname, argv[0]);

	while ((ch = getopt(argc, argv, "p:d:n:")) != EOF)	{

		switch (ch)	{
			case 'p':
				if (sscanf(optarg, "%d", &pid) != 1)	{
					fprintf(stderr,"%s: Bad pid.\n", progname);
					fprintf(stderr, usage, progname);
					exit(3);
		 		}
				args++;
#ifdef DEBUG
				fprintf(stderr, "pid = %d.\n", pid);

#endif

			break;

			case 'd':

				if (!strcmp(optarg, "TS"))	{
					policy = SCHED_OTHER;
		 		}
		 		else
		 		if (!strcmp(optarg, "RR"))	{
		 			policy = SCHED_RR;
		 		}
				else
				if (!strcmp(optarg, "ID"))
					policy = SCHED_IDLE;
		 		else
		 		if (!strcmp(optarg, "FF") || !strcmp(optarg, "FIFO"))	{
		 			policy = SCHED_FIFO;
		 		}
		 		else	{
					fprintf(stderr,"%s: Bad scheduling domain '%s'.\n", progname, optarg);
					fprintf(stderr, usage, progname);
					exit(4);
		 		}
				args++;
				do_policy++;
#ifdef DEBUG
				fprintf(stderr,"policy = %d.\n", policy);

#endif

			break;

			case 'n':
					/* we have one digit, let's get more */
				if (sscanf(optarg, "%d", &priority) != 1)	{
					fprintf(stderr,"%s: Bad priority value.\n", progname);
					fprintf(stderr, usage, progname);
					exit(4);
		 		}
				do_priority++;
				args++;
#ifdef DEBUG
				fprintf(stderr, "priority = %d.\n", priority);

#endif

			break;

			default:
				fprintf(stderr,"%s: Unrecognized option '%c'.\n", progname, ch);
				fprintf(stderr, usage, progname);
				exit(6);

			break;
		}

	}

	argc -= optind;
	argv += optind;

	if (args == 0)	{
		fprintf(stderr, usage, progname);
		exit(7);
	}


	if ((maxpri = sched_get_priority_max(SCHED_FIFO)) == -1)	{
		perror("sched_get_priority_max fails");
		exit(8);
	}

	if ((minpri = sched_get_priority_min(SCHED_FIFO)) == -1)	{
		perror("sched_get_priority_min fails");
		exit(10);
	}

	if (do_priority)	{
		if (priority > maxpri)	{
			fprintf(stderr,"%s: maximum priority allowed is %d.\n", progname, maxpri);
			exit(9);

}

		if (priority < minpri)	{
			fprintf(stderr,"%s: minimum priority allowed is %d.\n", progname, minpri);
			exit(11);
 		}
	}

	if (!do_policy)	{
		if ((policy = sched_getscheduler(pid)) == -1)	{
			policy = SCHED_OTHER;
		}
	}

	if (!do_priority)	{
		if (sched_getparam(pid, &param) == 0)	{ /* will fail for TS! */
			priority = param.sched_priority;

}
else {
priority = minpri;
}
if (!do_policy && pid) { /* print and exit */
static char *pol; /* default */ if (policy == SCHED_OTHER) pol = "TS"; if (policy == SCHED_FIFO) pol = "FIFO"; if (policy == SCHED_RR) pol = "RR";

fprintf(stdout,"%s: scheduler %s priority %d.\n", progname, pol, priority);

			exit(0);
 		}
	}
	if (policy == SCHED_OTHER || policy == SCHED_IDLE)
		priority = 0;

	param.sched_priority = priority;

	if (sched_setscheduler(pid, policy, &param) == -1)	{
		perror("sched_setscheduler fails");
		exit(13);
	}

	if (pid != 0)	{  /* non-local target => exit */
		exit(0);
	}

	if (!argv[0])	{
		fprintf(stderr,"%s: no valid command args.\n", progname);
		fprintf(stderr, usage, progname);

		exit(14);  /* don't exec if target was not us or no args  */
	}

		/* the big leap! */
		/* hal: dave, will I dream? */

	if (!execvp(argv[0], &argv[0]))	{
		perror("rtrun exec fails");
		exit(-1);
	}
}
#5 lore raw
In reply to [email protected]

P.S. You can save a little time by only running the process load. Edit irman.c:412 and set load_num to 3... no need to wait for the other two loads to complete first, it's the process load that starves the box to death here.

-Mike

#6 lore raw
Quoted 1 line · Mike Galbraith

P.S. You can save a little time by only running the process load. Edit irman.c:412 and set load_num to 3... no need to wait for the other two loads to complete first, it's the process load that starves the box to death here.

P.P.S. (last one;)

To make it cease and desist starving your box, renice the shell you're running irman from. 12 is the magic (hmm) number here at which starvation ceases.

It's interesting to watch the response times irman reports. The higher the renice level, the better response times it reports both in 2.5.64-virgin, and 2.5.64-combo (which looks like what is in bk2). /me has no clue whether that means anything or not ;-)

ciao,

-Mike