O(1) scheduler seems to lock up on sched_FIFO and sched_RR tasks

It seems that once a SCHED_FIFO or SCHED_RR tasks gets control it does not yield to other tasks of higher priority.

Attached is a test program (busyloop) that just loops doing gettimeofday() for the requested time and a little utility (rt) to run programs at real time priorities.

Here is an annoted example of the problem:

First, become root then:

At this point the bash at priority 90 should be available, but is not.
When the 10 second busyloop completes, bash returns.

SCHED_RR is a bit different:

At this point the bash at priority 90 prints the pid and a prompt and then does not respond for the duration of the busyloop. When the 10 second busyloop completes, bash returns.

The program rt will explain itself if given no parameters.

The program busyloop is just like sleep except that it loops on getimeofday() instead of sleeping.

This test was run on a 2.5.70 bk9 system...

George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

#include <time.h>
#include <stdlib.h>

int main(int argc, char *argv[])

{
	int end_time;
	struct timeval now;

        if (argc < 2) {
                end_time = 1;
        }else{
               	end_time = atoi(argv[1]);
        }
	if (!time) {
		printf("Bad parms, quiting.\n");
		return 1;
	}

	gettimeofday(&now);

	end_time += now.tv_sec + 1;

	do{
		gettimeofday(&now);

	}while (now.tv_sec < end_time);

	printf("bye now.\n");
	return 0;
}

/*
rt - a utility to set the realtime priority and scheduling policy and now also to set nice values.
*/

/* includes */
#include <stdio.h>
#include <stdlib.h>
#include <sched.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/resource.h>
#define _GNU_LIBRARY__
#include <getopt.h>

/* defines */

#define RUNSTRING "usage: rt [-f] [-v] prio [--] runstring \n \
or: rt [-f] [-v] -p PID prio\\ \n\n \
where: prio specifies the realtime priority \n \

       -f set scheduling policy to SCHED_FIFO \n \
       -o set scheduling policy to SCHED_OTHER \n \
          In this case prio is a nice value -20<->19 \n \
          Use '-- prio' if prio is negative \n \
       -v turns on verbose mode. \n \
       -p PID specifies an existing process to modify \n \
       runstring is a process and parameters \n \
       (use '--' if runstring contains options). \n"

#define POLICY(x) x ? x-1 ? "SCHED_RR" : "SCHED_FIFO" : "SCHED_OTHER"

/* prototypes */
void print_usage(char *[]);

/* globals */
int verbose=0; /* 0=none, !0=verbose */

main(int argc, char *argv[])

{
        struct sched_param prio_struct;
        int policy = -1;
        int pid = 0;
        int pidopt = 0;
	int nice;
        int optprobs = 0; /* problems parsing? */
        
        int  c;         /* generic single character */

        /* "standard" option parsing... */
        while ( (c=getopt(argc, argv, "+fop:v?")) != EOF)
        {
        switch (c) {
                case 'f':       /* set FIFO mode */
                        policy = SCHED_FIFO;
                        break;
                case 'o':       /* set SCHED_OTHER mode */
                        policy = SCHED_OTHER;
                        break;
                case 'p':       /* read PID */  
                        sscanf(optarg,"%d",&pid); 
                        pidopt=1;
                        break;
                case 'v':
                        verbose=1;      /* verbosity */
                        break;
                case '?':       /* help? */
                        printf("%s",RUNSTRING);
                        exit(0);
                default:        /* something went wrong */
                        optprobs=1;     /* we'll deal with this problem later */
                        break;
                }
        }

        if (optprobs) {
                fprintf(stderr,RUNSTRING);
                exit(1);
        }


        if((argc - optind) < 2-pidopt) {
                print_usage(argv);
        }

        sscanf(argv[optind], "%d", &(prio_struct.sched_priority));

        /* sanity checking... */
        if ( (prio_struct.sched_priority > 0) && (policy < 0 ) ) {
                policy=SCHED_RR;
                if (verbose)
                  printf("Defaulting sched policy to %s.\n", POLICY(policy));
        }

        if ( (prio_struct.sched_priority == 0 ) && (policy != SCHED_OTHER) ) {
                policy=SCHED_OTHER;
                fprintf(stderr,"Priority of %d implies sched policy of %s.\n",
                        prio_struct.sched_priority,  POLICY(policy)); 
        }


        //policy = (prio_struct.sched_priority)? policy : SCHED_OTHER;
	nice = prio_struct.sched_priority;
	if(policy==SCHED_OTHER)
		prio_struct.sched_priority = 0;	
        if( sched_setscheduler(pid,policy,&prio_struct)){
                perror("Priority out of range");
                print_usage(argv);
        }
	if(policy==SCHED_OTHER){
		if( setpriority(PRIO_PROCESS, pid, nice)){
			perror("Nice out of range");
			print_usage(argv);
		}
        }
        if ( pid ) exit(0);
        argv+=optind;   /* adjust argv to point to the runstring */
        argv++;
        execvp(argv[0],argv);
        perror("exec failed..");
}

void print_usage(char * who[])

{
        printf("%s",RUNSTRING);
        exit (1);
}

Hi George,
When I boost the priority of each of the per-cpu 'events/%d' daemon to
96, the problem goes away.

Joe

Seems like your saying that the events workqueues are involved in the scheduler in some ugly way. Certainly not what your average rt programmer would expect :( What is going on here?

George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
Preemption patch: http://www.kernel.org/pub/linux/kernel/people/rml

Various things like character drivers do rely upon keventd services. So it is possible that bash is stuck waiting on keyboard input, but there is no keyboard input because keventd is locked out.

I'll take a closer look at this, see if there is a specific case which can be fixed.

Arguably, keventd should be running max-prio RT because it is a kernel service, providing "process context interrupt service".

IIRC, Andrea's kernel runs keventd as SCHED_FIFO. I've tried to avoid making this change for ideological reasons ;) Userspace is more important than the kernel and the kernel has no damn right to be saying "oh my stuff is so important that it should run before latency-critical user code".

Tricky.

My guess is that tty/ptty driver character processing is done by the event daemons. In which case your high priority bash is running but the characters it puts out (if any) cannot be seen nor are the characters you type in passed to bash, until the mid-priority process exits and lets the event daemon run.

I view this as a kind of priority inversion. The only solution is for RT users to learn what each daemon does and hand-boost the priority of each to the RT priority appropriate to their mix of RT applications.

Joe