MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!
PATCH 1/1 MAX_USER_RT_PRIO and MAX_RT_PRIO are wrong!
According to the comments in include/linux/sched.h
/*
* Priority of a process goes from 0..MAX_PRIO-1, valid RT
* priority is 0..MAX_RT_PRIO-1, and SCHED_NORMAL tasks are
* in the range MAX_RT_PRIO..MAX_PRIO-1. Priority values
* are inverted: lower p->prio value means higher priority. *
* The MAX_USER_RT_PRIO value allows the actual maximum
* RT priority to be separate from the value exported to
* user-space. This allows kernel threads to set their
* priority to a value higher than any user task. Note:
* MAX_RT_PRIO must not be smaller than MAX_USER_RT_PRIO. */
This makes it look like the priority goes as follows:
prio: 0 .. MAX_RT_PRIO .. MAX_USER_RT_PRIO .. MAX_PRIO
where 0 is of highest priority
but in reality we have:
prio: 0 .. MAX_USER_RT_PRIO .. MAX_RT_PRIO .. MAX_PRIO
The comments say that MAX_RT_PRIO must not be smaller than MAX_USER_RT_PRIO, but if it is bigger (thinking bigger means greater than) then the system will crash on a SMP machine.
Here's how it works. The migration_thread sets the priority of its thread to MAX_RT_PRIO-1 via:
__setscheduler(p, SCHED_FIFO, MAX_RT_PRIO-1);
Now looking at __setscheduler
static void __setscheduler(struct task_struct *p, int policy, int prio)
{
BUG_ON(p->array);
p->policy = policy;
p->rt_priority = prio;
if (policy != SCHED_NORMAL)
p->prio = MAX_USER_RT_PRIO-1 - p->rt_priority;
else
p->prio = p->static_prio;
}If we have MAX_USER_RT_PRIO = 99 and MAX_RT_PRIO = 100 then we would get
p->prio = 99-1 - 100-1 = -1;
This would be very bad when it comes time to schedule. Not to mention that kstop_machine uses MAX_RT_PRIO and then calls sys_sched_setscheduler, which would fail if MAX_RT_PRIO > MAX_USER_RT_PRIO. Below is a patch that makes MAX_RT_PRIO work if it is greater than MAX_USER_RT_PRIO on a SMP machine. The p->mm is to allow kstop_machine to work and any other kernel threads.
I tested the patch on an SMP machine where MAX_RT_PRIO = 100 and MAX_USER_RT_PRIO = 99. Without the patch, the system crashes with a reboot.
Funny, back in July 2002, this was noticed by an Anton Wilson and he was just lost in the noise!
http://seclists.org/lists/linux-kernel/2002/Jul/1695.html
-- Steve
kernel/sched.c +3 -2
@@ -3347,7 +3347,7 @@ p->policy = policy; p->rt_priority = prio;if (policy !=SCHED_NORMAL ) - p->prio =MAX_USER_RT_PRIO -1 - p->rt_priority; + p->prio =MAX_RT_PRIO -1 - p->rt_priority;else p->prio = p->static_prio; }
@@ -3379,7 +3379,8 @@* 1..MAX_USER_RT_PRIO-1, valid priority for SCHED_NORMAL is 0. */ if (param->sched_priority <0 || - param->sched_priority >MAX_USER_RT_PRIO -1 ) + (p->mm && param->sched_priority >MAX_USER_RT_PRIO -1 ) || + (!p->mm && param->sched_priority >MAX_RT_PRIO -1 ))return -EINVAL ;if ((policy ==SCHED_NORMAL ) != (param->sched_priority ==0 ))return -EINVAL ;
Quoted 5 lines · Steven Rostedt
This makes it look like the priority goes as follows:
prio: 0 ..
MAX_RT_PRIO..MAX_USER_RT_PRIO..MAX_PRIOwhere 0 is of highest priority
I'm correcting my own post :-)
What we really want is:
prio: 0 .. [MAX_RT_PRIO - MAX_USER_RT_PRIO] .. MAX_RT_PRIO .. MAX_PRIO
|---- nice -------|
|------ user RT prio ------|
|------------ kernel RT prio -----------------|Remember, 0 is of highest priority.
-- Steve
Quoted 3 lines · Steven Rostedt
I tested the patch on an SMP machine where
MAX_RT_PRIO= 100 andMAX_USER_RT_PRIO= 99. Without the patch, the system crashes with a reboot.
Ingo
Quoted 7 lines · Ingo Molnar
Quoted 3 lines · Steven Rostedt
I tested the patch on an SMP machine where
MAX_RT_PRIO= 100 andMAX_USER_RT_PRIO= 99. Without the patch, the system crashes with a reboot.Acked-by Ingo Molnar <[email protected]>
If this patch does go in, then xpc_activating in arch/ia64/sn/kernel/xpc_main.c (from rc6) also needs to use MAX_RT_PRIO instead of MAX_USER_RT_PRIO. Unless it is OK that it runs lower in priority than other kernel threads.
-- Steve
Quoted 7 lines · Steven Rostedt
Quoted 6 lines · Ingo Molnar
Quoted 3 lines · Steven Rostedt
I tested the patch on an SMP machine where
MAX_RT_PRIO= 100 andMAX_USER_RT_PRIO= 99. Without the patch, the system crashes with a reboot.
Quoted 6 lines
Quoted 1 line
Acked-by Ingo Molnar <[email protected]>If this patch does go in, then
xpc_activatinginarch/ia64/sn/kernel/xpc_main.c(from rc6) also needs to useMAX_RT_PRIOinstead ofMAX_USER_RT_PRIO. Unless it is OK that it runs lower in priority than other kernel threads.
You are correct xpc_activating() needs to be changed to use MAX_RT_PRIO. So please do add that change to your patch.
Thanks,
Dean
Quoted 2 lines · Dean Nelson
You are correct
xpc_activating()needs to be changed to useMAX_RT_PRIO. So please do add that change to your patch.
I haven't tested this patch, I just used the previous patch (which I did test) and added your change.
-- Steve
kernel/sched.c renamed from kernel/sched.c.orig +3 -2
@@ -3347,7 +3347,7 @@ p->policy = policy; p->rt_priority = prio;if (policy !=SCHED_NORMAL ) - p->prio =MAX_USER_RT_PRIO -1 - p->rt_priority; + p->prio =MAX_RT_PRIO -1 - p->rt_priority;else p->prio = p->static_prio; }
@@ -3379,7 +3379,8 @@* 1..MAX_USER_RT_PRIO-1, valid priority for SCHED_NORMAL is 0. */ if (param->sched_priority <0 || - param->sched_priority >MAX_USER_RT_PRIO -1 ) + (p->mm && param->sched_priority >MAX_USER_RT_PRIO -1 ) || + (!p->mm && param->sched_priority >MAX_RT_PRIO -1 ))return -EINVAL ;if ((policy =SCHED_NORMAL ) != (param->sched_priority =0 ))return -EINVAL ;
arch/ia64/sn/kernel/xpc_main.c renamed from arch/ia64/sn/kernel/xpc_main.c.orig +1 -1
@@ -420,7 +420,7 @@partid_t partid = (u64 ) __partid;struct xpc_partition *part = &xpc_partitions[partid];unsigned long irq_flags; -struct sched_param param = { sched_priority:MAX_USER_RT_PRIO -1 }; +struct sched_param param = { sched_priority:MAX_RT_PRIO -1 };int ret;
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · Dean Nelson
You are correct
xpc_activating()needs to be changed to useMAX_RT_PRIO. So please do add that change to your patch.I haven't tested this patch, I just used the previous patch (which I did test) and added your change.
I just built and tested a kernel and xp/xpc/xpnet modules with your patch applied. It ran fine. The priorities of the xpc kthreads were correct.
Looks good to me.
Thanks,
Dean
Quoted 4 lines · Dean Nelson
I just built and tested a kernel and xp/xpc/xpnet modules with your patch applied. It ran fine. The priorities of the xpc kthreads were correct.
Looks good to me.
Dean,
If you can do me a favor, the way you really want to test this is by changing MAX_USER_RT_PRIO to 99 and MAX_RT_PRIO to (MAX_USER_RT_PRIO+1). This will make sure that the patch is working. Your kernel thread should still run at priority 99.
Check it with: ps -eo pid,rtprio,comm
And grep for your thread name.
Thanks,
-- Steve
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · Dean Nelson
I just built and tested a kernel and xp/xpc/xpnet modules with your patch applied. It ran fine. The priorities of the xpc kthreads were correct.
Looks good to me.
Dean,
Quoted 8 lines
If you can do me a favor, the way you really want to test this is by changing
MAX_USER_RT_PRIOto 99 andMAX_RT_PRIOto (MAX_USER_RT_PRIO+1). This will make sure that the patch is working. Your kernel thread should still run at priority 99.Check it with: ps -eo pid,rtprio,comm
And grep for your thread name.
Just did as you asked and things seem fine. Ran on an SGI altix. (The first process shown below shouldn't have a priority of 99, just the others.)
cranberry5:~ # ps -eo pid,rtprio,comm | grep xpc
13325 - xpc_hb
13327 99 xpc08 13467 99 xpc06 13469 99 xpc06c1 13501 99 xpc06c1 13502 99 xpc06c1
cranberry5:~ #