sched: SCHED_DEADLINE v9
Subsystems (10)
cover sched: SCHED_DEADLINE v9
Hello everyone,
SCHED_DEADLINE patchset v9, less than a month after last version [1]!
Changes w.r.t. v8:
- rebase on top of 3.12;
- all comments from Peter Zijlstra and Ingo Molnar applied (thanks!):
+ auxiliary functions for UP/SMP cases
+ add a smp_rmb to match dl_set_overload smp_wmb
+ different clocks for deadlines and runtime need further study
+ amend the comment about yield_task_dl semantic
+ use an accessor to read the rq clock
+ clarify 06/14 changelog (why we want to specify periods != deadlines)
The development is taking place at:
https://github.com/jlelli/sched-deadline
branch: sched-dl-V9-rebase (this patchset on top of tip/master).
Check the repositories frequently if you're interested, and feel free to e-mail me for any issue you run into.
Test applications:
https://github.com/gbagnoli/rt-app
https://github.com/jlelli/schedtool-dl
Development mailing list: linux-dl; you can subscribe from here:
http://feanor.sssup.it/mailman/listinfo/linux-dl
or via e-mail (send a message to [email protected] with just the word `help' as subject or in the body to receive info).
The code was being jointly developed by ReTiS Lab (http://retis.sssup.it)
and Evidence S.r.l (http://www.evidence.eu.com) in the context of the ACTORS EU-funded project (http://www.actors-project.eu) and the S(o)OS EU-funded project (http://www.soos-project.eu/). Development is now supported by the JUNIPER EU-funded project [2].
As usual, any kind of feedback is welcome and appreciated.
Thanks in advice and regards,
- Juri
[1] http://lwn.net/Articles/376502, http://lwn.net/Articles/353797,
http://lwn.net/Articles/412410, http://lwn.net/Articles/490944, http://lwn.net/Articles/498472, http://lwn.net/Articles/521091, http://lwn.net/Articles/537388, http://lwn.net/Articles/570293
[2] http://www.juniper-project.org/page/overview
Dario Faggioli (9):
sched: add sched_class->task_dead.
sched: add extended scheduling interface.
sched: SCHED_DEADLINE structures & implementation.
sched: SCHED_DEADLINE avg_update accounting.
sched: add schedstats for -deadline tasks.
sched: add latency tracing for -deadline tasks.
sched: drafted deadline inheritance logic.
sched: add bandwidth management for sched_dl.
sched: add sched_dl documentation.
Harald Gustafsson (1):
sched: add period support for -deadline tasks.
Juri Lelli (3):
sched: SCHED_DEADLINE SMP-related data structures & logic.
sched: make dl_bw a sub-quota of rt_bw
sched: speed up -dl pushes with a push-heap.
Peter Zijlstra (1):
rtmutex: turn the plist into an rb-tree.
Documentation/scheduler/sched-deadline.txt | 196 ++++ arch/arm/include/asm/unistd.h | 2 +- arch/arm/include/uapi/asm/unistd.h | 3 + arch/arm/kernel/calls.S | 3 + arch/x86/syscalls/syscall_32.tbl | 3 + arch/x86/syscalls/syscall_64.tbl | 3 + include/linux/init_task.h | 10 + include/linux/rtmutex.h | 18 +- include/linux/sched.h | 122 +- include/linux/sched/deadline.h | 26 + include/linux/sched/rt.h | 3 +- include/linux/sched/sysctl.h | 11 + include/linux/syscalls.h | 7 + include/uapi/linux/sched.h | 1 + kernel/fork.c | 8 +- kernel/futex.c | 2 + kernel/hrtimer.c | 3 +- kernel/rtmutex-debug.c | 8 +- kernel/rtmutex.c | 164 ++- kernel/rtmutex_common.h | 22 +- kernel/sched/Makefile | 4 +- kernel/sched/core.c | 674 ++++++++++- kernel/sched/cpudeadline.c | 216 ++++ kernel/sched/cpudeadline.h | 33 + kernel/sched/deadline.c | 1666 ++++++++++++++++++++++++++++ kernel/sched/debug.c | 46 + kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 150 +++ kernel/sched/stop_task.c | 2 +- kernel/sysctl.c | 7 + kernel/trace/trace_sched_wakeup.c | 45 +- kernel/trace/trace_selftest.c | 28 +- 32 files changed, 3363 insertions(+), 125 deletions(-) create mode 100644 Documentation/scheduler/sched-deadline.txt create mode 100644 include/linux/sched/deadline.h create mode 100644 kernel/sched/cpudeadline.c create mode 100644 kernel/sched/cpudeadline.h create mode 100644 kernel/sched/deadline.c
Signature · 1 line
1.7.9.5
PATCH 1/14 sched: add sched_class->task_dead.
From: Dario Faggioli <[email protected]>
Add a new function to the scheduling class interface. It is called at the end of a context switch, if the prev task is in TASK_DEAD state.
It might be useful for the scheduling classes that want to be notified when one of their task dies, e.g. to perform some cleanup actions.
--- kernel/sched/core.c | 3 +++ kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+)
kernel/sched/core.c +3 -0
@@ -1890,6 +1890,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list.
kernel/sched/sched.h +1 -0
@@ -992,6 +992,7 @@ struct sched_class {void (*set_curr_task) (struct rq *rq);void (*task_tick) (struct rq *rq,struct task_struct *p,int queued);void (*task_fork) (struct task_struct *p); +void (*task_dead) (struct task_struct *p);void (*switched_from) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);
Signature · 1 line
1.7.9.5
Re: [PATCH 01/14] sched: add sched_class->task_dead.
Quoted 7 lines · Juri Lelli
From: Dario Faggioli <[email protected]>
Add a new function to the scheduling class interface. It is called at the end of a context switch, if the prev task is in
TASK_DEADstate.It might be useful for the scheduling classes that want to be notified when one of their task dies, e.g. to perform some cleanup actions.
We could also usefully use this within SCHED_FAIR to remove a task's load contribution when it completes.
Quoted 27 lines
Signed-off-by Dario Faggioli <[email protected]>Signed-off-by Juri Lelli <[email protected]>--- kernel/sched/core.c | 3 +++ kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+)kernel/sched/core.c +3
-0@@ -1890,6 +1890,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list. kernel/sched/sched.h +0
-0
kernel/sched/sched.h struct sched_class { · line 998
Quoted 7 lines
kernel/sched/sched.h +1
-0@@ -992,6 +992,7 @@ struct sched_class {void (*set_curr_task) (struct rq *rq);void (*task_tick) (struct rq *rq,struct task_struct *p,int queued);void (*task_fork) (struct task_struct *p); +void (*task_dead) (struct task_struct *p);void (*switched_from) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);
Quoted 8 lines
--
1.7.9.5--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Steven Rostedt
maintainerRe: [PATCH 01/14] sched: add sched_class->task_dead.
In reply to Juri LelliQuoted 7 lines · On Thu, 7 Nov 2013 14:43:35 +0100 Juri Lelli <[email protected]> wrote:
From: Dario Faggioli <[email protected]>
Add a new function to the scheduling class interface. It is called at the end of a context switch, if the prev task is in
TASK_DEADstate.It might be useful for the scheduling classes that want to be notified when one of their task dies, e.g. to perform some cleanup actions.
Nit. s/task/tasks/
-- Steve
Quoted 34 lines
Signed-off-by Dario Faggioli <[email protected]>Signed-off-by Juri Lelli <[email protected]>--- kernel/sched/core.c | 3 +++ kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+)kernel/sched/core.c +3
-0@@ -1890,6 +1890,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list. kernel/sched/sched.h +1
-0@@ -992,6 +992,7 @@ struct sched_class {void (*set_curr_task) (struct rq *rq);void (*task_tick) (struct rq *rq,struct task_struct *p,int queued);void (*task_fork) (struct task_struct *p); +void (*task_dead) (struct task_struct *p);void (*switched_from) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);
Re: [PATCH 01/14] sched: add sched_class->task_dead.
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Thu, 7 Nov 2013 14:43:35 +0100 Juri Lelli <[email protected]> wrote:
From: Dario Faggioli <[email protected]>
Quoted 8 lines
Quoted 5 lines
Add a new function to the scheduling class interface. It is called at the end of a context switch, if the prev task is in
TASK_DEADstate.It might be useful for the scheduling classes that want to be notified when one of their task dies, e.g. to perform some cleanup actions.
Nit. s/task/tasks/
Amended, thanks!
Best,
- Juri
Quoted 35 lines
Quoted 34 lines
Signed-off-by Dario Faggioli <[email protected]>Signed-off-by Juri Lelli <[email protected]>--- kernel/sched/core.c | 3 +++ kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+)kernel/sched/core.c +3
-0@@ -1890,6 +1890,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (mm)mmdrop (mm);if (unlikely (prev_state ==TASK_DEAD )) { +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list. kernel/sched/sched.h +1
-0@@ -992,6 +992,7 @@ struct sched_class {void (*set_curr_task) (struct rq *rq);void (*task_tick) (struct rq *rq,struct task_struct *p,int queued);void (*task_fork) (struct task_struct *p); +void (*task_dead) (struct task_struct *p);void (*switched_from) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);
tip-bot for Dario Faggioli
merge bot[tip:sched/core] sched: Add sched_class->task_dead() method
In reply to Juri Lellisched: Add sched_class->task_dead() method
Commit-ID: e6c390f2dfd04c165ce45b0032f73fba85b1f282 Gitweb: http://git.kernel.org/tip/e6c390f2dfd04c165ce45b0032f73fba85b1f282 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:35 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 27 Nov 2013 14:08:50 +0100
sched: Add sched_class->task_dead() method
Add a new function to the scheduling class interface. It is called at the end of a context switch, if the prev task is in TASK_DEAD state.
It will be useful for the scheduling classes that want to be notified when one of their tasks dies, e.g. to perform some cleanup actions, such as SCHED_DEADLINE.
--- kernel/sched/core.c | 3 +++ kernel/sched/sched.h | 1 + 2 files changed, 4 insertions(+)
kernel/sched/core.c +3 -0
@@ -2003,6 +2003,9 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)if (unlikely (prev_state ==TASK_DEAD )) {task_numa_free (prev); +if (prev->sched_class->task_dead) + prev->sched_class->task_dead (prev); +/* * Remove function-return probe instances associated with this * task and put them back on the free list.
kernel/sched/sched.h +1 -0
@@ -1023,6 +1023,7 @@ struct sched_class {void (*set_curr_task) (struct rq *rq);void (*task_tick) (struct rq *rq,struct task_struct *p,int queued);void (*task_fork) (struct task_struct *p); +void (*task_dead) (struct task_struct *p);void (*switched_from) (struct rq *this_rq,struct task_struct *task);void (*switched_to) (struct rq *this_rq,struct task_struct *task);
PATCH 2/14 sched: add extended scheduling interface.
In reply to Juri LelliFrom: Dario Faggioli <[email protected]>
Add the interface bits needed for supporting scheduling algorithms with extended parameters (e.g., SCHED_DEADLINE).
In general, it makes possible to specify a periodic/sporadic task, that executes for a given amount of runtime at each instance, and is scheduled according to the urgency of their own timing constraints, i.e.:
- a (maximum/typical) instance execution time,
- a minimum interval between consecutive instances,
- a time constraint by which each instance must be completed.
Thus, both the data structure that holds the scheduling parameters of the tasks and the system calls dealing with it must be extended. Unfortunately, modifying the existing struct sched_param would break the ABI and result in potentially serious compatibility issues with legacy binaries.
For these reasons, this patch:
- defines the new struct sched_param2, containing all the fields that are necessary for specifying a task in the computational model described above;
- defines and implements the new scheduling related syscalls that manipulate it, i.e., sched_setscheduler2(), sched_setparam2() and sched_getparam2().
Syscalls are introduced for x86 (32 and 64 bits) and ARM only, as a proof of concept and for developing and testing purposes. Making them available on other architectures is straightforward.
Since no "user" for these new parameters is introduced in this patch, the implementation of the new system calls is just identical to their already existing counterpart. Future patches that implement scheduling policies able to exploit the new data structure must also take care of modifying the *2() calls accordingly with their own purposes.
--- arch/arm/include/asm/unistd.h | 2 +- arch/arm/include/uapi/asm/unistd.h | 3 + arch/arm/kernel/calls.S | 3 + arch/x86/syscalls/syscall_32.tbl | 3 + arch/x86/syscalls/syscall_64.tbl | 3 + include/linux/sched.h | 50 ++++++++++++++++ include/linux/syscalls.h | 7 +++ kernel/sched/core.c | 110 +++++++++++++++++++++++++++++++++++- 8 files changed, 177 insertions(+), 4 deletions(-)
arch/arm/include/asm/unistd.h +1 -1
@@ -15,7 +15,7 @@#include <uapi/asm/unistd.h> -#define __NR_syscalls (380) +#define __NR_syscalls (383)#define __ARM_NR_cmpxchg (__ARM_NR_BASE+0x00fff0)#define __ARCH_WANT_STAT64
arch/arm/include/uapi/asm/unistd.h +3 -0
@@ -406,6 +406,9 @@#define __NR_process_vm_writev (__NR_SYSCALL_BASE+377)#define __NR_kcmp (__NR_SYSCALL_BASE+378)#define __NR_finit_module (__NR_SYSCALL_BASE+379) +#define __NR_sched_setscheduler2 (__NR_SYSCALL_BASE+380) +#define __NR_sched_setparam2 (__NR_SYSCALL_BASE+381) +#define __NR_sched_getparam2 (__NR_SYSCALL_BASE+382)/* * This may need to be greater than __NR_last_syscall+1 in order to
arch/arm/kernel/calls.S +3 -0
@@ -389,6 +389,9 @@ CALL(sys_process_vm_writev) CALL(sys_kcmp) CALL(sys_finit_module) +/* 380 */ CALL(sys_sched_setscheduler2) + CALL(sys_sched_setparam2) + CALL(sys_sched_getparam2) #ifndef syscalls_counted .equ syscalls_padding, ((NR_syscalls + 3) & ~3) - NR_syscalls #define syscalls_counted
arch/x86/syscalls/syscall_32.tbl +3 -0
@@ -357,3 +357,6 @@ 348 i386 process_vm_writev sys_process_vm_writev compat_sys_process_vm_writev 349 i386 kcmp sys_kcmp 350 i386 finit_module sys_finit_module +351 i386 sched_setparam2 sys_sched_setparam2 +352 i386 sched_getparam2 sys_sched_getparam2 +353 i386 sched_setscheduler2 sys_sched_setscheduler2
arch/x86/syscalls/syscall_64.tbl +3 -0
@@ -320,6 +320,9 @@ 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp 313 common finit_module sys_finit_module +314 common sched_setparam2 sys_sched_setparam2 +315 common sched_getparam2 sys_sched_getparam2 +316 common sched_setscheduler2 sys_sched_setscheduler2 # # x32-specific system call numbers start at 512 to avoid cache impact
include/linux/sched.h +50 -0
62 lines not on this page · raw message
include/linux/syscalls.h +7 -0
@@ -38,6 +38,7 @@ struct rlimit;struct rlimit64 ;struct rusage ;struct sched_param ; +struct sched_param2 ;struct sel_arg_struct ;struct semaphore ;struct sembuf ;
@@ -277,11 +278,17 @@ asmlinkage long sys_clock_nanosleep(clockid_t which_clock, int flags,asmlinkage long sys_nice (int increment);asmlinkage long sys_sched_setscheduler (pid_t pid,int policy,struct sched_param __user *param); +asmlinkage long sys_sched_setscheduler2 (pid_t pid,int policy, +struct sched_param2 __user *param);asmlinkage long sys_sched_setparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_setparam2 (pid_t pid, +struct sched_param2 __user *param);asmlinkage long sys_sched_getscheduler (pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_getparam2 (pid_t pid, +struct sched_param2 __user *param);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
kernel/sched/core.c +107 -3
155 lines not on this page · raw message
Signature · 1 line
1.7.9.5
Steven Rostedt
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface.
Quoted 14 lines · On Thu, 7 Nov 2013 14:43:36 +0100 Juri Lelli <[email protected]> wrote:
+ * This is reflected by the actual fields of the
sched_param2structure:
+ *+ * @sched_priority task's priority (might still be useful) + * @sched_deadline representative of the task's deadline + * @sched_runtime representative of the task's runtime + * @sched_period representative of the task's period + * @sched_flags for customizing the scheduler behaviour+ *
+ * Given this task model, there are a multiplicity of scheduling algorithms
+ * and policies, that can be used to ensure all the tasks will make their
+ * timing constraints.
+ *
+ * @__unused padding to allow future expansion without ABI issues
+ */
include/linux/sched.h struct sched_param { · line 98
Quoted 3 lines
include/linux/sched.h +3
-0+struct sched_param2 { +int sched_priority; +unsigned int sched_flags;
I'm just thinking, if we are creating a new structure, and this structure already contains u64 elements, why not make sched_flags u64 too? We are now just limiting the total number of possible flags to 32. I'm not sure how many flags will be needed in the future, maybe 32 is good enough, but just something to think about.
Of course you can argue that the int sched_flags matches the int sched_priority leaving out any holes in the structure, which is a legitimate argument.
include/linux/sched.h struct sched_param { · line 103
Quoted 5 lines
include/linux/sched.h +5
-0+u64 sched_runtime; +u64 sched_deadline; +u64 sched_period; + +u64 __unused[12 ];
And in the future, we could use one of these __unused[12] as a sched_flags2;
I'm not saying we should make it u64, just wanted to make sure we are fine with it as 32 for now.
-- Steve
Quoted 2 lines
(quoted diff) +2
-0+}; +
Re: [PATCH 02/14] sched: add extended scheduling interface.
Quoted 41 lines · Steven Rostedt
Quoted 14 lines · On Thu, 7 Nov 2013 14:43:36 +0100 Juri Lelli <[email protected]> wrote:
+ * This is reflected by the actual fields of the
sched_param2structure:
+ *+ * @sched_priority task's priority (might still be useful) + * @sched_deadline representative of the task's deadline + * @sched_runtime representative of the task's runtime + * @sched_period representative of the task's period + * @sched_flags for customizing the scheduler behaviour+ *
+ * Given this task model, there are a multiplicity of scheduling algorithms
+ * and policies, that can be used to ensure all the tasks will make their
+ * timing constraints.
+ *
+ * @__unused padding to allow future expansion without ABI issues
+ */include/linux/sched.h struct sched_param { · line 98
Quoted 3 lines
include/linux/sched.h +3
-0+struct sched_param2 { +int sched_priority; +unsigned int sched_flags;I'm just thinking, if we are creating a new structure, and this structure already contains u64 elements, why not make
sched_flagsu64 too? We are now just limiting the total number of possible flags to 32. I'm not sure how many flags will be needed in the future, maybe 32 is good enough, but just something to think about.Of course you can argue that the int
sched_flagsmatches the intsched_priorityleaving out any holes in the structure, which is a legitimate argument.include/linux/sched.h struct sched_param { · line 103
Quoted 5 lines
include/linux/sched.h +5
-0+u64 sched_runtime; +u64 sched_deadline; +u64 sched_period; + +u64 __unused[12 ];And in the future, we could use one of these __unused[12] as a
sched_flags2;
Quoted 3 lines
I'm not saying we should make it u64, just wanted to make sure we are fine with it as 32 for now.
I'd stick with the current declaration for exactly the points you have made.
What others think?
Thanks,
- Juri
Steven Rostedt
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface.
In reply to Juri LelliQuoted 4 lines · On Thu, 7 Nov 2013 14:43:36 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/core.c +4
-0+static int +do_sched_setscheduler2 (pid_t pid,int policy, +struct sched_param2 __user *param2) +{
kernel/sched/core.c do_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param) · line 3525
Quoted 10 lines
kernel/sched/core.c +10
-0+struct sched_param2 lparam2; +struct task_struct *p; +int retval; + +if (!param2 || pid <0 ) +return -EINVAL ; + +memset (&lparam2,0 ,sizeof (struct sched_param2 )); +if (copy_from_user (&lparam2, param2,sizeof (struct sched_param2 ))) +return -EFAULT ;
Why the memset() before the copy_from_user()? We are copying sizeof(sched_param2) anyway, and should overwrite anything that was on the stack. I'm not aware of any possible leak from copying from userspace. I could understand it if we were copying to userspace.
do_sched_setscheduler() doesn't do that either.
Quoted 76 lines
kernel/sched/core.c +55
-0+ +rcu_read_lock (); + retval = -ESRCH ; + p =find_process_by_pid (pid); +if (p !=NULL ) + retval =sched_setscheduler2 (p, policy, &lparam2); +rcu_read_unlock (); + +return retval; +} +/** * sys_sched_setscheduler - set/change the scheduler policy and RT priority * @pid: the pid in question. @@ -3514,6 +3553,21 @@ SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy, }/** +* sys_sched_setscheduler2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @policy: new policy (could use extended sched_param). +* @param: structure containg the extended parameters. +*/ +SYSCALL_DEFINE3 (sched_setscheduler2, pid_t, pid, int, policy, + struct sched_param2 __user *, param2) +{ +if (policy <0 ) +return -EINVAL ; + +return do_sched_setscheduler2 (pid, policy, param2); +} + +/** * sys_sched_setparam - set/change the RT priority of a thread * @pid: the pid in question. * @param: structure containing the new RT priority. @@ -3526,6 +3580,17 @@ SYSCALL_DEFINE2(sched_setparam, pid_t, pid, struct sched_param __user *, param) }/** +* sys_sched_setparam2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @param2: structure containing the extended parameters. +*/ +SYSCALL_DEFINE2 (sched_setparam2, pid_t, pid, + struct sched_param2 __user *, param2) +{ +return do_sched_setscheduler2 (pid,-1 , param2); +} + +/** * sys_sched_getscheduler - get the policy (scheduling class) of a thread * @pid: the pid in question. * @@ -3595,6 +3660,45 @@ out_unlock:return retval; } +/** +* sys_sched_getparam2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @param2: structure containing the extended parameters. +*/ +SYSCALL_DEFINE2 (sched_getparam2, pid_t, pid, + struct sched_param2 __user *, param2) +{ +struct sched_param2 lp; +struct task_struct *p; +int retval; + +if (!param2 || pid <0 ) +return -EINVAL ; + +rcu_read_lock (); + p =find_process_by_pid (pid); + retval = -ESRCH ;
kernel/sched/core.c out_unlock: · line 3690
Quoted 10 lines
kernel/sched/core.c +10
-0+if (!p) + goto out_unlock; + + retval =security_task_getscheduler (p); +if (retval) + goto out_unlock; + + lp.sched_priority = p->rt_priority; +rcu_read_unlock (); +
OK, now we are missing the memset(). This does leak info, as lp never was set to zero, it just contains anything on the stack, and the only value you updated was sched_priority. We just copied to user memory from the kernel stack.
-- Steve
Quoted 14 lines
kernel/sched/core.c +11
-0+ retval =copy_to_user (param2, &lp, +sizeof (struct sched_param2 )) ? -EFAULT :0 ; + +return retval; + +out_unlock: +rcu_read_unlock (); +return retval; + +} +long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) {cpumask_var_t cpus_allowed, new_mask;
Re: [PATCH 02/14] sched: add extended scheduling interface.
Quoted 109 lines · Steven Rostedt
Quoted 4 lines · On Thu, 7 Nov 2013 14:43:36 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/core.c +4
-0+static int +do_sched_setscheduler2 (pid_t pid,int policy, +struct sched_param2 __user *param2) +{kernel/sched/core.c do_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param) · line 3525
Quoted 10 lines
kernel/sched/core.c +10
-0+struct sched_param2 lparam2; +struct task_struct *p; +int retval; + +if (!param2 || pid <0 ) +return -EINVAL ; + +memset (&lparam2,0 ,sizeof (struct sched_param2 )); +if (copy_from_user (&lparam2, param2,sizeof (struct sched_param2 ))) +return -EFAULT ;Why the
memset()before thecopy_from_user()? We are copyingsizeof(sched_param2)anyway, and should overwrite anything that was on the stack. I'm not aware of any possible leak from copying from userspace. I could understand it if we were copying to userspace.
do_sched_setscheduler()doesn't do that either.Quoted 83 lines
kernel/sched/core.c +62
-0+ +rcu_read_lock (); + retval = -ESRCH ; + p =find_process_by_pid (pid); +if (p !=NULL ) + retval =sched_setscheduler2 (p, policy, &lparam2); +rcu_read_unlock (); + +return retval; +} +/** * sys_sched_setscheduler - set/change the scheduler policy and RT priority * @pid: the pid in question. @@ -3514,6 +3553,21 @@ SYSCALL_DEFINE3(sched_setscheduler, pid_t, pid, int, policy, }/** +* sys_sched_setscheduler2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @policy: new policy (could use extended sched_param). +* @param: structure containg the extended parameters. +*/ +SYSCALL_DEFINE3 (sched_setscheduler2, pid_t, pid, int, policy, + struct sched_param2 __user *, param2) +{ +if (policy <0 ) +return -EINVAL ; + +return do_sched_setscheduler2 (pid, policy, param2); +} + +/** * sys_sched_setparam - set/change the RT priority of a thread * @pid: the pid in question. * @param: structure containing the new RT priority. @@ -3526,6 +3580,17 @@ SYSCALL_DEFINE2(sched_setparam, pid_t, pid, struct sched_param __user *, param) }/** +* sys_sched_setparam2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @param2: structure containing the extended parameters. +*/ +SYSCALL_DEFINE2 (sched_setparam2, pid_t, pid, + struct sched_param2 __user *, param2) +{ +return do_sched_setscheduler2 (pid,-1 , param2); +} + +/** * sys_sched_getscheduler - get the policy (scheduling class) of a thread * @pid: the pid in question. * @@ -3595,6 +3660,45 @@ out_unlock:return retval; } +/** +* sys_sched_getparam2 - same as above, but with extended sched_param +* @pid: the pid in question. +* @param2: structure containing the extended parameters. +*/ +SYSCALL_DEFINE2 (sched_getparam2, pid_t, pid, + struct sched_param2 __user *, param2) +{ +struct sched_param2 lp; +struct task_struct *p; +int retval; + +if (!param2 || pid <0 ) +return -EINVAL ; + +rcu_read_lock (); + p =find_process_by_pid (pid); + retval = -ESRCH ; +if (!p) + goto out_unlock; + + retval =security_task_getscheduler (p); +if (retval) + goto out_unlock; +
Quoted 8 lines
kernel/sched/core.c out_unlock: · line 3690
Quoted 3 lines
kernel/sched/core.c +3
-0+ lp.sched_priority = p->rt_priority; +rcu_read_unlock (); +OK, now we are missing the
memset(). This does leak info, as lp never was set to zero, it just contains anything on the stack, and the only value you updated wassched_priority. We just copied to user memory from the kernel stack.
Right! memset() moved:
@@ -3779,7 +3779,6 @@ do_sched_setscheduler2(pid_t pid, int policy,
if (!param2 || pid < 0)
return -EINVAL;- memset(&lparam2, 0, sizeof(struct sched_param2));
if (copy_from_user(&lparam2, param2, sizeof(struct sched_param2)))
return -EFAULT;
@@ -3937,6 +3936,8 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid,
if (!param2 || pid < 0)
return -EINVAL;+ memset(&lp, 0, sizeof(struct sched_param2));
+
rcu_read_lock();
p = find_process_by_pid(pid);
retval = -ESRCH;Thanks,
- Juri
Quoted 16 lines
Quoted 15 lines
kernel/sched/core.c +11
-0+ retval =copy_to_user (param2, &lp, +sizeof (struct sched_param2 )) ? -EFAULT :0 ; + +return retval; + +out_unlock: +rcu_read_unlock (); +return retval; + +} +long sched_setaffinity (pid_t pid,const struct cpumask *in_mask) {cpumask_var_t cpus_allowed, new_mask;
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
In reply to Juri LelliReview
Quoted 11 lines · Juri Lelli
+ * @__unused padding to allow future expansion without ABI issues
+ */(quoted diff) +9
-0+struct sched_param2 { + int sched_priority; + unsigned int sched_flags; + u64 sched_runtime; + u64 sched_deadline; + u64 sched_period; + + u64 __unused[12]; +};
So this really needs to use s32/u32.
But the bigger problem is that this is a rather dumb ABI which copies 128 bytes unconditionally. That will be enough up to the point we run out of it.
Instead I think what we want is a simple yet extensible ABI where the size of the parameters is part of the structure itself - which acts as a natural 'version'.
We already have such an extensible syscall implementation, see sys_perf_event_open in kernel/events/core.c: bits of which could be factored out to make all this easier and more robust.
To make this auto-versioning property more apparent I'd suggest a rename of the syscalls as well: sys_sched_setattr(), sys_sched_getattr(), or so.
The compatibility principle is: there's a 'struct sched_attr' with a sched_attr::size field (plus the fields above, and no padding). The sched_attr::size field us the structure size user-space expects.
There are 3 main compatibility cases:
- the kernel's 'sizeof sched_attr' is equal to sched_attr:size: the kernel version and user-space version matches, it's a straight ABI in this case with full functionality.
- the kernel's 'sizeof sched_attr' is larger than sched_attr::size [the kernel is newer than what user-space was built for], in this case the kernel assumes that all remaining values are zero and acts accordingly.
- the kernel's 'sizeof sched_attr' is smaller than sched_attr::size [the kernel is older than what user-space was built for]. In this case the kernel should return -ENOSYS if any of the additional fields are nonzero. If those are all zero then it will work as if a smaller structure was passed in.
This ensures maximal upwards and downwards compatibility and keeps the syscall ABI compat yet extensible. The ABI is the quickest when tool version matches kernel version - but that's the typical case for distros. Yet even the mismatching versions work fine and the ABI is kept.
( See kernel/events/core.c for more details. Some of the helpers there
should be factored out to allow easier support for such syscalls. )
Note that I did a few other small fixes to the changelog and to the code as well - see the patch attached below - please work based on this version.
Thanks,
Ingo
======================>
Subject: sched: Add 3 new scheduler syscalls to support an extended scheduling parameters ABI
From: Dario Faggioli <[email protected]>
Date: Thu, 7 Nov 2013 14:43:36 +0100
Add the syscalls needed for supporting scheduling algorithms with extended scheduling parameters (e.g., SCHED_DEADLINE).
In general, it makes possible to specify a periodic/sporadic task, that executes for a given amount of runtime at each instance, and is scheduled according to the urgency of their own timing constraints, i.e.:
- a (maximum/typical) instance execution time,
- a minimum interval between consecutive instances,
- a time constraint by which each instance must be completed.
Thus, both the data structure that holds the scheduling parameters of the tasks and the system calls dealing with it must be extended. Unfortunately, modifying the existing struct sched_param would break the ABI and result in potentially serious compatibility issues with legacy binaries.
For these reasons, this patch:
- defines the new struct sched_param2, containing all the fields that are necessary for specifying a task in the computational model described above;
- defines and implements the new scheduling related syscalls that manipulate it, i.e., sched_setscheduler2(), sched_setparam2() and sched_getparam2().
Syscalls are introduced for x86 (32 and 64 bits) and ARM only, as a proof of concept and for developing and testing purposes. Making them available on other architectures is straightforward.
Since no "user" for these new parameters is introduced in this patch, the implementation of the new system calls is just identical to their already existing counterpart. Future patches that implement scheduling policies able to exploit the new data structure must also take care of modifying the *2() calls accordingly with their own purposes.
--- arch/arm/include/asm/unistd.h | 2 arch/arm/include/uapi/asm/unistd.h | 3 + arch/arm/kernel/calls.S | 3 + arch/x86/syscalls/syscall_32.tbl | 3 + arch/x86/syscalls/syscall_64.tbl | 3 + include/linux/sched.h | 50 +++++++++++++++++ include/linux/syscalls.h | 7 ++ kernel/sched/core.c | 106 +++++++++++++++++++++++++++++++++++-- 8 files changed, 173 insertions(+), 4 deletions(-)
arch/arm/include/asm/unistd.h +1 -1
@@ -15,7 +15,7 @@#include <uapi/asm/unistd.h> -#define __NR_syscalls (380) +#define __NR_syscalls (383)#define __ARM_NR_cmpxchg (__ARM_NR_BASE+0x00fff0)#define __ARCH_WANT_STAT64
arch/arm/include/uapi/asm/unistd.h +3 -0
@@ -406,6 +406,9 @@#define __NR_process_vm_writev (__NR_SYSCALL_BASE+377)#define __NR_kcmp (__NR_SYSCALL_BASE+378)#define __NR_finit_module (__NR_SYSCALL_BASE+379) +#define __NR_sched_setscheduler2 (__NR_SYSCALL_BASE+380) +#define __NR_sched_setparam2 (__NR_SYSCALL_BASE+381) +#define __NR_sched_getparam2 (__NR_SYSCALL_BASE+382)/* * This may need to be greater than __NR_last_syscall+1 in order to
arch/arm/kernel/calls.S +3 -0
@@ -389,6 +389,9 @@ CALL(sys_process_vm_writev) CALL(sys_kcmp) CALL(sys_finit_module) +/* 380 */ CALL(sys_sched_setscheduler2) + CALL(sys_sched_setparam2) + CALL(sys_sched_getparam2) #ifndef syscalls_counted .equ syscalls_padding, ((NR_syscalls + 3) & ~3) - NR_syscalls #define syscalls_counted
arch/x86/syscalls/syscall_32.tbl +3 -0
@@ -357,3 +357,6 @@ 348 i386 process_vm_writev sys_process_vm_writev compat_sys_process_vm_writev 349 i386 kcmp sys_kcmp 350 i386 finit_module sys_finit_module +351 i386 sched_setparam2 sys_sched_setparam2 +352 i386 sched_getparam2 sys_sched_getparam2 +353 i386 sched_setscheduler2 sys_sched_setscheduler2
arch/x86/syscalls/syscall_64.tbl +3 -0
@@ -320,6 +320,9 @@ 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp 313 common finit_module sys_finit_module +314 common sched_setparam2 sys_sched_setparam2 +315 common sched_getparam2 sys_sched_getparam2 +316 common sched_setscheduler2 sys_sched_setscheduler2 # # x32-specific system call numbers start at 512 to avoid cache impact
include/linux/sched.h +50 -0
62 lines not on this page · raw message
include/linux/syscalls.h +7 -0
@@ -38,6 +38,7 @@ struct rlimit;struct rlimit64 ;struct rusage ;struct sched_param ; +struct sched_param2 ;struct sel_arg_struct ;struct semaphore ;struct sembuf ;
@@ -277,11 +278,17 @@ asmlinkage long sys_clock_nanosleep(clocasmlinkage long sys_nice (int increment);asmlinkage long sys_sched_setscheduler (pid_t pid,int policy,struct sched_param __user *param); +asmlinkage long sys_sched_setscheduler2 (pid_t pid,int policy, +struct sched_param2 __user *param);asmlinkage long sys_sched_setparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_setparam2 (pid_t pid, +struct sched_param2 __user *param);asmlinkage long sys_sched_getscheduler (pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_getparam2 (pid_t pid, +struct sched_param2 __user *param);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
kernel/sched/core.c +103 -3
151 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 10 lines · Ingo Molnar
There are 3 main compatibility cases:
- the kernel's 'sizeof
sched_attr' is equal to sched_attr:size: the kernel version and user-space version matches, it's a straight ABI in this case with full functionality.- the kernel's 'sizeof
sched_attr' is larger thansched_attr::size[the kernel is newer than what user-space was built for], in this case the kernel assumes that all remaining values are zero and acts accordingly.
It also needs to fail sched_getparam() when any of the fields that do not fit in the smaller struct provided are !0.
Quoted 5 lines
- the kernel's 'sizeof
sched_attr' is smaller thansched_attr::size[the kernel is older than what user-space was built for]. In this case the kernel should return-ENOSYSif any of the additional fields are nonzero. If those are all zero then it will work as if a smaller structure was passed in.
So the problem I see with this one is that because you're allowed to call sched_setparam() or whatever it will be called next on another task; a task can very easily fail its sched_getparam() call.
Suppose the application is 'old' and only supports a subset of the fields; but its wants to get, modify and set its params. This will work as long nothing will set anything it doesn't know about.
As soon as some external entity -- say a sysad using schedtool -- sets a param field it doesn't support the get, modify, set routing completely fails.
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 25 lines · Peter Zijlstra
Quoted 10 lines · Ingo Molnar
There are 3 main compatibility cases:
- the kernel's 'sizeof
sched_attr' is equal to sched_attr:size: the kernel version and user-space version matches, it's a straight ABI in this case with full functionality.- the kernel's 'sizeof
sched_attr' is larger thansched_attr::size[the kernel is newer than what user-space was built for], in this case the kernel assumes that all remaining values are zero and acts accordingly.It also needs to fail
sched_getparam()when any of the fields that do not fit in the smaller struct provided are !0.Quoted 5 lines
- the kernel's 'sizeof
sched_attr' is smaller thansched_attr::size[the kernel is older than what user-space was built for]. In this case the kernel should return-ENOSYSif any of the additional fields are nonzero. If those are all zero then it will work as if a smaller structure was passed in.So the problem I see with this one is that because you're allowed to call
sched_setparam()or whatever it will be called next on another task; a task can very easily fail itssched_getparam()call.
Quoted 7 lines
Suppose the application is 'old' and only supports a subset of the fields; but its wants to get, modify and set its params. This will work as long nothing will set anything it doesn't know about.
As soon as some external entity -- say a sysad using schedtool -- sets a param field it doesn't support the get, modify, set routing completely fails.
There are two approaches to this that I can see:
1)
allow partial information to be returned to user-space, for existing input parameters. The new fields won't be displayed, but the tool doesn't know about them anyway so it's OK. The tool can still display all the other existing parameters.
2)
Return -ENOSYS if the 'extra' fields are nonzero. In this case the usual case of old tooling + new kernel will still work just fine, because old tooling won't set the new fields to any non-default (nonzero) values. In the 'mixed' case old tooling will not be able to change/display those fields.
I tend to lean towards #1. What do you think?
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 12 lines · Ingo Molnar
Quoted 11 lines
So the problem I see with this one is that because you're allowed to call
sched_setparam()or whatever it will be called next on another task; a task can very easily fail itssched_getparam()call.Suppose the application is 'old' and only supports a subset of the fields; but its wants to get, modify and set its params. This will work as long nothing will set anything it doesn't know about.
As soon as some external entity -- say a sysad using schedtool -- sets a param field it doesn't support the get, modify, set routing completely fails.
Quoted 8 lines
There are two approaches to this that I can see:
1)
allow partial information to be returned to user-space, for existing input parameters. The new fields won't be displayed, but the tool doesn't know about them anyway so it's OK. The tool can still display all the other existing parameters.
But suppose a task simply wants to lower/raise its static (FIFO) priority and does:
sched_getparam(¶ms);
params.prio += 1;sched_setparam(¶ms);
If anything outside of the known param fields was set, we just silently lost it, for the setparam() call will fill out 0s for the unprovided fields.
Quoted 9 lines
2)
Return
-ENOSYSif the 'extra' fields are nonzero. In this case the usual case of old tooling + new kernel will still work just fine, because old tooling won't set the new fields to any non-default (nonzero) values. In the 'mixed' case old tooling will not be able to change/display those fields.I tend to lean towards #1. What do you think?
As per the above that can result in silent unexpected behavioural changes.
I'd much rather be explicit and break hard; so 2).
So mixing new tools (schedtool, chrt etc) and old apps will give pain, but at least not silent surprises.
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 43 lines · Peter Zijlstra
Quoted 12 lines · Ingo Molnar
Quoted 11 lines
So the problem I see with this one is that because you're allowed to call
sched_setparam()or whatever it will be called next on another task; a task can very easily fail itssched_getparam()call.Suppose the application is 'old' and only supports a subset of the fields; but its wants to get, modify and set its params. This will work as long nothing will set anything it doesn't know about.
As soon as some external entity -- say a sysad using schedtool -- sets a param field it doesn't support the get, modify, set routing completely fails.
Quoted 8 lines
There are two approaches to this that I can see:
1)
allow partial information to be returned to user-space, for existing input parameters. The new fields won't be displayed, but the tool doesn't know about them anyway so it's OK. The tool can still display all the other existing parameters.
But suppose a task simply wants to lower/raise its static (FIFO) priority and does:
sched_getparam(¶ms);
params.prio += 1;sched_setparam(¶ms);If anything outside of the known param fields was set, we just silently lost it, for the
setparam()call will fill out 0s for the unprovided fields.Quoted 9 lines
2)
Return
-ENOSYSif the 'extra' fields are nonzero. In this case the usual case of old tooling + new kernel will still work just fine, because old tooling won't set the new fields to any non-default (nonzero) values. In the 'mixed' case old tooling will not be able to change/display those fields.I tend to lean towards #1. What do you think?
Quoted 7 lines
As per the above that can result in silent unexpected behavioural changes.
I'd much rather be explicit and break hard; so 2).
So mixing new tools (schedtool, chrt etc) and old apps will give pain, but at least not silent surprises.
You are right, I concur.
Thanks,
Ingo
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 49 lines · Ingo Molnar
Quoted 46 lines · Peter Zijlstra
Quoted 12 lines · Ingo Molnar
Quoted 11 lines
So the problem I see with this one is that because you're allowed to call
sched_setparam()or whatever it will be called next on another task; a task can very easily fail itssched_getparam()call.Suppose the application is 'old' and only supports a subset of the fields; but its wants to get, modify and set its params. This will work as long nothing will set anything it doesn't know about.
As soon as some external entity -- say a sysad using schedtool -- sets a param field it doesn't support the get, modify, set routing completely fails.
Quoted 8 lines
There are two approaches to this that I can see:
1)
allow partial information to be returned to user-space, for existing input parameters. The new fields won't be displayed, but the tool doesn't know about them anyway so it's OK. The tool can still display all the other existing parameters.
But suppose a task simply wants to lower/raise its static (FIFO) priority and does:
sched_getparam(¶ms);
params.prio += 1;sched_setparam(¶ms);If anything outside of the known param fields was set, we just silently lost it, for the
setparam()call will fill out 0s for the unprovided fields.Quoted 9 lines
2)
Return
-ENOSYSif the 'extra' fields are nonzero. In this case the usual case of old tooling + new kernel will still work just fine, because old tooling won't set the new fields to any non-default (nonzero) values. In the 'mixed' case old tooling will not be able to change/display those fields.I tend to lean towards #1. What do you think?
As per the above that can result in silent unexpected behavioural changes.
Quoted 7 lines
Quoted 4 lines
I'd much rather be explicit and break hard; so 2).
So mixing new tools (schedtool, chrt etc) and old apps will give pain, but at least not silent surprises.
You are right, I concur.
Ok, I came up with what follows. I slightly tested it, but this is intended to be my first try to understand what you asking for :).
Question:
I do (copying from perf_copy_attr())
if (!size) /* abi compat */ size = SCHED_ATTR_SIZE_VER0;
Shouldn't I return some type of error in this case? We know that there is nothing before VER0 to which we have to be bwd compatible to. And with this my "old" schedtool still works since he doesn't know about size...
Comments?
Thanks,
- Juri
Author: Juri Lelli <[email protected]> Date: Thu Nov 28 11:07:47 2013 +0100
fixup: ABI changed, builds and runs
arch/arm/include/uapi/asm/unistd.h +2 -2
@@ -407,8 +407,8 @@#define __NR_kcmp (__NR_SYSCALL_BASE+378)#define __NR_finit_module (__NR_SYSCALL_BASE+379)#define __NR_sched_setscheduler2 (__NR_SYSCALL_BASE+380) -#define __NR_sched_setparam2 (__NR_SYSCALL_BASE+381) -#define __NR_sched_getparam2 (__NR_SYSCALL_BASE+382) +#define __NR_sched_setattr (__NR_SYSCALL_BASE+381) +#define __NR_sched_getattr (__NR_SYSCALL_BASE+382)/* * This may need to be greater than __NR_last_syscall+1 in order to
arch/arm/kernel/calls.S +2 -2
@@ -390,8 +390,8 @@ CALL(sys_kcmp) CALL(sys_finit_module) /* 380 */ CALL(sys_sched_setscheduler2) - CALL(sys_sched_setparam2) - CALL(sys_sched_getparam2) + CALL(sys_sched_setattr) + CALL(sys_sched_getattr) #ifndef syscalls_counted .equ syscalls_padding, ((NR_syscalls + 3) & ~3) - NR_syscalls #define syscalls_counted
arch/x86/syscalls/syscall_32.tbl +2 -2
@@ -357,6 +357,6 @@ 348 i386 process_vm_writev sys_process_vm_writev compat_sys_process_vm_writev 349 i386 kcmp sys_kcmp 350 i386 finit_module sys_finit_module -351 i386 sched_setparam2 sys_sched_setparam2 -352 i386 sched_getparam2 sys_sched_getparam2 +351 i386 sched_setattr sys_sched_setattr +352 i386 sched_getattr sys_sched_getattr 353 i386 sched_setscheduler2 sys_sched_setscheduler2
arch/x86/syscalls/syscall_64.tbl +2 -2
@@ -320,8 +320,8 @@ 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp 313 common finit_module sys_finit_module -314 common sched_setparam2 sys_sched_setparam2 -315 common sched_getparam2 sys_sched_getparam2 +314 common sched_setattr sys_sched_setattr +315 common sched_getattr sys_sched_getattr 316 common sched_setscheduler2 sys_sched_setscheduler2 #
include/linux/sched.h +11 -7
@@ -56,6 +56,8 @@ struct sched_param {#include <asm/processor.h> +#define SCHED_ATTR_SIZE_VER0 40/* sizeof first published struct */ +/* * Extended scheduling parameters data structure. *
@@ -67,7 +69,7 @@ struct sched_param {* the tasks may be useful for a wide variety of application fields, e.g., * multimedia, streaming, automation and control, and many others. * -* This variant (sched_param2) is meant at describing a so-called +* This variant (sched_attr) is meant at describing a so-called * sporadic time-constrained task. In such model a task is specified by: * - the activation period or minimum instance inter-arrival time; * - the maximum (or average, depending on the actual scheduling
@@ -80,28 +82,30 @@ struct sched_param {* than the runtime and must be completed by time instant t equal to * the instance activation time + the deadline. * -* This is reflected by the actual fields of the sched_param2 structure: +* This is reflected by the actual fields of the sched_attr structure: * * @sched_priority task's priority (might still be useful) +* @sched_flags for customizing the scheduler behaviour * @sched_deadline representative of the task's deadline * @sched_runtime representative of the task's runtime * @sched_period representative of the task's period -* @sched_flags for customizing the scheduler behaviour * * Given this task model, there are a multiplicity of scheduling algorithms * and policies, that can be used to ensure all the tasks will make their * timing constraints. * -* @__unused padding to allow future expansion without ABI issues +* @size size of the structure, for fwd/bwd compat. */ -struct sched_param2 { +struct sched_attr {int sched_priority;unsigned int sched_flags;u64 sched_runtime;u64 sched_deadline;u64 sched_period; +u32 size; -u64 __unused[12 ]; +/* Align to u64. */ +u32 __reserved; };struct exec_domain ;
@@ -2011,7 +2015,7 @@ extern int sched_setscheduler(struct task_struct *, int,extern int sched_setscheduler_nocheck (struct task_struct *,int ,const struct sched_param *);extern int sched_setscheduler2 (struct task_struct *,int , -const struct sched_param2 *); +const struct sched_attr *);extern struct task_struct *idle_task (int cpu);/** * is_idle_task - is the specified task an idle task?
include/linux/syscalls.h +5 -5
@@ -279,16 +279,16 @@ asmlinkage long sys_nice(int increment);asmlinkage long sys_sched_setscheduler (pid_t pid,int policy,struct sched_param __user *param);asmlinkage long sys_sched_setscheduler2 (pid_t pid,int policy, -struct sched_param2 __user *param); +struct sched_attr __user *attr);asmlinkage long sys_sched_setparam (pid_t pid,struct sched_param __user *param); -asmlinkage long sys_sched_setparam2 (pid_t pid, -struct sched_param2 __user *param); +asmlinkage long sys_sched_setattr (pid_t pid, +struct sched_attr __user *attr);asmlinkage long sys_sched_getscheduler (pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param); -asmlinkage long sys_sched_getparam2 (pid_t pid, -struct sched_param2 __user *param); +asmlinkage long sys_sched_getattr (pid_t pid, +struct sched_attr __user *attr);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
kernel/sched/core.c +95 -27
210 lines not on this page · raw message
Peter Zijlstra
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 17 lines · Juri Lelli
kernel/sched/core.c +5
-2+SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) { -struct sched_param2 lp; +struct sched_attr lp;struct task_struct *p;int retval; -if (!param2 || pid <0 ) +if (!attr || pid <0 )return -EINVAL ; +memset (&lp,0 ,sizeof (struct sched_attr )); +rcu_read_lock (); p =find_process_by_pid (pid); retval = -ESRCH ;
kernel/sched/core.c SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param · line 3501
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -3427,7 +3495,7 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (param2, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ;return retval; out_unlock:
So this side needs a bit more care; suppose the kernel has a larger attr than userspace knows about.
What would make more sense; add another syscall argument with the userspace sizeof(struct sched_attr), or expect userspace to initialize attr->size to the right value before calling sched_getattr() ?
To me the extra argument makes more sense; that is:
struct sched_attr attr;
ret = sched_getattr(0, &attr, sizeof(attr));
seems like a saner thing than:
struct sched_attr attr = { .size = sizeof(attr), };
ret = sched_getattr(0, &attr);Mostly because the former has a clear separation between input and output arguments, whereas for the second form the attr argument is both input and output.
Ingo?
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 47 lines · Peter Zijlstra
Quoted 17 lines · Juri Lelli
kernel/sched/core.c +5
-2+SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) { -struct sched_param2 lp; +struct sched_attr lp;struct task_struct *p;int retval; -if (!param2 || pid <0 ) +if (!attr || pid <0 )return -EINVAL ; +memset (&lp,0 ,sizeof (struct sched_attr )); +rcu_read_lock (); p =find_process_by_pid (pid); retval = -ESRCH ;kernel/sched/core.c SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param · line 3501
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -3427,7 +3495,7 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (param2, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ;return retval; out_unlock:So this side needs a bit more care; suppose the kernel has a larger attr than userspace knows about.
What would make more sense; add another syscall argument with the userspace
sizeof(struct sched_attr), or expect userspace to initializeattr->sizeto the right value before callingsched_getattr()?To me the extra argument makes more sense; that is:
struct
sched_attrattr;ret = sched_getattr(0, &attr, sizeof(attr));seems like a saner thing than:
struct sched_attr attr = { .size = sizeof(attr), }; ret = sched_getattr(0, &attr);
Quoted 5 lines
Mostly because the former has a clear separation between input and output arguments, whereas for the second form the attr argument is both input and output.
Ingo?
I suppose so - in the sys_perf_event_open() case we ran out of arguments, so attr::size was the only sane way to do it.
[ Btw., perf events side note: for completeness and for debugging we
probably want to add a sys_perf_event_get() method as well, to recover the attributes of an existing event. Unidirectional APIs are not very nice. ]
Thanks,
Ingo
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 50 lines · Ingo Molnar
Quoted 47 lines · Peter Zijlstra
Quoted 17 lines · Juri Lelli
(quoted diff) +5
-2+SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) { - struct sched_param2 lp; + struct sched_attr lp; struct task_struct *p; int retval; - if (!param2 || pid < 0) + if (!attr || pid < 0) return -EINVAL; + memset(&lp, 0, sizeof(struct sched_attr)); + rcu_read_lock(); p = find_process_by_pid(pid); retval = -ESRCH;Review SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param · line 3501
Quoted 8 lines
(quoted diff) +1
-1@@ -3427,7 +3495,7 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param lp.sched_priority = p->rt_priority; rcu_read_unlock(); - retval = copy_to_user(param2, &lp, sizeof(lp)) ? -EFAULT : 0; + retval = copy_to_user(attr, &lp, sizeof(lp)) ? -EFAULT : 0; return retval; out_unlock:So this side needs a bit more care; suppose the kernel has a larger attr than userspace knows about.
What would make more sense; add another syscall argument with the userspace
sizeof(struct sched_attr), or expect userspace to initializeattr->sizeto the right value before callingsched_getattr()?To me the extra argument makes more sense; that is:
struct
sched_attrattr;ret = sched_getattr(0, &attr, sizeof(attr));seems like a saner thing than:
struct sched_attr attr = { .size = sizeof(attr), }; ret = sched_getattr(0, &attr);
Quoted 9 lines
Quoted 5 lines
Mostly because the former has a clear separation between input and output arguments, whereas for the second form the attr argument is both input and output.
Ingo?
I suppose so - in the
sys_perf_event_open()case we ran out of arguments, soattr::sizewas the only sane way to do it.
Ok, I modified it like this:
------------------------------------------------------------
Subject: [PATCH] fixup: add checks for sys_sched_getattr
Add an extra argument to the syscall with the userspace sizeof(struct sched_attr) to be able to handle situations when the kernel has a larger attr than userspace knows about.
--- include/linux/syscalls.h | 3 ++- kernel/sched/core.c | 55 ++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 5 deletions(-)
include/linux/syscalls.h +2 -1
@@ -288,7 +288,8 @@ asmlinkage long sys_sched_getscheduler(pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param);asmlinkage long sys_sched_getattr (pid_t pid, -struct sched_attr __user *attr); +struct sched_attr __user *attr, +unsigned int size);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
kernel/sched/core.c +51 -4
83 lines not on this page · raw message
-----------------------------------------------------------
Do we need to make sched_setattr symmetrical, or, since the user has to fill the fields anyway, we leave it as is?
Thanks,
- Juri
Steven Rostedt
maintainerRe: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 103 lines · On Tue, 03 Dec 2013 17:13:44 +0100 Juri Lelli <[email protected]> wrote:
Quoted 50 lines · Ingo Molnar
Quoted 47 lines · Peter Zijlstra
Quoted 17 lines · Juri Lelli
kernel/sched/core.c +5
-2+SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) { -struct sched_param2 lp; +struct sched_attr lp;struct task_struct *p;int retval; -if (!param2 || pid <0 ) +if (!attr || pid <0 )return -EINVAL ; +memset (&lp,0 ,sizeof (struct sched_attr )); +rcu_read_lock (); p =find_process_by_pid (pid); retval = -ESRCH ;kernel/sched/core.c SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param · line 3501
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -3427,7 +3495,7 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (param2, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ;return retval; out_unlock:So this side needs a bit more care; suppose the kernel has a larger attr than userspace knows about.
What would make more sense; add another syscall argument with the userspace
sizeof(struct sched_attr), or expect userspace to initializeattr->sizeto the right value before callingsched_getattr()?To me the extra argument makes more sense; that is:
struct
sched_attrattr;ret = sched_getattr(0, &attr, sizeof(attr));seems like a saner thing than:
struct sched_attr attr = { .size = sizeof(attr), }; ret = sched_getattr(0, &attr);Quoted 9 lines
Quoted 5 lines
Mostly because the former has a clear separation between input and output arguments, whereas for the second form the attr argument is both input and output.
Ingo?
I suppose so - in the
sys_perf_event_open()case we ran out of arguments, soattr::sizewas the only sane way to do it.Ok, I modified it like this:
------------------------------------------------------------
Subject: [PATCH] fixup: add checks forsys_sched_getattrAdd an extra argument to the syscall with the userspace
sizeof(struct sched_attr)to be able to handle situations when the kernel has a larger attr than userspace knows about.--- include/linux/syscalls.h | 3 ++- kernel/sched/core.c | 55 ++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 5 deletions(-)include/linux/syscalls.h +2
-1@@ -288,7 +288,8 @@ asmlinkage long sys_sched_getscheduler(pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param);asmlinkage long sys_sched_getattr (pid_t pid, -struct sched_attr __user *attr); +struct sched_attr __user *attr, +unsigned int size);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,kernel/sched/core.c +1
-1@@ -3507,7 +3507,7 @@ do_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param) }/* -* Mimics kerner/events/core.c perf_copy_attr(). +* Mimics kernel/events/core.c perf_copy_attr(). */ static int sched_copy_attr (struct sched_attr __user *uattr,struct sched_attr *attr)@@ -3726,18 +3726,65 @@ out_unlock:return retval;
kernel/sched/core.c out_unlock: · line 3736
Quoted 10 lines
kernel/sched/core.c +8
-0} +static int sched_read_attr (struct sched_attr __user *uattr, +struct sched_attr *attr, +unsigned int size, +unsigned int usize) +{ +int ret; + +if (!access_ok (VERIFY_WRITE , uattr,SCHED_ATTR_SIZE_VER0 ))
We want to verify from uattr to usize, right? As that is what we are writing to.
kernel/sched/core.c out_unlock: · line 3742
Quoted 6 lines
kernel/sched/core.c +6
-0+return -EFAULT ; + +/* +* zero the full structure, so that a short copy will be nice. +*/ +memset (uattr,0 ,sizeof (*uattr));
Wait! We can't write to user space like this, not to mention that usize may even be less than sizeof(struct sched_attr).
-- Steve
Quoted 67 lines
kernel/sched/core.c +36
-4+ +/* +* If we're handed a smaller struct than we know of, +* ensure all the unknown bits are 0 - i.e. old +* user-space does not get uncomplete information. +*/ +if (usize <sizeof (*attr)) { +unsigned char *addr; +unsigned char *end; + + addr = (void *)attr + usize; + end = (void *)attr + size; + +for (; addr < end; addr++) +if (*addr) + goto err_size; + } + + ret =copy_to_user (uattr, attr, usize); +if (ret) +return -EFAULT ; + +out: +return ret; + +err_size: + ret = -E2BIG ; + goto out; +} +/** * sys_sched_getattr - same as above, but with extended "sched_param" * @pid: the pid in question. * @attr: structure containing the extended parameters. +* @size: sizeof(attr) for fwd/bwd comp. */ -SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) +SYSCALL_DEFINE3(sched_getattr, pid_t, pid, struct sched_attr __user *, attr, + unsigned int, size) {struct sched_attr lp;struct task_struct *p;int retval; -if (!attr || pid <0 ) +if (!attr || pid <0 || size >PAGE_SIZE || + size <SCHED_ATTR_SIZE_VER0 )return -EINVAL ;memset (&lp,0 ,sizeof (struct sched_attr ));@@ -3758,7 +3805,7 @@ SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =sched_read_attr (attr, &lp,sizeof (lp), size);return retval; out_unlock: -----------------------------------------------------------Do we need to make
sched_setattrsymmetrical, or, since the user has to fill the fields anyway, we leave it as is?Thanks,
- Juri
Re: [PATCH 02/14] sched: add extended scheduling interface. (new ABI)
Quoted 108 lines · Steven Rostedt
Quoted 105 lines · On Tue, 03 Dec 2013 17:13:44 +0100 Juri Lelli <[email protected]> wrote:
Quoted 50 lines · Ingo Molnar
Quoted 47 lines · Peter Zijlstra
Quoted 17 lines · Juri Lelli
kernel/sched/core.c +5
-2+SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) { -struct sched_param2 lp; +struct sched_attr lp;struct task_struct *p;int retval; -if (!param2 || pid <0 ) +if (!attr || pid <0 )return -EINVAL ; +memset (&lp,0 ,sizeof (struct sched_attr )); +rcu_read_lock (); p =find_process_by_pid (pid); retval = -ESRCH ;kernel/sched/core.c SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param · line 3501
Quoted 8 lines
kernel/sched/core.c +1
-1@@ -3427,7 +3495,7 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid, struct sched_param2 __user *, param lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (param2, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ;return retval; out_unlock:So this side needs a bit more care; suppose the kernel has a larger attr than userspace knows about.
What would make more sense; add another syscall argument with the userspace
sizeof(struct sched_attr), or expect userspace to initializeattr->sizeto the right value before callingsched_getattr()?To me the extra argument makes more sense; that is:
struct
sched_attrattr;ret = sched_getattr(0, &attr, sizeof(attr));seems like a saner thing than:
struct sched_attr attr = { .size = sizeof(attr), }; ret = sched_getattr(0, &attr);Quoted 9 lines
Quoted 5 lines
Mostly because the former has a clear separation between input and output arguments, whereas for the second form the attr argument is both input and output.
Ingo?
I suppose so - in the
sys_perf_event_open()case we ran out of arguments, soattr::sizewas the only sane way to do it.Ok, I modified it like this:
------------------------------------------------------------
Subject: [PATCH] fixup: add checks forsys_sched_getattrAdd an extra argument to the syscall with the userspace
sizeof(struct sched_attr)to be able to handle situations when the kernel has a larger attr than userspace knows about.--- include/linux/syscalls.h | 3 ++- kernel/sched/core.c | 55 ++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 5 deletions(-)include/linux/syscalls.h +2
-1@@ -288,7 +288,8 @@ asmlinkage long sys_sched_getscheduler(pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param);asmlinkage long sys_sched_getattr (pid_t pid, -struct sched_attr __user *attr); +struct sched_attr __user *attr, +unsigned int size);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,kernel/sched/core.c +1
-1@@ -3507,7 +3507,7 @@ do_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param) }/* -* Mimics kerner/events/core.c perf_copy_attr(). +* Mimics kernel/events/core.c perf_copy_attr(). */ static int sched_copy_attr (struct sched_attr __user *uattr,struct sched_attr *attr)@@ -3726,18 +3726,65 @@ out_unlock:return retval; }
Quoted 12 lines
kernel/sched/core.c out_unlock: · line 3736
Quoted 8 lines
kernel/sched/core.c +8
-0+static int sched_read_attr (struct sched_attr __user *uattr, +struct sched_attr *attr, +unsigned int size, +unsigned int usize) +{ +int ret; + +if (!access_ok (VERIFY_WRITE , uattr,SCHED_ATTR_SIZE_VER0 ))We want to verify from uattr to usize, right? As that is what we are writing to.
Right. s/SCHED_ATTR_SIZE_VER0/usize/
Quoted 10 lines
kernel/sched/core.c out_unlock: · line 3742
Quoted 6 lines
kernel/sched/core.c +6
-0+return -EFAULT ; + +/* +* zero the full structure, so that a short copy will be nice. +*/ +memset (uattr,0 ,sizeof (*uattr));Wait! We can't write to user space like this, not to mention that usize may even be less than
sizeof(struct sched_attr).
Ouch! I should never copy and paste.
Quoted 11 lines
Quoted 11 lines
kernel/sched/core.c +10
-0+ +/* +* If we're handed a smaller struct than we know of, +* ensure all the unknown bits are 0 - i.e. old +* user-space does not get uncomplete information. +*/ +if (usize <sizeof (*attr)) { +unsigned char *addr; +unsigned char *end; +
Quoted 8 lines
kernel/sched/core.c out_unlock: · line 3760
Quoted 8 lines
kernel/sched/core.c +8
-0+ addr = (void *)attr + usize; + end = (void *)attr + size; + +for (; addr < end; addr++) +if (*addr) + goto err_size; + } +
But, if we got here, we know that all is zero after usize and that usize >= SCHED_ATTR_SIZE_VER0 (see below). So it should be safe to copy_to_user() without the memset (since attr has been already zeroed), do I get it right?
Thanks,
- Juri
Quoted 50 lines
Quoted 49 lines
kernel/sched/core.c +18
-4+ ret =copy_to_user (uattr, attr, usize); +if (ret) +return -EFAULT ; + +out: +return ret; + +err_size: + ret = -E2BIG ; + goto out; +} +/** * sys_sched_getattr - same as above, but with extended "sched_param" * @pid: the pid in question. * @attr: structure containing the extended parameters. +* @size: sizeof(attr) for fwd/bwd comp. */ -SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) +SYSCALL_DEFINE3(sched_getattr, pid_t, pid, struct sched_attr __user *, attr, + unsigned int, size) {struct sched_attr lp;struct task_struct *p;int retval; -if (!attr || pid <0 ) +if (!attr || pid <0 || size >PAGE_SIZE || + size <SCHED_ATTR_SIZE_VER0 )return -EINVAL ;memset (&lp,0 ,sizeof (struct sched_attr ));@@ -3758,7 +3805,7 @@ SYSCALL_DEFINE2(sched_getattr, pid_t, pid, struct sched_attr __user *, attr) lp.sched_priority = p->rt_priority;rcu_read_unlock (); - retval =copy_to_user (attr, &lp,sizeof (lp)) ? -EFAULT :0 ; + retval =sched_read_attr (attr, &lp,sizeof (lp), size);return retval; out_unlock: -----------------------------------------------------------Do we need to make
sched_setattrsymmetrical, or, since the user has to fill the fields anyway, we leave it as is?Thanks,
- Juri
tip-bot for Dario Faggioli
merge bot[tip:sched/core] sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
In reply to Juri Lellisched: Add new scheduler syscalls to support an extended scheduling parameters ABI
Commit-ID: d50dde5a10f305253cbc3855307f608f8a3c5f73 Gitweb: http://git.kernel.org/tip/d50dde5a10f305253cbc3855307f608f8a3c5f73 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:36 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:04 +0100
sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
Add the syscalls needed for supporting scheduling algorithms with extended scheduling parameters (e.g., SCHED_DEADLINE).
In general, it makes possible to specify a periodic/sporadic task, that executes for a given amount of runtime at each instance, and is scheduled according to the urgency of their own timing constraints, i.e.:
- a (maximum/typical) instance execution time,
- a minimum interval between consecutive instances,
- a time constraint by which each instance must be completed.
Thus, both the data structure that holds the scheduling parameters of the tasks and the system calls dealing with it must be extended. Unfortunately, modifying the existing struct sched_param would break the ABI and result in potentially serious compatibility issues with legacy binaries.
For these reasons, this patch:
- defines the new struct sched_attr, containing all the fields that are necessary for specifying a task in the computational model described above;
- defines and implements the new scheduling related syscalls that manipulate it, i.e., sched_setattr() and sched_getattr().
Syscalls are introduced for x86 (32 and 64 bits) and ARM only, as a proof of concept and for developing and testing purposes. Making them available on other architectures is straightforward.
Since no "user" for these new parameters is introduced in this patch, the implementation of the new system calls is just identical to their already existing counterpart. Future patches that implement scheduling policies able to exploit the new data structure must also take care of modifying the sched_*attr() calls accordingly with their own purposes.
--- arch/arm/include/asm/unistd.h | 2 +- arch/arm/include/uapi/asm/unistd.h | 2 + arch/arm/kernel/calls.S | 2 + arch/x86/syscalls/syscall_32.tbl | 2 + arch/x86/syscalls/syscall_64.tbl | 2 + include/linux/sched.h | 62 +++++++++ include/linux/syscalls.h | 6 + kernel/sched/core.c | 263 ++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 9 +- 9 files changed, 326 insertions(+), 24 deletions(-)
arch/arm/include/asm/unistd.h +1 -1
@@ -15,7 +15,7 @@#include <uapi/asm/unistd.h> -#define __NR_syscalls (380) +#define __NR_syscalls (384)#define __ARM_NR_cmpxchg (__ARM_NR_BASE+0x00fff0)#define __ARCH_WANT_STAT64
arch/arm/include/uapi/asm/unistd.h +2 -0
@@ -406,6 +406,8 @@#define __NR_process_vm_writev (__NR_SYSCALL_BASE+377)#define __NR_kcmp (__NR_SYSCALL_BASE+378)#define __NR_finit_module (__NR_SYSCALL_BASE+379) +#define __NR_sched_setattr (__NR_SYSCALL_BASE+380) +#define __NR_sched_getattr (__NR_SYSCALL_BASE+381)/* * This may need to be greater than __NR_last_syscall+1 in order to
arch/arm/kernel/calls.S +2 -0
@@ -389,6 +389,8 @@ CALL(sys_process_vm_writev) CALL(sys_kcmp) CALL(sys_finit_module) +/* 380 */ CALL(sys_sched_setattr) + CALL(sys_sched_getattr) #ifndef syscalls_counted .equ syscalls_padding, ((NR_syscalls + 3) & ~3) - NR_syscalls #define syscalls_counted
arch/x86/syscalls/syscall_32.tbl +2 -0
@@ -357,3 +357,5 @@ 348 i386 process_vm_writev sys_process_vm_writev compat_sys_process_vm_writev 349 i386 kcmp sys_kcmp 350 i386 finit_module sys_finit_module +351 i386 sched_setattr sys_sched_setattr +352 i386 sched_getattr sys_sched_getattr
arch/x86/syscalls/syscall_64.tbl +2 -0
@@ -320,6 +320,8 @@ 311 64 process_vm_writev sys_process_vm_writev 312 common kcmp sys_kcmp 313 common finit_module sys_finit_module +314 common sched_setattr sys_sched_setattr +315 common sched_getattr sys_sched_getattr # # x32-specific system call numbers start at 512 to avoid cache impact
include/linux/sched.h +62 -0
74 lines not on this page · raw message
include/linux/syscalls.h +6 -0
@@ -38,6 +38,7 @@ struct rlimit;struct rlimit64 ;struct rusage ;struct sched_param ; +struct sched_attr ;struct sel_arg_struct ;struct semaphore ;struct sembuf ;
@@ -279,9 +280,14 @@ asmlinkage long sys_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param);asmlinkage long sys_sched_setparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_setattr (pid_t pid, +struct sched_attr __user *attr);asmlinkage long sys_sched_getscheduler (pid_t pid);asmlinkage long sys_sched_getparam (pid_t pid,struct sched_param __user *param); +asmlinkage long sys_sched_getattr (pid_t pid, +struct sched_attr __user *attr, +unsigned int size);asmlinkage long sys_sched_setaffinity (pid_t pid,unsigned int len,unsigned long __user *user_mask_ptr); asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
kernel/sched/core.c +243 -20
372 lines not on this page · raw message
kernel/sched/sched.h +6 -3
@@ -81,11 +81,14 @@ extern void update_cpu_load_active(struct rq *this_rq);*/ #define RUNTIME_INF ((u64)~0ULL) +static inline int fair_policy (int policy) +{ +return policy ==SCHED_NORMAL || policy ==SCHED_BATCH ; +} +static inline int rt_policy (int policy) { -if (policy ==SCHED_FIFO || policy ==SCHED_RR ) -return 1 ; -return 0 ; +return policy ==SCHED_FIFO || policy ==SCHED_RR ; }static inline int task_has_rt_policy (struct task_struct *p)
Peter Zijlstra
maintainerRFC PATCH 1/1 sched: Move SCHED_RESET_ON_FORK into attr::sched_flags
Does something like the below make sense to you guys?
---
Subject: sched: Move SCHED_RESET_ON_FORK into attr::sched_flags
From: Peter Zijlstra <[email protected]>
Date: Wed Jan 15 17:05:04 CET 2014
I noticed the new sched_{set,get}attr() calls didn't properly deal with the SCHED_RESET_ON_FORK hack.
Instead of propagating the flags in high bits nonsense use the brand spanking new attr::sched_flags field.
--- include/uapi/linux/sched.h | 5 +++++ kernel/sched/core.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 14 deletions(-)
include/uapi/linux/sched.h +5 -0
@@ -40,8 +40,13 @@/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5#define SCHED_DEADLINE 6 +/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000 +/* +* For the sched_{set,get}attr() calls +*/ +#define SCHED_FLAG_RESET_ON_FORK 0x01#endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +28 -14
78 lines not on this page · raw message
tip-bot for Peter Zijlstra
merge bot[tip:sched/core] sched: Move SCHED_RESET_ON_FORK into attr:: sched_flags
sched: Move SCHED_RESET_ON_FORK into attr::sched_flags
Commit-ID: 7479f3c9cf67edf5e8a76b21ea3726757f35cf53 Gitweb: http://git.kernel.org/tip/7479f3c9cf67edf5e8a76b21ea3726757f35cf53 Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 15 Jan 2014 17:05:04 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Thu, 16 Jan 2014 09:27:17 +0100
sched: Move SCHED_RESET_ON_FORK into attr::sched_flags
I noticed the new sched_{set,get}attr() calls didn't properly deal with the SCHED_RESET_ON_FORK hack.
Instead of propagating the flags in high bits nonsense use the brand spanking new attr::sched_flags field.
--- include/uapi/linux/sched.h | 5 +++++ kernel/sched/core.c | 42 ++++++++++++++++++++++++++++-------------- 2 files changed, 33 insertions(+), 14 deletions(-)
include/uapi/linux/sched.h +5 -0
@@ -40,8 +40,13 @@/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5#define SCHED_DEADLINE 6 +/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000 +/* +* For the sched_{set,get}attr() calls +*/ +#define SCHED_FLAG_RESET_ON_FORK 0x01#endif /* _UAPI_LINUX_SCHED_H */
kernel/sched/core.c +28 -14
78 lines not on this page · raw message
Re: [tip:sched/core] sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
In reply to tip-bot for Dario FaggioliQuoted 11 lines · tip-bot for Dario Faggioli
Commit-ID: d50dde5a10f305253cbc3855307f608f8a3c5f73 Gitweb: http://git.kernel.org/tip/d50dde5a10f305253cbc3855307f608f8a3c5f73 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:36 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:04 +0100sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
Add the syscalls needed for supporting scheduling algorithms with extended scheduling parameters (e.g.,
SCHED_DEADLINE).
I'm seeing a regression in next-20140116 on my system, and "git bisect" points at this patch. I noticed the fix patch "sched: Fix up scheduler syscall LTP fails", but applying that doesn't solve the problem.
The symptoms are that when the system boots, I see the login prompt over the serial port OK, but after I've entered my user/password, the login process simply hangs. CTRL-C drops me back to the login prompt.
Do you have any hints how I should debug this?
Re: [tip:sched/core] sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
Quoted 8 lines · Stephen Warren
Quoted 7 lines · tip-bot for Dario Faggioli
Commit-ID: d50dde5a10f305253cbc3855307f608f8a3c5f73 Gitweb: http://git.kernel.org/tip/d50dde5a10f305253cbc3855307f608f8a3c5f73 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:36 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:04 +0100
Quoted 8 lines
Quoted 4 lines
sched: Add new scheduler syscalls to support an extended scheduling parameters ABI
Add the syscalls needed for supporting scheduling algorithms with extended scheduling parameters (e.g.,
SCHED_DEADLINE).I'm seeing a regression in next-20140116 on my system, and "git bisect" points at this patch. I noticed the fix patch "sched: Fix up scheduler syscall LTP fails", but applying that doesn't solve the problem.
Never mind; additional searching revealed "sched: Fix __sched_setscheduler() nice test", which does fix the issue for me.
PATCH 3/14 sched: SCHED_DEADLINE structures & implementation.
In reply to Juri LelliFrom: Dario Faggioli <[email protected]>
Introduces the data structures, constants and symbols needed for SCHED_DEADLINE implementation.
Core data structure of SCHED_DEADLINE are defined, along with their initializers. Hooks for checking if a task belong to the new policy are also added where they are needed.
Adds a scheduling class, in sched/dl.c and a new policy called SCHED_DEADLINE. It is an implementation of the Earliest Deadline First (EDF) scheduling algorithm, augmented with a mechanism (called Constant Bandwidth Server, CBS) that makes it possible to isolate the behaviour of tasks between each other.
The typical -deadline task will be made up of a computation phase (instance) which is activated on a periodic or sporadic fashion. The expected (maximum) duration of such computation is called the task's runtime; the time interval by which each instance need to be completed is called the task's relative deadline. The task's absolute deadline is dynamically calculated as the time instant a task (better, an instance) activates plus the relative deadline.
The EDF algorithms selects the task with the smallest absolute deadline as the one to be executed first, while the CBS ensures each task to run for at most its runtime every (relative) deadline length time interval, avoiding any interference between different tasks (bandwidth isolation).
Thanks to this feature, also tasks that do not strictly comply with the computational model sketched above can effectively use the new policy.
To summarize, this patch:
- introduces the data structures, constants and symbols needed;
- implements the core logic of the scheduling algorithm in the new scheduling class file;
- provides all the glue code between the new scheduling class and the core scheduler and refines the interactions between sched/dl and the other existing scheduling classes.
--- arch/arm/include/asm/unistd.h | 2 +- include/linux/sched.h | 46 ++- include/linux/sched/deadline.h | 24 ++ include/linux/sched/rt.h | 2 +- include/uapi/linux/sched.h | 1 + kernel/fork.c | 4 +- kernel/hrtimer.c | 3 +- kernel/sched/Makefile | 2 +- kernel/sched/core.c | 111 ++++++- kernel/sched/deadline.c | 682 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 28 ++ kernel/sched/stop_task.c | 2 +- 12 files changed, 884 insertions(+), 23 deletions(-) create mode 100644 include/linux/sched/deadline.h create mode 100644 kernel/sched/deadline.c
arch/arm/include/asm/unistd.h +1 -1
@@ -15,7 +15,7 @@#include <uapi/asm/unistd.h> -#define __NR_syscalls (383) +#define __NR_syscalls (384)#define __ARM_NR_cmpxchg (__ARM_NR_BASE+0x00fff0)#define __ARCH_WANT_STAT64
include/linux/sched.h +45 -1
70 lines not on this page · raw message
include/linux/sched/deadline.h new file +24 -0
@@ -0,0 +1,24 @@ +#ifndef _SCHED_DEADLINE_H +#define _SCHED_DEADLINE_H + +/* +* SCHED_DEADLINE tasks has negative priorities, reflecting +* the fact that any of them has higher prio than RT and +* NORMAL/BATCH tasks. +*/ + +#define MAX_DL_PRIO 0 + +static inline int dl_prio (int prio) +{ +if (unlikely (prio <MAX_DL_PRIO )) +return 1 ; +return 0 ; +} + +static inline int dl_task (struct task_struct *p) +{ +return dl_prio (p->prio); +} + +#endif /* _SCHED_DEADLINE_H */
include/linux/sched/rt.h +1 -1
@@ -22,7 +22,7 @@static inline int rt_prio (int prio) { -if (unlikely (prio <MAX_RT_PRIO )) +if ((unsigned )prio <MAX_RT_PRIO )return 1 ;return 0 ; }
include/uapi/linux/sched.h +1 -0
@@ -39,6 +39,7 @@#define SCHED_BATCH 3/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5 +#define SCHED_DEADLINE 6/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000
kernel/fork.c +3 -1
@@ -1313,7 +1313,9 @@ static struct task_struct *copy_process(unsigned long clone_flags,#endif /* Perform scheduler related setup. Assign this task to a CPU. */ -sched_fork (p); + retval =sched_fork (p); +if (retval) + goto bad_fork_cleanup_policy; retval =perf_event_init_task (p);if (retval)
kernel/hrtimer.c +2 -1
@@ -46,6 +46,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/timer.h> #include <linux/freezer.h>
@@ -1610,7 +1611,7 @@ long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp,unsigned long slack; slack = current->timer_slack_ns; -if (rt_task (current)) +if (dl_task (current) ||rt_task (current)) slack =0 ;hrtimer_init_on_stack (&t.timer, clockid, mode);
kernel/sched/Makefile +1 -1
@@ -11,7 +11,7 @@ ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer endif -obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o stop_task.o +obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o deadline.o stop_task.o obj-$(CONFIG_SMP) += cpupri.o obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o obj-$(CONFIG_SCHEDSTATS) += stats.o
kernel/sched/core.c +95 -16
240 lines not on this page · raw message
kernel/sched/deadline.c new file +682 -0
682 lines not on this page · raw message
kernel/sched/sched.h +28 -0
@@ -2,6 +2,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/mutex.h> #include <linux/spinlock.h> #include <linux/stop_machine.h>
@@ -87,11 +88,23 @@ static inline int rt_policy(int policy)return 0 ; } +static inline int dl_policy (int policy) +{ +if (unlikely (policy ==SCHED_DEADLINE )) +return 1 ; +return 0 ; +} +static inline int task_has_rt_policy (struct task_struct *p) {return rt_policy (p->policy); } +static inline int task_has_dl_policy (struct task_struct *p) +{ +return dl_policy (p->policy); +} +/* * This is the priority-queue data structure of the RT scheduling class: */
@@ -363,6 +376,15 @@ struct rt_rq {#endif }; +/* Deadline class' related fields in a runqueue */ +struct dl_rq { +/* runqueue is an rbtree, ordered by deadline */ +struct rb_root rb_root; +struct rb_node *rb_leftmost; + +unsigned long dl_nr_running; +}; +#ifdef CONFIG_SMP /*
@@ -427,6 +449,7 @@ struct rq {struct cfs_rq cfs;struct rt_rq rt; +struct dl_rq dl;#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */
@@ -957,6 +980,7 @@ static const u32 prio_to_wmult[40] = {#else #define ENQUEUE_WAKING 0#endif +#define ENQUEUE_REPLENISH 8#define DEQUEUE_SLEEP 1
@@ -1012,6 +1036,7 @@ struct sched_class {for (class = sched_class_highest; class; class = class->next) extern const struct sched_class stop_sched_class; +extern const struct sched_class dl_sched_class;extern const struct sched_class rt_sched_class;extern const struct sched_class fair_sched_class;extern const struct sched_class idle_sched_class;
@@ -1047,6 +1072,8 @@ extern void resched_cpu(int cpu);extern struct rt_bandwidth def_rt_bandwidth;extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); +extern void init_dl_task_timer (struct sched_dl_entity *dl_se); +extern void update_idle_cpu_load (struct rq *this_rq);extern void init_task_runnable_average (struct task_struct *p);
@@ -1305,6 +1332,7 @@ extern void print_rt_stats(struct seq_file *m, int cpu);extern void init_cfs_rq (struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq); +extern void init_dl_rq (struct dl_rq *rt_rq,struct rq *rq);extern void account_cfs_bandwidth_used (int enabled,int was_enabled);
kernel/sched/stop_task.c +1 -1
@@ -103,7 +103,7 @@ get_rr_interval_stop(struct rq *rq, struct task_struct *task)* Simple, special scheduling class for the per-CPU stop tasks: */ const struct sched_class stop_sched_class = { - .next = &rt_sched_class, + .next = &dl_sched_class, .enqueue_task = enqueue_task_stop, .dequeue_task = dequeue_task_stop,
Signature · 1 line
1.7.9.5
Steven Rostedt
maintainerRe: [PATCH 03/14] sched: SCHED_DEADLINE structures & implementation.
Quoted 34 lines · On Thu, 7 Nov 2013 14:43:37 +0100 Juri Lelli <[email protected]> wrote:
From: Dario Faggioli <[email protected]>
include/linux/sched/deadline.h new file +24
-0@@ -0,0 +1,24 @@ +#ifndef _SCHED_DEADLINE_H +#define _SCHED_DEADLINE_H + +/* +* SCHED_DEADLINE tasks has negative priorities, reflecting +* the fact that any of them has higher prio than RT and +* NORMAL/BATCH tasks. +*/ + +#define MAX_DL_PRIO 0 + +static inline int dl_prio (int prio) +{ +if (unlikely (prio <MAX_DL_PRIO )) +return 1 ; +return 0 ; +} + +static inline int dl_task (struct task_struct *p) +{ +return dl_prio (p->prio); +} + +#endif /* _SCHED_DEADLINE_H */ include/linux/sched/rt.h +0
-0
include/linux/sched/rt.h line 25
Quoted 5 lines
include/linux/sched/rt.h +1
-1@@ -22,7 +22,7 @@static inline int rt_prio (int prio) { -if (unlikely (prio <MAX_RT_PRIO )) +if ((unsigned )prio <MAX_RT_PRIO )
Why remove the "unlikely" here?
Quoted 43 lines
include/linux/sched/rt.h +0
-0return 1 ;return 0 ; }include/uapi/linux/sched.h +1
-0@@ -39,6 +39,7 @@#define SCHED_BATCH 3/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5 +#define SCHED_DEADLINE 6/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000kernel/fork.c +3
-1@@ -1313,7 +1313,9 @@ static struct task_struct *copy_process(unsigned long clone_flags,#endif /* Perform scheduler related setup. Assign this task to a CPU. */ -sched_fork (p); + retval =sched_fork (p); +if (retval) + goto bad_fork_cleanup_policy; retval =perf_event_init_task (p);if (retval)kernel/hrtimer.c +1
-0@@ -46,6 +46,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/timer.h> #include <linux/freezer.h>
kernel/hrtimer.c long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp, · line 1614
Quoted 5 lines
kernel/hrtimer.c +1
-1@@ -1610,7 +1611,7 @@ long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp,unsigned long slack; slack = current->timer_slack_ns; -if (rt_task (current)) +if (dl_task (current) ||rt_task (current))
Since dl_task() checks if prio is less than 0, and rt_task checks for prio < MAX_RT_PRIO, I wonder if we can introduce a
dl_or_rt_task(current)
that does a signed compare against MAX_RT_PRIO to eliminate the double compare (in case gcc doesn't figure it out).
Not something that we need to change now, but something in the future maybe.
Quoted 63 lines
kernel/hrtimer.c +0
-0slack =0 ;hrtimer_init_on_stack (&t.timer, clockid, mode);kernel/sched/Makefile +1
-1@@ -11,7 +11,7 @@ ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer endif -obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o stop_task.o +obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o deadline.o stop_task.o obj-$(CONFIG_SMP) += cpupri.o obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o obj-$(CONFIG_SCHEDSTATS) += stats.okernel/sched/core.c +11
-3@@ -903,7 +903,9 @@ static inline int normal_prio(struct task_struct *p) {int prio; -if (task_has_rt_policy (p)) +if (task_has_dl_policy (p)) + prio =MAX_DL_PRIO -1 ; +else if (task_has_rt_policy (p)) prio =MAX_RT_PRIO -1 - p->rt_priority;else prio =__normal_prio (p);@@ -1611,6 +1613,12 @@ static void __sched_fork(struct task_struct *p)memset (&p->se.statistics,0 ,sizeof (p->se.statistics));#endif +RB_CLEAR_NODE (&p->dl.rb_node); +hrtimer_init (&p->dl.dl_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); + p->dl.dl_runtime = p->dl.runtime =0 ; + p->dl.dl_deadline = p->dl.deadline =0 ; + p->dl.flags =0 ; +INIT_LIST_HEAD (&p->rt.run_list);#ifdef CONFIG_PREEMPT_NOTIFIERS @@ -1654,7 +1662,7 @@ void set_numabalancing_state(bool enabled)/* * fork()/clone()-time setup: */ -void sched_fork (struct task_struct *p) +int sched_fork (struct task_struct *p) {unsigned long flags;int cpu =get_cpu ();@@ -1676,7 +1684,7 @@ void sched_fork(struct task_struct *p)* Revert to default priority/policy on fork if requested. */ if (unlikely (p->sched_reset_on_fork)) { -if (task_has_rt_policy (p)) { +if (task_has_dl_policy (p) ||task_has_rt_policy (p)) { p->policy =SCHED_NORMAL ; p->static_prio =NICE_TO_PRIO (0 ); p->rt_priority =0 ;
kernel/sched/core.c void sched_fork(struct task_struct *p) · line 1706
Quoted 7 lines
kernel/sched/core.c +3
-1@@ -1693,8 +1701,14 @@ void sched_fork(struct task_struct *p) p->sched_reset_on_fork =0 ; } -if (!rt_prio (p->prio)) +if (dl_prio (p->prio)) { +put_cpu (); +return -EAGAIN ;
Deadline tasks are not allowed to fork? Well, without setting reset_on_fork. Probably should add a comment here.
Quoted 17 lines
kernel/sched/core.c +5
-0+ }else if (rt_prio (p->prio)) { + p->sched_class = &rt_sched_class; + }else { p->sched_class = &fair_sched_class; + }if (p->sched_class->task_fork) p->sched_class->task_fork (p);@@ -1726,6 +1740,7 @@ void sched_fork(struct task_struct *p)#endif put_cpu (); +return 0 ; }/*
kernel/sched/core.c void rt_mutex_setprio(struct task_struct *p, int prio) · line 3047
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -3029,7 +3044,7 @@ void rt_mutex_setprio(struct task_struct *p, int prio)struct rq *rq;const struct sched_class *prev_class; -BUG_ON (prio <0 || prio >MAX_PRIO ); +BUG_ON (prio >MAX_PRIO );
Should we have this be:
BUG_ON(prio < -MAX_PRIO || prio > MAX_PRIO);
?
Quoted 119 lines
kernel/sched/core.c +58
-7rq =__task_rq_lock (p);@@ -3061,7 +3076,9 @@ void rt_mutex_setprio(struct task_struct *p, int prio)if (running) p->sched_class->put_prev_task (rq, p); -if (rt_prio (prio)) +if (dl_prio (prio)) + p->sched_class = &dl_sched_class; +else if (rt_prio (prio)) p->sched_class = &rt_sched_class;else p->sched_class = &fair_sched_class;@@ -3095,9 +3112,9 @@ void set_user_nice(struct task_struct *p, long nice)* The RT priorities are set via sched_setscheduler(), but we still * allow the 'normal' nice value to be set - but as expected * it wont have any effect on scheduling until the task is -* SCHED_FIFO/SCHED_RR: +* SCHED_DEADLINE, SCHED_FIFO or SCHED_RR: */ -if (task_has_rt_policy (p)) { +if (task_has_dl_policy (p) ||task_has_rt_policy (p)) { p->static_prio =NICE_TO_PRIO (nice); goto out_unlock; }@@ -3261,7 +3278,9 @@ __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) p->normal_prio =normal_prio (p);/* we are holding p->pi_lock already */ p->prio =rt_mutex_getprio (p); -if (rt_prio (p->prio)) +if (dl_prio (p->prio)) + p->sched_class = &dl_sched_class; +else if (rt_prio (p->prio)) p->sched_class = &rt_sched_class;else p->sched_class = &fair_sched_class;@@ -3269,6 +3288,50 @@ __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) }/* +* This function initializes the sched_dl_entity of a newly becoming +* SCHED_DEADLINE task. +* +* Only the static values are considered here, the actual runtime and the +* absolute deadline will be properly calculated when the task is enqueued +* for the first time with its new policy. +*/ +static void +__setparam_dl (struct task_struct *p,const struct sched_param2 *param2) +{ +struct sched_dl_entity *dl_se = &p->dl; + +init_dl_task_timer (dl_se); + dl_se->dl_runtime = param2->sched_runtime; + dl_se->dl_deadline = param2->sched_deadline; + dl_se->flags = param2->sched_flags; + dl_se->dl_throttled =0 ; + dl_se->dl_new =1 ; +} + +static void +__getparam_dl (struct task_struct *p,struct sched_param2 *param2) +{ +struct sched_dl_entity *dl_se = &p->dl; + + param2->sched_priority = p->rt_priority; + param2->sched_runtime = dl_se->dl_runtime; + param2->sched_deadline = dl_se->dl_deadline; + param2->sched_flags = dl_se->flags; +} + +/* +* This function validates the new parameters of a -deadline task. +* We ask for the deadline not being zero, and greater or equal +* than the runtime. +*/ +static bool +__checkparam_dl (const struct sched_param2 *prm) +{ +return prm && (&prm->sched_deadline) !=0 && + (s64)(&prm->sched_deadline - &prm->sched_runtime) >=0 ; +} + +/* * check the target process has a UID that matches the current process's */ static bool check_same_owner (struct task_struct *p)@@ -3305,7 +3368,8 @@ recheck: reset_on_fork = !!(policy &SCHED_RESET_ON_FORK ); policy &= ~SCHED_RESET_ON_FORK ; -if (policy !=SCHED_FIFO && policy !=SCHED_RR && +if (policy !=SCHED_DEADLINE && + policy !=SCHED_FIFO && policy !=SCHED_RR && policy !=SCHED_NORMAL && policy !=SCHED_BATCH && policy !=SCHED_IDLE )return -EINVAL ;@@ -3320,7 +3384,8 @@ recheck: (p->mm && param->sched_priority >MAX_USER_RT_PRIO -1 ) || (!p->mm && param->sched_priority > MAX_RT_PRIO-1)) return -EINVAL; -if (rt_policy (policy) != (param->sched_priority !=0 )) +if ((dl_policy (policy) && !__checkparam_dl (param)) || + (rt_policy (policy) != (param->sched_priority !=0 )))return -EINVAL ;/* @@ -3386,7 +3451,8 @@ recheck:* If not changing anything there's no need to proceed further: */ if (unlikely (policy == p->policy && (!rt_policy (policy) || - param->sched_priority == p->rt_priority))) { + param->sched_priority == p->rt_priority) && + !dl_policy (policy))) {task_rq_unlock (rq, p, &flags);return 0 ; }
kernel/sched/core.c recheck: · line 3496
Quoted 9 lines
kernel/sched/core.c +5
-1@@ -3423,7 +3489,11 @@ recheck: oldprio = p->prio; prev_class = p->sched_class; -__setscheduler (rq, p, policy, param->sched_priority); +if (dl_policy (policy)) { +__setparam_dl (p, param); +__setscheduler (rq, p, policy, param->sched_priority); + }else +__setscheduler (rq, p, policy, param->sched_priority);
Why the double "__setscheduler()" call? Why not just:
if (dl_policy(policy)) __setparam_dl(p, param); __setscheduler(rq, p, policy, param->sched_priority);
?
Quoted 53 lines
kernel/sched/core.c +11
-2if (running) p->sched_class->set_curr_task (rq);@@ -3527,8 +3597,11 @@ do_sched_setscheduler2(pid_t pid, int policy,rcu_read_lock (); retval = -ESRCH ; p =find_process_by_pid (pid); -if (p !=NULL ) +if (p !=NULL ) { +if (dl_policy (policy)) + lparam2.sched_priority =0 ; retval =sched_setscheduler2 (p, policy, &lparam2); + }rcu_read_unlock ();return retval;@@ -3685,7 +3758,10 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid,if (retval) goto out_unlock; - lp.sched_priority = p->rt_priority; +if (task_has_dl_policy (p)) +__getparam_dl (p, &lp); +else + lp.sched_priority = p->rt_priority;rcu_read_unlock (); retval = copy_to_user(param2, &lp,@@ -4120,6 +4196,7 @@ SYSCALL_DEFINE1(sched_get_priority_max, int, policy)case SCHED_RR : ret =MAX_USER_RT_PRIO -1 ;break ; +case SCHED_DEADLINE :case SCHED_NORMAL :case SCHED_BATCH :case SCHED_IDLE :@@ -4146,6 +4223,7 @@ SYSCALL_DEFINE1(sched_get_priority_min, int, policy)case SCHED_RR : ret =1 ;break ; +case SCHED_DEADLINE :case SCHED_NORMAL :case SCHED_BATCH :case SCHED_IDLE :@@ -6563,6 +6641,7 @@ void __init sched_init(void) rq->calc_load_update = jiffies +LOAD_FREQ ;init_cfs_rq (&rq->cfs);init_rt_rq (&rq->rt, rq); +init_dl_rq (&rq->dl, rq);#ifdef CONFIG_FAIR_GROUP_SCHED root_task_group.shares =ROOT_TASK_GROUP_LOAD ;INIT_LIST_HEAD (&rq->leaf_cfs_rq_list);
kernel/sched/core.c void normalize_rt_tasks(void) · line 6828
Quoted 5 lines
kernel/sched/core.c +1
-1@@ -6746,7 +6825,7 @@ void normalize_rt_tasks(void) p->se.statistics.block_start =0 ;#endif -if (!rt_task (p)) { +if (!dl_task (p) && !rt_task (p)) {
Again, a future enhancement is to combine these two.
Quoted 19 lines
kernel/sched/core.c +0
-0/* * Renice negative nice level userspace * tasks back to 0: kernel/sched/deadline.c new file +10
-0@@ -0,0 +1,682 @@ +/* +* Deadline Scheduling Class (SCHED_DEADLINE) +* +* Earliest Deadline First (EDF) + Constant Bandwidth Server (CBS). +* +* Tasks that periodically executes their instances for less than their +* runtime won't miss any of their deadlines. +* Tasks that are not periodic or sporadic or that tries to execute more +* than their reserved bandwidth will be slowed down (and may potentially +* miss some of their deadlines), and won't affect any other task.
kernel/sched/deadline.c line 20
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+* +* Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Michael Trimarchi <[email protected]>, +* Fabio Checconi <[email protected]> +*/ +#include "sched.h" + +static inline int dl_time_before (u64 a,u64 b) +{ +return (s64)(a - b) <0 ;
I think I've seen this pattern enough that we probably should have a helper function that does this. Off topic for this thread, but just something for us to think about.
Quoted 132 lines
kernel/sched/deadline.c new file +132
-0+} + +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) +{ + return container_of(dl_se, struct task_struct, dl); +} + +static inline struct rq *rq_of_dl_rq (struct dl_rq *dl_rq) +{ + return container_of(dl_rq, struct rq, dl); +} + +static inline struct dl_rq *dl_rq_of_se (struct sched_dl_entity *dl_se) +{ +struct task_struct *p =dl_task_of (dl_se); +struct rq *rq =task_rq (p); + +return &rq->dl; +} + +static inline int on_dl_rq (struct sched_dl_entity *dl_se) +{ +return !RB_EMPTY_NODE (&dl_se->rb_node); +} + +static inline int is_leftmost (struct task_struct *p,struct dl_rq *dl_rq) +{ +struct sched_dl_entity *dl_se = &p->dl; + +return dl_rq->rb_leftmost == &dl_se->rb_node; +} + +void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) +{ + dl_rq->rb_root =RB_ROOT ; +} + +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags); +static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); +static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p, +int flags); + +/* +* We are being explicitly informed that a new instance is starting, +* and this means that: +* - the absolute deadline of the entity has to be placed at +* current time + relative deadline; +* - the runtime of the entity has to be set to the maximum value. +* +* The capability of specifying such event is useful whenever a -deadline +* entity wants to (try to!) synchronize its behaviour with the scheduler's +* one, and to (try to!) reconcile itself with its own scheduling +* parameters. +*/ +static inline void setup_new_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +WARN_ON (!dl_se->dl_new || dl_se->dl_throttled); + +/* +* We use the regular wall clock time to set deadlines in the +* future; in fact, we must consider execution overheads (time +* spent on hardirq context, etc.). +*/ + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime; + dl_se->dl_new =0 ; +} + +/* +* Pure Earliest Deadline First (EDF) scheduling does not deal with the +* possibility of a entity lasting more than what it declared, and thus +* exhausting its runtime. +* +* Here we are interested in making runtime overrun possible, but we do +* not want a entity which is misbehaving to affect the scheduling of all +* other entities. +* Therefore, a budgeting strategy called Constant Bandwidth Server (CBS) +* is used, in order to confine each entity within its own bandwidth. +* +* This function deals exactly with that, and ensures that when the runtime +* of a entity is replenished, its deadline is also postponed. That ensures +* the overrunning entity can't interfere with other entity in the system and +* can't make them miss their deadlines. Reasons why this kind of overruns +* could happen are, typically, a entity voluntarily trying to overcome its +* runtime, or it just underestimated it during sched_setscheduler_ex(). +*/ +static void replenish_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* We keep moving the deadline away until we get some +* available runtime for the entity. This ensures correct +* handling of situations where the runtime overrun is +* arbitrary large. +*/ +while (dl_se->runtime <=0 ) { + dl_se->deadline += dl_se->dl_deadline; + dl_se->runtime += dl_se->dl_runtime; + } + +/* +* At this point, the deadline really should be "in +* the future" with respect to rq->clock. If it's +* not, we are, for some reason, lagging too much! +* Anyway, after having warn userspace abut that, +* we still try to keep the things running by +* resetting the deadline and the budget of the +* entity. +*/ +if (dl_time_before (dl_se->deadline,rq_clock (rq))) { +static bool lag_once = false; + +if (!lag_once) { + lag_once = true; +printk_sched ("sched: DL replenish lagged to much\n" ); + } + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime; + } +} + +/* +* Here we check if --at time t-- an entity (which is probably being +* [re]activated or, in general, enqueued) can use its remaining runtime +* and its current deadline _without_ exceeding the bandwidth it is +* assigned (function returns true if it can't). We are in fact applying +* one of the CBS rules: when a task wakes up, if the residual runtime
kernel/sched/deadline.c line 162
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+* over residual deadline fits within the allocated bandwidth, then we +* can keep the current (absolute) deadline and residual budget without +* disrupting the schedulability of the system. Otherwise, we should +* refill the runtime and set the deadline a period in the future, +* because keeping the current (absolute) deadline of the task would +* result in breaking guarantees promised to other tasks. +* +* This function returns true if: +* +* runtime / (deadline - t) > dl_runtime / dl_deadline ,
I'm a bit confused here about this algorithm. If I state I want to have a run time of 20 ms out of 100 ms, so my dl_runtime / dl_deadline is 1/5 or %20.
Now if I don't don't get scheduled at the start of my new period because other tasks are running, and I start say, at 50ms in. Thus, my deadline - t == 50, but my runtime remaining is still 20ms. Thus I have 20/50 which is 2/5 or %40. Doing the calculation I have 40 > 20 being true, and thus we label this as an overflow.
Is this what we really want? Or did I misunderstand something here?
Is this only for tasks that went to sleep voluntarily and have just woken up?
Quoted 48 lines
kernel/sched/deadline.c new file +48
-0+* +* IOW we can't recycle current parameters. +*/ +static bool dl_entity_overflow (struct sched_dl_entity *dl_se,u64 t) +{ +u64 left, right; + +/* +* left and right are the two sides of the equation above, +* after a bit of shuffling to use multiplications instead +* of divisions. +* +* Note that none of the time values involved in the two +* multiplications are absolute: dl_deadline and dl_runtime +* are the relative deadline and the maximum runtime of each +* instance, runtime is the runtime left for the last instance +* and (deadline - t), since t is rq->clock, is the time left +* to the (absolute) deadline. Even if overflowing the u64 type +* is very unlikely to occur in both cases, here we scale down +* as we want to avoid that risk at all. Scaling down by 10 +* means that we reduce granularity to 1us. We are fine with it, +* since this is only a true/false check and, anyway, thinking +* of anything below microseconds resolution is actually fiction +* (but still we want to give the user that illusion >;). +*/ + left = (dl_se->dl_deadline >>10 ) * (dl_se->runtime >>10 ); + right = ((dl_se->deadline - t) >>10 ) * (dl_se->dl_runtime >>10 ); + +return dl_time_before (right, left); +} + +/* +* When a -deadline entity is queued back on the runqueue, its runtime and +* deadline might need updating. +* +* The policy here is that we update the deadline of the entity only if: +* - the current deadline is in the past, +* - using the remaining runtime with the current deadline would make +* the entity exceed its bandwidth. +*/ +static void update_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* The arrival of a new instance needs special treatment, i.e., +* the actual scheduling parameters have to be "renewed".
kernel/sched/deadline.c line 220
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+*/ +if (dl_se->dl_new) { +setup_new_dl_entity (dl_se); +return ; + } + +if (dl_time_before (dl_se->deadline,rq_clock (rq)) || +dl_entity_overflow (dl_se,rq_clock (rq))) { + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime;
In the case that I stated, we just moved the deadline forward.
Or is this only because it was just enqueued, and we fear that it will starve out other tasks? That is, this only happens if the task was sleeping and just woke up.
-- Steve
Quoted 568 lines
kernel/sched/deadline.c new file +462
-0+ } +} + +/* +* If the entity depleted all its runtime, and if we want it to sleep +* while waiting for some new execution time to become available, we +* set the bandwidth enforcement timer to the replenishment instant +* and try to activate it. +* +* Notice that it is important for the caller to know if the timer +* actually started or not (i.e., the replenishment instant is in +* the future or in the past). +*/ +static int start_dl_timer (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); +ktime_t now, act; +ktime_t soft, hard; +unsigned long range; +s64 delta; + +/* +* We want the timer to fire at the deadline, but considering +* that it is actually coming from rq->clock and not from +* hrtimer's time base reading. +*/ + act =ns_to_ktime (dl_se->deadline); + now =hrtimer_cb_get_time (&dl_se->dl_timer); + delta =ktime_to_ns (now) -rq_clock (rq); + act =ktime_add_ns (act, delta); + +/* +* If the expiry time already passed, e.g., because the value +* chosen as the deadline is too small, don't even try to +* start the timer in the past! +*/ +if (ktime_us_delta (act, now) <0 ) +return 0 ; + +hrtimer_set_expires (&dl_se->dl_timer, act); + + soft =hrtimer_get_softexpires (&dl_se->dl_timer); + hard =hrtimer_get_expires (&dl_se->dl_timer); + range =ktime_to_ns (ktime_sub (hard, soft)); +__hrtimer_start_range_ns (&dl_se->dl_timer, soft, + range,HRTIMER_MODE_ABS ,0 ); + +return hrtimer_active (&dl_se->dl_timer); +} + +/* +* This is the bandwidth enforcement timer callback. If here, we know +* a task is not on its dl_rq, since the fact that the timer was running +* means the task is throttled and needs a runtime replenishment. +* +* However, what we actually do depends on the fact the task is active, +* (it is on its rq) or has been removed from there by a call to +* dequeue_task_dl(). In the former case we must issue the runtime +* replenishment and add the task back to the dl_rq; in the latter, we just +* do nothing but clearing dl_throttled, so that runtime and deadline +* updating (and the queueing back to dl_rq) will be done by the +* next call to enqueue_task_dl(). +*/ +static enum hrtimer_restart dl_task_timer (struct hrtimer *timer) +{ +struct sched_dl_entity *dl_se =container_of (timer, + struct sched_dl_entity, + dl_timer); +struct task_struct *p =dl_task_of (dl_se); +struct rq *rq =task_rq (p); +raw_spin_lock (&rq->lock); + +/* +* We need to take care of a possible races here. In fact, the +* task might have changed its scheduling policy to something +* different from SCHED_DEADLINE or changed its reservation +* parameters (through sched_setscheduler()). +*/ +if (!dl_task (p) || dl_se->dl_new) + goto unlock; + + dl_se->dl_throttled =0 ; +if (p->on_rq) { +enqueue_task_dl (rq, p,ENQUEUE_REPLENISH ); +if (task_has_dl_policy (rq->curr)) +check_preempt_curr_dl (rq, p,0 ); +else +resched_task (rq->curr); + } +unlock: +raw_spin_unlock (&rq->lock); + +return HRTIMER_NORESTART ; +} + +void init_dl_task_timer (struct sched_dl_entity *dl_se) +{ +struct hrtimer *timer = &dl_se->dl_timer; + +if (hrtimer_active (timer)) { +hrtimer_try_to_cancel (timer); +return ; + } + +hrtimer_init (timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); + timer->function = dl_task_timer; +} + +static +int dl_runtime_exceeded (struct rq *rq,struct sched_dl_entity *dl_se) +{ +int dmiss =dl_time_before (dl_se->deadline,rq_clock (rq)); +int rorun = dl_se->runtime <=0 ; + +if (!rorun && !dmiss) +return 0 ; + +/* +* If we are beyond our current deadline and we are still +* executing, then we have already used some of the runtime of +* the next instance. Thus, if we do not account that, we are +* stealing bandwidth from the system at each deadline miss! +*/ +if (dmiss) { + dl_se->runtime = rorun ? dl_se->runtime :0 ; + dl_se->runtime -=rq_clock (rq) - dl_se->deadline; + } + +return 1 ; +} + +/* +* Update the current task's runtime statistics (provided it is still +* a -deadline task and has not been removed from the dl_rq). +*/ +static void update_curr_dl (struct rq *rq) +{ +struct task_struct *curr = rq->curr; +struct sched_dl_entity *dl_se = &curr->dl; +u64 delta_exec; + +if (!dl_task (curr) || !on_dl_rq (dl_se)) +return ; + +/* +* Consumed budget is computed considering the time as +* observed by schedulable tasks (excluding time spent +* in hardirq context, etc.). Deadlines are instead +* computed using hard walltime. This seems to be the more +* natural solution, but the full ramifications of this +* approach need further study. +*/ + delta_exec =rq_clock_task (rq) - curr->se.exec_start; +if (unlikely ((s64 )delta_exec <0 )) + delta_exec =0 ; + +schedstat_set (curr->se.statistics.exec_max, +max (curr->se.statistics.exec_max, delta_exec)); + + curr->se.sum_exec_runtime += delta_exec; +account_group_exec_runtime (curr, delta_exec); + + curr->se.exec_start =rq_clock_task (rq); +cpuacct_charge (curr, delta_exec); + + dl_se->runtime -= delta_exec; +if (dl_runtime_exceeded (rq, dl_se)) { +__dequeue_task_dl (rq, curr,0 ); +if (likely (start_dl_timer (dl_se))) + dl_se->dl_throttled =1 ; +else +enqueue_task_dl (rq, curr,ENQUEUE_REPLENISH ); + +if (!is_leftmost (curr, &rq->dl)) +resched_task (curr); + } +} + +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rb_node **link = &dl_rq->rb_root.rb_node; +struct rb_node *parent =NULL ; +struct sched_dl_entity *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&dl_se->rb_node)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct sched_dl_entity, rb_node); +if (dl_time_before (dl_se->deadline, entry->deadline)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->rb_leftmost = &dl_se->rb_node; + +rb_link_node (&dl_se->rb_node, parent, link); +rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); + + dl_rq->dl_nr_running++; +} + +static void __dequeue_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); + +if (RB_EMPTY_NODE (&dl_se->rb_node)) +return ; + +if (dl_rq->rb_leftmost == &dl_se->rb_node) { +struct rb_node *next_node; + + next_node =rb_next (&dl_se->rb_node); + dl_rq->rb_leftmost = next_node; + } + +rb_erase (&dl_se->rb_node, &dl_rq->rb_root); +RB_CLEAR_NODE (&dl_se->rb_node); + + dl_rq->dl_nr_running--; +} + +static void +enqueue_dl_entity (struct sched_dl_entity *dl_se,int flags) +{ +BUG_ON (on_dl_rq (dl_se)); + +/* +* If this is a wakeup or a new instance, the scheduling +* parameters of the task might need updating. Otherwise, +* we want a replenishment of its runtime. +*/ +if (!dl_se->dl_new && flags &ENQUEUE_REPLENISH ) +replenish_dl_entity (dl_se); +else +update_dl_entity (dl_se); + +__enqueue_dl_entity (dl_se); +} + +static void dequeue_dl_entity (struct sched_dl_entity *dl_se) +{ +__dequeue_dl_entity (dl_se); +} + +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +/* +* If p is throttled, we do nothing. In fact, if it exhausted +* its budget it needs a replenishment and, since it now is on +* its rq, the bandwidth timer callback (which clearly has not +* run yet) will take care of this. +*/ +if (p->dl.dl_throttled) +return ; + +enqueue_dl_entity (&p->dl, flags); +inc_nr_running (rq); +} + +static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +dequeue_dl_entity (&p->dl); +} + +static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +update_curr_dl (rq); +__dequeue_task_dl (rq, p, flags); + +dec_nr_running (rq); +} + +/* +* Yield task semantic for -deadline tasks is: +* +* get off from the CPU until our next instance, with +* a new runtime. This is of little use now, since we +* don't have a bandwidth reclaiming mechanism. Anyway, +* bandwidth reclaiming is planned for the future, and +* yield_task_dl will indicate that some spare budget +* is available for other task instances to use it. +*/ +static void yield_task_dl (struct rq *rq) +{ +struct task_struct *p = rq->curr; + +/* +* We make the task go to sleep until its current deadline by +* forcing its runtime to zero. This way, update_curr_dl() stops +* it and the bandwidth timer will wake it up and will give it +* new scheduling parameters (thanks to dl_new=1). +*/ +if (p->dl.runtime >0 ) { + rq->curr->dl.dl_new =1 ; + p->dl.runtime =0 ; + } +update_curr_dl (rq); +} + +/* +* Only called when both the current and waking task are -deadline +* tasks. +*/ +static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p, +int flags) +{ +if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +resched_task (rq->curr); +} + +#ifdef CONFIG_SCHED_HRTICK +static void start_hrtick_dl (struct rq *rq,struct task_struct *p) +{ +s64 delta = p->dl.dl_runtime - p->dl.runtime; + +if (delta >10000 ) +hrtick_start (rq, p->dl.runtime); +} +#endif + +static struct sched_dl_entity *pick_next_dl_entity (struct rq *rq, +struct dl_rq *dl_rq) +{ +struct rb_node *left = dl_rq->rb_leftmost; + +if (!left) +return NULL ; + + return rb_entry(left, struct sched_dl_entity, rb_node); +} + +struct task_struct *pick_next_task_dl (struct rq *rq) +{ +struct sched_dl_entity *dl_se; +struct task_struct *p; +struct dl_rq *dl_rq; + + dl_rq = &rq->dl; + +if (unlikely (!dl_rq->dl_nr_running)) +return NULL ; + + dl_se =pick_next_dl_entity (rq, dl_rq); +BUG_ON (!dl_se); + + p =dl_task_of (dl_se); + p->se.exec_start =rq_clock_task (rq); +#ifdef CONFIG_SCHED_HRTICK +if (hrtick_enabled (rq)) +start_hrtick_dl (rq, p); +#endif +return p; +} + +static void put_prev_task_dl (struct rq *rq,struct task_struct *p) +{ +update_curr_dl (rq); +} + +static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued) +{ +update_curr_dl (rq); + +#ifdef CONFIG_SCHED_HRTICK +if (hrtick_enabled (rq) && queued && p->dl.runtime >0 ) +start_hrtick_dl (rq, p); +#endif +} + +static void task_fork_dl (struct task_struct *p) +{ +/* +* SCHED_DEADLINE tasks cannot fork and this is achieved through +* sched_fork() +*/ +} + +static void task_dead_dl (struct task_struct *p) +{ +struct hrtimer *timer = &p->dl.dl_timer; + +if (hrtimer_active (timer)) +hrtimer_try_to_cancel (timer); +} + +static void set_curr_task_dl (struct rq *rq) +{ +struct task_struct *p = rq->curr; + + p->se.exec_start =rq_clock_task (rq); +} + +static void switched_from_dl (struct rq *rq,struct task_struct *p) +{ +if (hrtimer_active (&p->dl.dl_timer)) +hrtimer_try_to_cancel (&p->dl.dl_timer); +} + +static void switched_to_dl (struct rq *rq,struct task_struct *p) +{ +/* +* If p is throttled, don't consider the possibility +* of preempting rq->curr, the check will be done right +* after its runtime will get replenished. +*/ +if (unlikely (p->dl.dl_throttled)) +return ; + +if (!p->on_rq || rq->curr != p) { +if (task_has_dl_policy (rq->curr)) +check_preempt_curr_dl (rq, p,0 ); +else +resched_task (rq->curr); + } +} + +static void prio_changed_dl (struct rq *rq,struct task_struct *p, +int oldprio) +{ +switched_to_dl (rq, p); +} + +#ifdef CONFIG_SMP +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +return task_cpu (p); +} +#endif + +const struct sched_class dl_sched_class = { + .next = &rt_sched_class, + .enqueue_task = enqueue_task_dl, + .dequeue_task = dequeue_task_dl, + .yield_task = yield_task_dl, + + .check_preempt_curr = check_preempt_curr_dl, + + .pick_next_task = pick_next_task_dl, + .put_prev_task = put_prev_task_dl, + +#ifdef CONFIG_SMP + .select_task_rq = select_task_rq_dl, +#endif + + .set_curr_task = set_curr_task_dl, + .task_tick = task_tick_dl, + .task_fork = task_fork_dl, + .task_dead = task_dead_dl, + + .prio_changed = prio_changed_dl, + .switched_from = switched_from_dl, + .switched_to = switched_to_dl, +};kernel/sched/sched.h +28
-0@@ -2,6 +2,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/mutex.h> #include <linux/spinlock.h> #include <linux/stop_machine.h> @@ -87,11 +88,23 @@ static inline int rt_policy(int policy)return 0 ; } +static inline int dl_policy (int policy) +{ +if (unlikely (policy ==SCHED_DEADLINE )) +return 1 ; +return 0 ; +} +static inline int task_has_rt_policy (struct task_struct *p) {return rt_policy (p->policy); } +static inline int task_has_dl_policy (struct task_struct *p) +{ +return dl_policy (p->policy); +} +/* * This is the priority-queue data structure of the RT scheduling class: */ @@ -363,6 +376,15 @@ struct rt_rq {#endif }; +/* Deadline class' related fields in a runqueue */ +struct dl_rq { +/* runqueue is an rbtree, ordered by deadline */ +struct rb_root rb_root; +struct rb_node *rb_leftmost; + +unsigned long dl_nr_running; +}; +#ifdef CONFIG_SMP /* @@ -427,6 +449,7 @@ struct rq {struct cfs_rq cfs;struct rt_rq rt; +struct dl_rq dl;#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */ @@ -957,6 +980,7 @@ static const u32 prio_to_wmult[40] = {#else #define ENQUEUE_WAKING 0#endif +#define ENQUEUE_REPLENISH 8#define DEQUEUE_SLEEP 1@@ -1012,6 +1036,7 @@ struct sched_class {for (class = sched_class_highest; class; class = class->next) extern const struct sched_class stop_sched_class; +extern const struct sched_class dl_sched_class;extern const struct sched_class rt_sched_class;extern const struct sched_class fair_sched_class;extern const struct sched_class idle_sched_class;@@ -1047,6 +1072,8 @@ extern void resched_cpu(int cpu);extern struct rt_bandwidth def_rt_bandwidth;extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); +extern void init_dl_task_timer (struct sched_dl_entity *dl_se); +extern void update_idle_cpu_load (struct rq *this_rq);extern void init_task_runnable_average (struct task_struct *p);@@ -1305,6 +1332,7 @@ extern void print_rt_stats(struct seq_file *m, int cpu);extern void init_cfs_rq (struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq); +extern void init_dl_rq (struct dl_rq *rt_rq,struct rq *rq);extern void account_cfs_bandwidth_used (int enabled,int was_enabled);kernel/sched/stop_task.c +1
-1@@ -103,7 +103,7 @@ get_rr_interval_stop(struct rq *rq, struct task_struct *task)* Simple, special scheduling class for the per-CPU stop tasks: */ const struct sched_class stop_sched_class = { - .next = &rt_sched_class, + .next = &dl_sched_class, .enqueue_task = enqueue_task_stop, .dequeue_task = dequeue_task_stop,
Re: [PATCH 03/14] sched: SCHED_DEADLINE structures & implementation.
Quoted 38 lines · Steven Rostedt
Quoted 35 lines · On Thu, 7 Nov 2013 14:43:37 +0100 Juri Lelli <[email protected]> wrote:
From: Dario Faggioli <[email protected]>
include/linux/sched/deadline.h new file +24
-0@@ -0,0 +1,24 @@ +#ifndef _SCHED_DEADLINE_H +#define _SCHED_DEADLINE_H + +/* +* SCHED_DEADLINE tasks has negative priorities, reflecting +* the fact that any of them has higher prio than RT and +* NORMAL/BATCH tasks. +*/ + +#define MAX_DL_PRIO 0 + +static inline int dl_prio (int prio) +{ +if (unlikely (prio <MAX_DL_PRIO )) +return 1 ; +return 0 ; +} + +static inline int dl_task (struct task_struct *p) +{ +return dl_prio (p->prio); +} + +#endif /* _SCHED_DEADLINE_H */ include/linux/sched/rt.h +0
-0@@ -22,7 +22,7 @@
Quoted 7 lines
include/linux/sched/rt.h line 25
Quoted 4 lines
include/linux/sched/rt.h +1
-1static inline int rt_prio (int prio) { -if (unlikely (prio <MAX_RT_PRIO )) +if ((unsigned )prio <MAX_RT_PRIO )Why remove the "unlikely" here?
No reason that I can recall, most probably something went wrong with successive rebases. Grrrr! Fixed.
Quoted 52 lines
Quoted 43 lines
include/linux/sched/rt.h +0
-0return 1 ;return 0 ; }include/uapi/linux/sched.h +1
-0@@ -39,6 +39,7 @@#define SCHED_BATCH 3/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5 +#define SCHED_DEADLINE 6/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000kernel/fork.c +3
-1@@ -1313,7 +1313,9 @@ static struct task_struct *copy_process(unsigned long clone_flags,#endif /* Perform scheduler related setup. Assign this task to a CPU. */ -sched_fork (p); + retval =sched_fork (p); +if (retval) + goto bad_fork_cleanup_policy; retval =perf_event_init_task (p);if (retval)kernel/hrtimer.c +1
-0@@ -46,6 +46,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/timer.h> #include <linux/freezer.h> kernel/hrtimer.c long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp, · line 1614
Quoted 5 lines
kernel/hrtimer.c +1
-1@@ -1610,7 +1611,7 @@ long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp,unsigned long slack; slack = current->timer_slack_ns; -if (rt_task (current)) +if (dl_task (current) ||rt_task (current))Since
dl_task()checks if prio is less than 0, andrt_taskchecks for prio <MAX_RT_PRIO, I wonder if we can introduce a
Quoted 8 lines
dl_or_rt_task(current)that does a signed compare against
MAX_RT_PRIOto eliminate the double compare (in case gcc doesn't figure it out).Not something that we need to change now, but something in the future maybe.
Ok.
Quoted 66 lines
Quoted 66 lines
kernel/hrtimer.c +0
-0slack =0 ;hrtimer_init_on_stack (&t.timer, clockid, mode);kernel/sched/Makefile +1
-1@@ -11,7 +11,7 @@ ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer endif -obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o stop_task.o +obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o deadline.o stop_task.o obj-$(CONFIG_SMP) += cpupri.o obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o obj-$(CONFIG_SCHEDSTATS) += stats.okernel/sched/core.c +11
-3@@ -903,7 +903,9 @@ static inline int normal_prio(struct task_struct *p) {int prio; -if (task_has_rt_policy (p)) +if (task_has_dl_policy (p)) + prio =MAX_DL_PRIO -1 ; +else if (task_has_rt_policy (p)) prio =MAX_RT_PRIO -1 - p->rt_priority;else prio =__normal_prio (p);@@ -1611,6 +1613,12 @@ static void __sched_fork(struct task_struct *p)memset (&p->se.statistics,0 ,sizeof (p->se.statistics));#endif +RB_CLEAR_NODE (&p->dl.rb_node); +hrtimer_init (&p->dl.dl_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); + p->dl.dl_runtime = p->dl.runtime =0 ; + p->dl.dl_deadline = p->dl.deadline =0 ; + p->dl.flags =0 ; +INIT_LIST_HEAD (&p->rt.run_list);#ifdef CONFIG_PREEMPT_NOTIFIERS @@ -1654,7 +1662,7 @@ void set_numabalancing_state(bool enabled)/* * fork()/clone()-time setup: */ -void sched_fork (struct task_struct *p) +int sched_fork (struct task_struct *p) {unsigned long flags;int cpu =get_cpu ();@@ -1676,7 +1684,7 @@ void sched_fork(struct task_struct *p)* Revert to default priority/policy on fork if requested. */ if (unlikely (p->sched_reset_on_fork)) { -if (task_has_rt_policy (p)) { +if (task_has_dl_policy (p) ||task_has_rt_policy (p)) { p->policy =SCHED_NORMAL ; p->static_prio =NICE_TO_PRIO (0 ); p->rt_priority =0 ;@@ -1693,8 +1701,14 @@ void sched_fork(struct task_struct *p) p->sched_reset_on_fork =0 ; }
Quoted 8 lines
kernel/sched/core.c void sched_fork(struct task_struct *p) · line 1706
Quoted 4 lines
kernel/sched/core.c +3
-1-if (!rt_prio (p->prio)) +if (dl_prio (p->prio)) { +put_cpu (); +return -EAGAIN ;Deadline tasks are not allowed to fork? Well, without setting
reset_on_fork. Probably should add a comment here.
Yes, because with didn't yet decide how to split a father's bandwidth among children. This way we force the user to think about it. I'll add a comment here.
Peter was actually wondering if we should return a different error:
Quoted 39 lines · Juri Lelli
Quoted 8 lines
kernel/sched/core.c +0
-0> @@ -1693,8 +1701,14 @@ void sched_fork(struct task_struct *p) > p->sched_reset_on_fork = 0; > }kernel/sched/core.c void sched_fork(struct task_struct *p) · line 1706
Quoted 5 lines
kernel/sched/core.c +3
-1-if (!rt_prio (p->prio)) +if (dl_prio (p->prio)) { +put_cpu (); +return -EAGAIN ;Is this really the error we want to return on
fork()?EAGAIN to me indicates a spurious error and we should try again later; however as it obvious from the code above; we'll always fail, there's no point in trying again later.
I would think something like EINVAL; even though there are no arguments to
fork(); would me a better option.Then again; I really don't care too much; anybody any preferences?
Quoted 20 lines
kernel/sched/core.c +5
-0+ }else if (rt_prio (p->prio)) { + p->sched_class = &rt_sched_class; + }else { p->sched_class = &fair_sched_class; + }if (p->sched_class->task_fork) p->sched_class->task_fork (p);@@ -1726,6 +1740,7 @@ void sched_fork(struct task_struct *p)#endif put_cpu (); +return 0 ; }/* @@ -3029,7 +3044,7 @@ void rt_mutex_setprio(struct task_struct *p, int prio)struct rq *rq;const struct sched_class *prev_class;
Quoted 8 lines
kernel/sched/core.c void rt_mutex_setprio(struct task_struct *p, int prio) · line 3047
Quoted 2 lines
kernel/sched/core.c +1
-1-BUG_ON (prio <0 || prio >MAX_PRIO ); +BUG_ON (prio >MAX_PRIO );Should we have this be:
BUG_ON(prio < -MAX_PRIO || prio > MAX_PRIO);?
BUG_ON(prio < (MAX_DL_PRIO-1) || prio > MAX_PRIO); ?
Quoted 131 lines
Quoted 119 lines
kernel/sched/core.c +58
-7rq =__task_rq_lock (p);@@ -3061,7 +3076,9 @@ void rt_mutex_setprio(struct task_struct *p, int prio)if (running) p->sched_class->put_prev_task (rq, p); -if (rt_prio (prio)) +if (dl_prio (prio)) + p->sched_class = &dl_sched_class; +else if (rt_prio (prio)) p->sched_class = &rt_sched_class;else p->sched_class = &fair_sched_class;@@ -3095,9 +3112,9 @@ void set_user_nice(struct task_struct *p, long nice)* The RT priorities are set via sched_setscheduler(), but we still * allow the 'normal' nice value to be set - but as expected * it wont have any effect on scheduling until the task is -* SCHED_FIFO/SCHED_RR: +* SCHED_DEADLINE, SCHED_FIFO or SCHED_RR: */ -if (task_has_rt_policy (p)) { +if (task_has_dl_policy (p) ||task_has_rt_policy (p)) { p->static_prio =NICE_TO_PRIO (nice); goto out_unlock; }@@ -3261,7 +3278,9 @@ __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) p->normal_prio =normal_prio (p);/* we are holding p->pi_lock already */ p->prio =rt_mutex_getprio (p); -if (rt_prio (p->prio)) +if (dl_prio (p->prio)) + p->sched_class = &dl_sched_class; +else if (rt_prio (p->prio)) p->sched_class = &rt_sched_class;else p->sched_class = &fair_sched_class;@@ -3269,6 +3288,50 @@ __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) }/* +* This function initializes the sched_dl_entity of a newly becoming +* SCHED_DEADLINE task. +* +* Only the static values are considered here, the actual runtime and the +* absolute deadline will be properly calculated when the task is enqueued +* for the first time with its new policy. +*/ +static void +__setparam_dl (struct task_struct *p,const struct sched_param2 *param2) +{ +struct sched_dl_entity *dl_se = &p->dl; + +init_dl_task_timer (dl_se); + dl_se->dl_runtime = param2->sched_runtime; + dl_se->dl_deadline = param2->sched_deadline; + dl_se->flags = param2->sched_flags; + dl_se->dl_throttled =0 ; + dl_se->dl_new =1 ; +} + +static void +__getparam_dl (struct task_struct *p,struct sched_param2 *param2) +{ +struct sched_dl_entity *dl_se = &p->dl; + + param2->sched_priority = p->rt_priority; + param2->sched_runtime = dl_se->dl_runtime; + param2->sched_deadline = dl_se->dl_deadline; + param2->sched_flags = dl_se->flags; +} + +/* +* This function validates the new parameters of a -deadline task. +* We ask for the deadline not being zero, and greater or equal +* than the runtime. +*/ +static bool +__checkparam_dl (const struct sched_param2 *prm) +{ +return prm && (&prm->sched_deadline) !=0 && + (s64)(&prm->sched_deadline - &prm->sched_runtime) >=0 ; +} + +/* * check the target process has a UID that matches the current process's */ static bool check_same_owner (struct task_struct *p)@@ -3305,7 +3368,8 @@ recheck: reset_on_fork = !!(policy &SCHED_RESET_ON_FORK ); policy &= ~SCHED_RESET_ON_FORK ; -if (policy !=SCHED_FIFO && policy !=SCHED_RR && +if (policy !=SCHED_DEADLINE && + policy !=SCHED_FIFO && policy !=SCHED_RR && policy !=SCHED_NORMAL && policy !=SCHED_BATCH && policy !=SCHED_IDLE )return -EINVAL ;@@ -3320,7 +3384,8 @@ recheck: (p->mm && param->sched_priority >MAX_USER_RT_PRIO -1 ) || (!p->mm && param->sched_priority > MAX_RT_PRIO-1)) return -EINVAL; -if (rt_policy (policy) != (param->sched_priority !=0 )) +if ((dl_policy (policy) && !__checkparam_dl (param)) || + (rt_policy (policy) != (param->sched_priority !=0 )))return -EINVAL ;/* @@ -3386,7 +3451,8 @@ recheck:* If not changing anything there's no need to proceed further: */ if (unlikely (policy == p->policy && (!rt_policy (policy) || - param->sched_priority == p->rt_priority))) { + param->sched_priority == p->rt_priority) && + !dl_policy (policy))) {task_rq_unlock (rq, p, &flags);return 0 ; }kernel/sched/core.c recheck: · line 3496
Quoted 9 lines
kernel/sched/core.c +5
-1@@ -3423,7 +3489,11 @@ recheck: oldprio = p->prio; prev_class = p->sched_class; -__setscheduler (rq, p, policy, param->sched_priority); +if (dl_policy (policy)) { +__setparam_dl (p, param); +__setscheduler (rq, p, policy, param->sched_priority); + }else +__setscheduler (rq, p, policy, param->sched_priority);Why the double "
__setscheduler()" call? Why not just:
Quoted 6 lines
if (dl_policy(policy)) __setparam_dl(p, param); __setscheduler(rq, p, policy, param->sched_priority);?
Right, changed.
Quoted 56 lines
Quoted 56 lines
kernel/sched/core.c +11
-2if (running) p->sched_class->set_curr_task (rq);@@ -3527,8 +3597,11 @@ do_sched_setscheduler2(pid_t pid, int policy,rcu_read_lock (); retval = -ESRCH ; p =find_process_by_pid (pid); -if (p !=NULL ) +if (p !=NULL ) { +if (dl_policy (policy)) + lparam2.sched_priority =0 ; retval =sched_setscheduler2 (p, policy, &lparam2); + }rcu_read_unlock ();return retval;@@ -3685,7 +3758,10 @@ SYSCALL_DEFINE2(sched_getparam2, pid_t, pid,if (retval) goto out_unlock; - lp.sched_priority = p->rt_priority; +if (task_has_dl_policy (p)) +__getparam_dl (p, &lp); +else + lp.sched_priority = p->rt_priority;rcu_read_unlock (); retval = copy_to_user(param2, &lp,@@ -4120,6 +4196,7 @@ SYSCALL_DEFINE1(sched_get_priority_max, int, policy)case SCHED_RR : ret =MAX_USER_RT_PRIO -1 ;break ; +case SCHED_DEADLINE :case SCHED_NORMAL :case SCHED_BATCH :case SCHED_IDLE :@@ -4146,6 +4223,7 @@ SYSCALL_DEFINE1(sched_get_priority_min, int, policy)case SCHED_RR : ret =1 ;break ; +case SCHED_DEADLINE :case SCHED_NORMAL :case SCHED_BATCH :case SCHED_IDLE :@@ -6563,6 +6641,7 @@ void __init sched_init(void) rq->calc_load_update = jiffies +LOAD_FREQ ;init_cfs_rq (&rq->cfs);init_rt_rq (&rq->rt, rq); +init_dl_rq (&rq->dl, rq);#ifdef CONFIG_FAIR_GROUP_SCHED root_task_group.shares =ROOT_TASK_GROUP_LOAD ;INIT_LIST_HEAD (&rq->leaf_cfs_rq_list);@@ -6746,7 +6825,7 @@ void normalize_rt_tasks(void) p->se.statistics.block_start =0 ;#endif
Quoted 5 lines
kernel/sched/core.c void normalize_rt_tasks(void) · line 6828
Quoted 2 lines
kernel/sched/core.c +1
-1-if (!rt_task (p)) { +if (!dl_task (p) && !rt_task (p)) {Again, a future enhancement is to combine these two.
Ok.
Quoted 188 lines
Quoted 19 lines
kernel/sched/core.c +0
-0/* * Renice negative nice level userspace * tasks back to 0: kernel/sched/deadline.c new file +10
-0@@ -0,0 +1,682 @@ +/* +* Deadline Scheduling Class (SCHED_DEADLINE) +* +* Earliest Deadline First (EDF) + Constant Bandwidth Server (CBS). +* +* Tasks that periodically executes their instances for less than their +* runtime won't miss any of their deadlines. +* Tasks that are not periodic or sporadic or that tries to execute more +* than their reserved bandwidth will be slowed down (and may potentially +* miss some of their deadlines), and won't affect any other task. kernel/sched/deadline.c line 20
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+* +* Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Michael Trimarchi <[email protected]>, +* Fabio Checconi <[email protected]> +*/ +#include "sched.h" + +static inline int dl_time_before (u64 a,u64 b) +{ +return (s64)(a - b) <0 ;I think I've seen this pattern enough that we probably should have a helper function that does this. Off topic for this thread, but just something for us to think about.
Quoted 132 lines
kernel/sched/deadline.c new file +132
-0+} + +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) +{ + return container_of(dl_se, struct task_struct, dl); +} + +static inline struct rq *rq_of_dl_rq (struct dl_rq *dl_rq) +{ + return container_of(dl_rq, struct rq, dl); +} + +static inline struct dl_rq *dl_rq_of_se (struct sched_dl_entity *dl_se) +{ +struct task_struct *p =dl_task_of (dl_se); +struct rq *rq =task_rq (p); + +return &rq->dl; +} + +static inline int on_dl_rq (struct sched_dl_entity *dl_se) +{ +return !RB_EMPTY_NODE (&dl_se->rb_node); +} + +static inline int is_leftmost (struct task_struct *p,struct dl_rq *dl_rq) +{ +struct sched_dl_entity *dl_se = &p->dl; + +return dl_rq->rb_leftmost == &dl_se->rb_node; +} + +void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) +{ + dl_rq->rb_root =RB_ROOT ; +} + +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags); +static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); +static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p, +int flags); + +/* +* We are being explicitly informed that a new instance is starting, +* and this means that: +* - the absolute deadline of the entity has to be placed at +* current time + relative deadline; +* - the runtime of the entity has to be set to the maximum value. +* +* The capability of specifying such event is useful whenever a -deadline +* entity wants to (try to!) synchronize its behaviour with the scheduler's +* one, and to (try to!) reconcile itself with its own scheduling +* parameters. +*/ +static inline void setup_new_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +WARN_ON (!dl_se->dl_new || dl_se->dl_throttled); + +/* +* We use the regular wall clock time to set deadlines in the +* future; in fact, we must consider execution overheads (time +* spent on hardirq context, etc.). +*/ + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime; + dl_se->dl_new =0 ; +} + +/* +* Pure Earliest Deadline First (EDF) scheduling does not deal with the +* possibility of a entity lasting more than what it declared, and thus +* exhausting its runtime. +* +* Here we are interested in making runtime overrun possible, but we do +* not want a entity which is misbehaving to affect the scheduling of all +* other entities. +* Therefore, a budgeting strategy called Constant Bandwidth Server (CBS) +* is used, in order to confine each entity within its own bandwidth. +* +* This function deals exactly with that, and ensures that when the runtime +* of a entity is replenished, its deadline is also postponed. That ensures +* the overrunning entity can't interfere with other entity in the system and +* can't make them miss their deadlines. Reasons why this kind of overruns +* could happen are, typically, a entity voluntarily trying to overcome its +* runtime, or it just underestimated it during sched_setscheduler_ex(). +*/ +static void replenish_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* We keep moving the deadline away until we get some +* available runtime for the entity. This ensures correct +* handling of situations where the runtime overrun is +* arbitrary large. +*/ +while (dl_se->runtime <=0 ) { + dl_se->deadline += dl_se->dl_deadline; + dl_se->runtime += dl_se->dl_runtime; + } + +/* +* At this point, the deadline really should be "in +* the future" with respect to rq->clock. If it's +* not, we are, for some reason, lagging too much! +* Anyway, after having warn userspace abut that, +* we still try to keep the things running by +* resetting the deadline and the budget of the +* entity. +*/ +if (dl_time_before (dl_se->deadline,rq_clock (rq))) { +static bool lag_once = false; + +if (!lag_once) { + lag_once = true; +printk_sched ("sched: DL replenish lagged to much\n" ); + } + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime; + } +} + +/* +* Here we check if --at time t-- an entity (which is probably being +* [re]activated or, in general, enqueued) can use its remaining runtime +* and its current deadline _without_ exceeding the bandwidth it is +* assigned (function returns true if it can't). We are in fact applying +* one of the CBS rules: when a task wakes up, if the residual runtime kernel/sched/deadline.c line 162
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+* over residual deadline fits within the allocated bandwidth, then we +* can keep the current (absolute) deadline and residual budget without +* disrupting the schedulability of the system. Otherwise, we should +* refill the runtime and set the deadline a period in the future, +* because keeping the current (absolute) deadline of the task would +* result in breaking guarantees promised to other tasks. +* +* This function returns true if: +* +* runtime / (deadline - t) > dl_runtime / dl_deadline , I'm a bit confused here about this algorithm. If I state I want to have a run time of 20 ms out of 100 ms, so my
dl_runtime/dl_deadlineis 1/5 or %20.Now if I don't don't get scheduled at the start of my new period because other tasks are running, and I start say, at 50ms in. Thus, my deadline - t == 50, but my runtime remaining is still 20ms. Thus I have 20/50 which is 2/5 or %40. Doing the calculation I have 40 > 20 being true, and thus we label this as an overflow.
Quoted 5 lines
Is this what we really want? Or did I misunderstand something here?
Is this only for tasks that went to sleep voluntarily and have just woken up?
The fact is that this rule is applied only for a task that have just woken up (in general enqueued back). In the case you depicted the task didn't sleep after he was activated. If someone was executing instead of him in its first 50ms it means that that task was having earliest deadline. Since your task set passed the admission control, and since no check is made when a task is simply elected for execution, your task will run with its current parameters having the chance to execute for 20ms in its last 50ms.
Does this clarify?
Quoted 62 lines
Quoted 49 lines
kernel/sched/deadline.c new file +48
-0+* +* IOW we can't recycle current parameters. +*/ +static bool dl_entity_overflow (struct sched_dl_entity *dl_se,u64 t) +{ +u64 left, right; + +/* +* left and right are the two sides of the equation above, +* after a bit of shuffling to use multiplications instead +* of divisions. +* +* Note that none of the time values involved in the two +* multiplications are absolute: dl_deadline and dl_runtime +* are the relative deadline and the maximum runtime of each +* instance, runtime is the runtime left for the last instance +* and (deadline - t), since t is rq->clock, is the time left +* to the (absolute) deadline. Even if overflowing the u64 type +* is very unlikely to occur in both cases, here we scale down +* as we want to avoid that risk at all. Scaling down by 10 +* means that we reduce granularity to 1us. We are fine with it, +* since this is only a true/false check and, anyway, thinking +* of anything below microseconds resolution is actually fiction +* (but still we want to give the user that illusion >;). +*/ + left = (dl_se->dl_deadline >>10 ) * (dl_se->runtime >>10 ); + right = ((dl_se->deadline - t) >>10 ) * (dl_se->dl_runtime >>10 ); + +return dl_time_before (right, left); +} + +/* +* When a -deadline entity is queued back on the runqueue, its runtime and +* deadline might need updating. +* +* The policy here is that we update the deadline of the entity only if: +* - the current deadline is in the past, +* - using the remaining runtime with the current deadline would make +* the entity exceed its bandwidth. +*/ +static void update_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* The arrival of a new instance needs special treatment, i.e., +* the actual scheduling parameters have to be "renewed". kernel/sched/deadline.c line 220
Quoted 10 lines
kernel/sched/deadline.c new file +10
-0+*/ +if (dl_se->dl_new) { +setup_new_dl_entity (dl_se); +return ; + } + +if (dl_time_before (dl_se->deadline,rq_clock (rq)) || +dl_entity_overflow (dl_se,rq_clock (rq))) { + dl_se->deadline =rq_clock (rq) + dl_se->dl_deadline; + dl_se->runtime = dl_se->dl_runtime;In the case that I stated, we just moved the deadline forward.
Quoted 4 lines
Or is this only because it was just enqueued, and we fear that it will starve out other tasks? That is, this only happens if the task was sleeping and just woke up.
Yes, only if the task was sleeping.
Thanks,
- Juri
Quoted 569 lines
Quoted 568 lines
kernel/sched/deadline.c new file +462
-0+ } +} + +/* +* If the entity depleted all its runtime, and if we want it to sleep +* while waiting for some new execution time to become available, we +* set the bandwidth enforcement timer to the replenishment instant +* and try to activate it. +* +* Notice that it is important for the caller to know if the timer +* actually started or not (i.e., the replenishment instant is in +* the future or in the past). +*/ +static int start_dl_timer (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rq *rq =rq_of_dl_rq (dl_rq); +ktime_t now, act; +ktime_t soft, hard; +unsigned long range; +s64 delta; + +/* +* We want the timer to fire at the deadline, but considering +* that it is actually coming from rq->clock and not from +* hrtimer's time base reading. +*/ + act =ns_to_ktime (dl_se->deadline); + now =hrtimer_cb_get_time (&dl_se->dl_timer); + delta =ktime_to_ns (now) -rq_clock (rq); + act =ktime_add_ns (act, delta); + +/* +* If the expiry time already passed, e.g., because the value +* chosen as the deadline is too small, don't even try to +* start the timer in the past! +*/ +if (ktime_us_delta (act, now) <0 ) +return 0 ; + +hrtimer_set_expires (&dl_se->dl_timer, act); + + soft =hrtimer_get_softexpires (&dl_se->dl_timer); + hard =hrtimer_get_expires (&dl_se->dl_timer); + range =ktime_to_ns (ktime_sub (hard, soft)); +__hrtimer_start_range_ns (&dl_se->dl_timer, soft, + range,HRTIMER_MODE_ABS ,0 ); + +return hrtimer_active (&dl_se->dl_timer); +} + +/* +* This is the bandwidth enforcement timer callback. If here, we know +* a task is not on its dl_rq, since the fact that the timer was running +* means the task is throttled and needs a runtime replenishment. +* +* However, what we actually do depends on the fact the task is active, +* (it is on its rq) or has been removed from there by a call to +* dequeue_task_dl(). In the former case we must issue the runtime +* replenishment and add the task back to the dl_rq; in the latter, we just +* do nothing but clearing dl_throttled, so that runtime and deadline +* updating (and the queueing back to dl_rq) will be done by the +* next call to enqueue_task_dl(). +*/ +static enum hrtimer_restart dl_task_timer (struct hrtimer *timer) +{ +struct sched_dl_entity *dl_se =container_of (timer, + struct sched_dl_entity, + dl_timer); +struct task_struct *p =dl_task_of (dl_se); +struct rq *rq =task_rq (p); +raw_spin_lock (&rq->lock); + +/* +* We need to take care of a possible races here. In fact, the +* task might have changed its scheduling policy to something +* different from SCHED_DEADLINE or changed its reservation +* parameters (through sched_setscheduler()). +*/ +if (!dl_task (p) || dl_se->dl_new) + goto unlock; + + dl_se->dl_throttled =0 ; +if (p->on_rq) { +enqueue_task_dl (rq, p,ENQUEUE_REPLENISH ); +if (task_has_dl_policy (rq->curr)) +check_preempt_curr_dl (rq, p,0 ); +else +resched_task (rq->curr); + } +unlock: +raw_spin_unlock (&rq->lock); + +return HRTIMER_NORESTART ; +} + +void init_dl_task_timer (struct sched_dl_entity *dl_se) +{ +struct hrtimer *timer = &dl_se->dl_timer; + +if (hrtimer_active (timer)) { +hrtimer_try_to_cancel (timer); +return ; + } + +hrtimer_init (timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); + timer->function = dl_task_timer; +} + +static +int dl_runtime_exceeded (struct rq *rq,struct sched_dl_entity *dl_se) +{ +int dmiss =dl_time_before (dl_se->deadline,rq_clock (rq)); +int rorun = dl_se->runtime <=0 ; + +if (!rorun && !dmiss) +return 0 ; + +/* +* If we are beyond our current deadline and we are still +* executing, then we have already used some of the runtime of +* the next instance. Thus, if we do not account that, we are +* stealing bandwidth from the system at each deadline miss! +*/ +if (dmiss) { + dl_se->runtime = rorun ? dl_se->runtime :0 ; + dl_se->runtime -=rq_clock (rq) - dl_se->deadline; + } + +return 1 ; +} + +/* +* Update the current task's runtime statistics (provided it is still +* a -deadline task and has not been removed from the dl_rq). +*/ +static void update_curr_dl (struct rq *rq) +{ +struct task_struct *curr = rq->curr; +struct sched_dl_entity *dl_se = &curr->dl; +u64 delta_exec; + +if (!dl_task (curr) || !on_dl_rq (dl_se)) +return ; + +/* +* Consumed budget is computed considering the time as +* observed by schedulable tasks (excluding time spent +* in hardirq context, etc.). Deadlines are instead +* computed using hard walltime. This seems to be the more +* natural solution, but the full ramifications of this +* approach need further study. +*/ + delta_exec =rq_clock_task (rq) - curr->se.exec_start; +if (unlikely ((s64 )delta_exec <0 )) + delta_exec =0 ; + +schedstat_set (curr->se.statistics.exec_max, +max (curr->se.statistics.exec_max, delta_exec)); + + curr->se.sum_exec_runtime += delta_exec; +account_group_exec_runtime (curr, delta_exec); + + curr->se.exec_start =rq_clock_task (rq); +cpuacct_charge (curr, delta_exec); + + dl_se->runtime -= delta_exec; +if (dl_runtime_exceeded (rq, dl_se)) { +__dequeue_task_dl (rq, curr,0 ); +if (likely (start_dl_timer (dl_se))) + dl_se->dl_throttled =1 ; +else +enqueue_task_dl (rq, curr,ENQUEUE_REPLENISH ); + +if (!is_leftmost (curr, &rq->dl)) +resched_task (curr); + } +} + +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); +struct rb_node **link = &dl_rq->rb_root.rb_node; +struct rb_node *parent =NULL ; +struct sched_dl_entity *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&dl_se->rb_node)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct sched_dl_entity, rb_node); +if (dl_time_before (dl_se->deadline, entry->deadline)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->rb_leftmost = &dl_se->rb_node; + +rb_link_node (&dl_se->rb_node, parent, link); +rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); + + dl_rq->dl_nr_running++; +} + +static void __dequeue_dl_entity (struct sched_dl_entity *dl_se) +{ +struct dl_rq *dl_rq =dl_rq_of_se (dl_se); + +if (RB_EMPTY_NODE (&dl_se->rb_node)) +return ; + +if (dl_rq->rb_leftmost == &dl_se->rb_node) { +struct rb_node *next_node; + + next_node =rb_next (&dl_se->rb_node); + dl_rq->rb_leftmost = next_node; + } + +rb_erase (&dl_se->rb_node, &dl_rq->rb_root); +RB_CLEAR_NODE (&dl_se->rb_node); + + dl_rq->dl_nr_running--; +} + +static void +enqueue_dl_entity (struct sched_dl_entity *dl_se,int flags) +{ +BUG_ON (on_dl_rq (dl_se)); + +/* +* If this is a wakeup or a new instance, the scheduling +* parameters of the task might need updating. Otherwise, +* we want a replenishment of its runtime. +*/ +if (!dl_se->dl_new && flags &ENQUEUE_REPLENISH ) +replenish_dl_entity (dl_se); +else +update_dl_entity (dl_se); + +__enqueue_dl_entity (dl_se); +} + +static void dequeue_dl_entity (struct sched_dl_entity *dl_se) +{ +__dequeue_dl_entity (dl_se); +} + +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +/* +* If p is throttled, we do nothing. In fact, if it exhausted +* its budget it needs a replenishment and, since it now is on +* its rq, the bandwidth timer callback (which clearly has not +* run yet) will take care of this. +*/ +if (p->dl.dl_throttled) +return ; + +enqueue_dl_entity (&p->dl, flags); +inc_nr_running (rq); +} + +static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +dequeue_dl_entity (&p->dl); +} + +static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) +{ +update_curr_dl (rq); +__dequeue_task_dl (rq, p, flags); + +dec_nr_running (rq); +} + +/* +* Yield task semantic for -deadline tasks is: +* +* get off from the CPU until our next instance, with +* a new runtime. This is of little use now, since we +* don't have a bandwidth reclaiming mechanism. Anyway, +* bandwidth reclaiming is planned for the future, and +* yield_task_dl will indicate that some spare budget +* is available for other task instances to use it. +*/ +static void yield_task_dl (struct rq *rq) +{ +struct task_struct *p = rq->curr; + +/* +* We make the task go to sleep until its current deadline by +* forcing its runtime to zero. This way, update_curr_dl() stops +* it and the bandwidth timer will wake it up and will give it +* new scheduling parameters (thanks to dl_new=1). +*/ +if (p->dl.runtime >0 ) { + rq->curr->dl.dl_new =1 ; + p->dl.runtime =0 ; + } +update_curr_dl (rq); +} + +/* +* Only called when both the current and waking task are -deadline +* tasks. +*/ +static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p, +int flags) +{ +if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +resched_task (rq->curr); +} + +#ifdef CONFIG_SCHED_HRTICK +static void start_hrtick_dl (struct rq *rq,struct task_struct *p) +{ +s64 delta = p->dl.dl_runtime - p->dl.runtime; + +if (delta >10000 ) +hrtick_start (rq, p->dl.runtime); +} +#endif + +static struct sched_dl_entity *pick_next_dl_entity (struct rq *rq, +struct dl_rq *dl_rq) +{ +struct rb_node *left = dl_rq->rb_leftmost; + +if (!left) +return NULL ; + + return rb_entry(left, struct sched_dl_entity, rb_node); +} + +struct task_struct *pick_next_task_dl (struct rq *rq) +{ +struct sched_dl_entity *dl_se; +struct task_struct *p; +struct dl_rq *dl_rq; + + dl_rq = &rq->dl; + +if (unlikely (!dl_rq->dl_nr_running)) +return NULL ; + + dl_se =pick_next_dl_entity (rq, dl_rq); +BUG_ON (!dl_se); + + p =dl_task_of (dl_se); + p->se.exec_start =rq_clock_task (rq); +#ifdef CONFIG_SCHED_HRTICK +if (hrtick_enabled (rq)) +start_hrtick_dl (rq, p); +#endif +return p; +} + +static void put_prev_task_dl (struct rq *rq,struct task_struct *p) +{ +update_curr_dl (rq); +} + +static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued) +{ +update_curr_dl (rq); + +#ifdef CONFIG_SCHED_HRTICK +if (hrtick_enabled (rq) && queued && p->dl.runtime >0 ) +start_hrtick_dl (rq, p); +#endif +} + +static void task_fork_dl (struct task_struct *p) +{ +/* +* SCHED_DEADLINE tasks cannot fork and this is achieved through +* sched_fork() +*/ +} + +static void task_dead_dl (struct task_struct *p) +{ +struct hrtimer *timer = &p->dl.dl_timer; + +if (hrtimer_active (timer)) +hrtimer_try_to_cancel (timer); +} + +static void set_curr_task_dl (struct rq *rq) +{ +struct task_struct *p = rq->curr; + + p->se.exec_start =rq_clock_task (rq); +} + +static void switched_from_dl (struct rq *rq,struct task_struct *p) +{ +if (hrtimer_active (&p->dl.dl_timer)) +hrtimer_try_to_cancel (&p->dl.dl_timer); +} + +static void switched_to_dl (struct rq *rq,struct task_struct *p) +{ +/* +* If p is throttled, don't consider the possibility +* of preempting rq->curr, the check will be done right +* after its runtime will get replenished. +*/ +if (unlikely (p->dl.dl_throttled)) +return ; + +if (!p->on_rq || rq->curr != p) { +if (task_has_dl_policy (rq->curr)) +check_preempt_curr_dl (rq, p,0 ); +else +resched_task (rq->curr); + } +} + +static void prio_changed_dl (struct rq *rq,struct task_struct *p, +int oldprio) +{ +switched_to_dl (rq, p); +} + +#ifdef CONFIG_SMP +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +return task_cpu (p); +} +#endif + +const struct sched_class dl_sched_class = { + .next = &rt_sched_class, + .enqueue_task = enqueue_task_dl, + .dequeue_task = dequeue_task_dl, + .yield_task = yield_task_dl, + + .check_preempt_curr = check_preempt_curr_dl, + + .pick_next_task = pick_next_task_dl, + .put_prev_task = put_prev_task_dl, + +#ifdef CONFIG_SMP + .select_task_rq = select_task_rq_dl, +#endif + + .set_curr_task = set_curr_task_dl, + .task_tick = task_tick_dl, + .task_fork = task_fork_dl, + .task_dead = task_dead_dl, + + .prio_changed = prio_changed_dl, + .switched_from = switched_from_dl, + .switched_to = switched_to_dl, +};kernel/sched/sched.h +28
-0@@ -2,6 +2,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/mutex.h> #include <linux/spinlock.h> #include <linux/stop_machine.h> @@ -87,11 +88,23 @@ static inline int rt_policy(int policy)return 0 ; } +static inline int dl_policy (int policy) +{ +if (unlikely (policy ==SCHED_DEADLINE )) +return 1 ; +return 0 ; +} +static inline int task_has_rt_policy (struct task_struct *p) {return rt_policy (p->policy); } +static inline int task_has_dl_policy (struct task_struct *p) +{ +return dl_policy (p->policy); +} +/* * This is the priority-queue data structure of the RT scheduling class: */ @@ -363,6 +376,15 @@ struct rt_rq {#endif }; +/* Deadline class' related fields in a runqueue */ +struct dl_rq { +/* runqueue is an rbtree, ordered by deadline */ +struct rb_root rb_root; +struct rb_node *rb_leftmost; + +unsigned long dl_nr_running; +}; +#ifdef CONFIG_SMP /* @@ -427,6 +449,7 @@ struct rq {struct cfs_rq cfs;struct rt_rq rt; +struct dl_rq dl;#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */ @@ -957,6 +980,7 @@ static const u32 prio_to_wmult[40] = {#else #define ENQUEUE_WAKING 0#endif +#define ENQUEUE_REPLENISH 8#define DEQUEUE_SLEEP 1@@ -1012,6 +1036,7 @@ struct sched_class {for (class = sched_class_highest; class; class = class->next) extern const struct sched_class stop_sched_class; +extern const struct sched_class dl_sched_class;extern const struct sched_class rt_sched_class;extern const struct sched_class fair_sched_class;extern const struct sched_class idle_sched_class;@@ -1047,6 +1072,8 @@ extern void resched_cpu(int cpu);extern struct rt_bandwidth def_rt_bandwidth;extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); +extern void init_dl_task_timer (struct sched_dl_entity *dl_se); +extern void update_idle_cpu_load (struct rq *this_rq);extern void init_task_runnable_average (struct task_struct *p);@@ -1305,6 +1332,7 @@ extern void print_rt_stats(struct seq_file *m, int cpu);extern void init_cfs_rq (struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq); +extern void init_dl_rq (struct dl_rq *rt_rq,struct rq *rq);extern void account_cfs_bandwidth_used (int enabled,int was_enabled);kernel/sched/stop_task.c +1
-1@@ -103,7 +103,7 @@ get_rr_interval_stop(struct rq *rq, struct task_struct *task)* Simple, special scheduling class for the per-CPU stop tasks: */ const struct sched_class stop_sched_class = { - .next = &rt_sched_class, + .next = &dl_sched_class, .enqueue_task = enqueue_task_stop, .dequeue_task = dequeue_task_stop,
Steven Rostedt
maintainerRe: [PATCH 03/14] sched: SCHED_DEADLINE structures & implementation.
In reply to Juri Lellikernel/sched/core.c __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) · line 3331
Quoted 10 lines · On Thu, 7 Nov 2013 14:43:37 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/core.c +10
-0+/* +* This function validates the new parameters of a -deadline task. +* We ask for the deadline not being zero, and greater or equal +* than the runtime. +*/ +static bool +__checkparam_dl (const struct sched_param2 *prm) +{ +return prm && (&prm->sched_deadline) !=0 && + (s64)(&prm->sched_deadline - &prm->sched_runtime) >=0 ;
Patch 6 brought this to my attention. Looks like using the address of the fields is wrong. I know patch 6 fixes this, but lets make it correct in this patch first.
Thanks,
-- Steve
Quoted 6 lines
kernel/sched/core.c +3
-0+} + +/* * check the target process has a UID that matches the current process's */ static bool check_same_owner (struct task_struct *p)
Re: [PATCH 03/14] sched: SCHED_DEADLINE structures & implementation.
Quoted 4 lines · Steven Rostedt
On Thu, 7 Nov 2013 14:43:37 +0100
Juri Lelli <[email protected]> wrote:
Quoted 15 lines
kernel/sched/core.c __setscheduler(struct rq *rq, struct task_struct *p, int policy, int prio) · line 3331
Quoted 10 lines
kernel/sched/core.c +10
-0+/* +* This function validates the new parameters of a -deadline task. +* We ask for the deadline not being zero, and greater or equal +* than the runtime. +*/ +static bool +__checkparam_dl (const struct sched_param2 *prm) +{ +return prm && (&prm->sched_deadline) !=0 && + (s64)(&prm->sched_deadline - &prm->sched_runtime) >=0 ;Patch 6 brought this to my attention. Looks like using the address of the fields is wrong. I know patch 6 fixes this, but lets make it correct in this patch first.
Fixed.
Thanks,
- Juri
Quoted 7 lines
Quoted 7 lines
kernel/sched/core.c +3
-0+} + +/* * check the target process has a UID that matches the current process's */ static bool check_same_owner (struct task_struct *p)
tip-bot for Dario Faggioli
merge bot[tip:sched/core] sched/deadline: Add SCHED_DEADLINE structures & implementation
In reply to Juri Lellisched/deadline: Add SCHED_DEADLINE structures & implementation
Commit-ID: aab03e05e8f7e26f51dee792beddcb5cca9215a5 Gitweb: http://git.kernel.org/tip/aab03e05e8f7e26f51dee792beddcb5cca9215a5 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 28 Nov 2013 11:14:43 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:06 +0100
sched/deadline: Add SCHED_DEADLINE structures & implementation
Introduces the data structures, constants and symbols needed for SCHED_DEADLINE implementation.
Core data structure of SCHED_DEADLINE are defined, along with their initializers. Hooks for checking if a task belong to the new policy are also added where they are needed.
Adds a scheduling class, in sched/dl.c and a new policy called SCHED_DEADLINE. It is an implementation of the Earliest Deadline First (EDF) scheduling algorithm, augmented with a mechanism (called Constant Bandwidth Server, CBS) that makes it possible to isolate the behaviour of tasks between each other.
The typical -deadline task will be made up of a computation phase (instance) which is activated on a periodic or sporadic fashion. The expected (maximum) duration of such computation is called the task's runtime; the time interval by which each instance need to be completed is called the task's relative deadline. The task's absolute deadline is dynamically calculated as the time instant a task (better, an instance) activates plus the relative deadline.
The EDF algorithms selects the task with the smallest absolute deadline as the one to be executed first, while the CBS ensures each task to run for at most its runtime every (relative) deadline length time interval, avoiding any interference between different tasks (bandwidth isolation).
Thanks to this feature, also tasks that do not strictly comply with the computational model sketched above can effectively use the new policy.
To summarize, this patch:
- introduces the data structures, constants and symbols needed;
- implements the core logic of the scheduling algorithm in the new scheduling class file;
- provides all the glue code between the new scheduling class and the core scheduler and refines the interactions between sched/dl and the other existing scheduling classes.
--- include/linux/sched.h | 46 ++- include/linux/sched/deadline.h | 24 ++ include/uapi/linux/sched.h | 1 + kernel/fork.c | 4 +- kernel/hrtimer.c | 3 +- kernel/sched/Makefile | 3 +- kernel/sched/core.c | 109 ++++++- kernel/sched/deadline.c | 684 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 26 ++ kernel/sched/stop_task.c | 2 +- 10 files changed, 882 insertions(+), 20 deletions(-)
include/linux/sched.h +45 -1
@@ -97,6 +97,10 @@ struct sched_param {* Given this task model, there are a multiplicity of scheduling algorithms * and policies, that can be used to ensure all the tasks will make their * timing constraints. +* +* As of now, the SCHED_DEADLINE policy (sched_dl scheduling class) is the +* only user of this new interface. More information about the algorithm +* available in the scheduling class file or in Documentation/. */ struct sched_attr {u32 size;
@@ -1088,6 +1092,45 @@ struct sched_rt_entity {#endif }; +struct sched_dl_entity { +struct rb_node rb_node; + +/* +* Original scheduling parameters. Copied here from sched_attr +* during sched_setscheduler2(), they will remain the same until +* the next sched_setscheduler2(). +*/ +u64 dl_runtime;/* maximum runtime for each instance */ +u64 dl_deadline;/* relative deadline of each instance */ + +/* +* Actual scheduling parameters. Initialized with the values above, +* they are continously updated during task execution. Note that +* the remaining runtime could be < 0 in case we are in overrun. +*/ +s64 runtime;/* remaining runtime for this instance */ +u64 deadline;/* absolute deadline for this instance */ +unsigned int flags;/* specifying the scheduler behaviour */ + +/* +* Some bool flags: +* +* @dl_throttled tells if we exhausted the runtime. If so, the +* task has to wait for a replenishment to be performed at the +* next firing of dl_timer. +* +* @dl_new tells if a new instance arrived. If so we must +* start executing it with full runtime and reset its absolute +* deadline; +*/ +int dl_throttled, dl_new; + +/* +* Bandwidth enforcement timer. Each -deadline task has its +* own bandwidth to be enforced, thus we need one timer per task. +*/ +struct hrtimer dl_timer; +};struct rcu_node ;
@@ -1124,6 +1167,7 @@ struct task_struct {#ifdef CONFIG_CGROUP_SCHED struct task_group *sched_task_group;#endif +struct sched_dl_entity dl;#ifdef CONFIG_PREEMPT_NOTIFIERS /* list of struct preempt_notifier: */
@@ -2099,7 +2143,7 @@ extern void wake_up_new_task(struct task_struct *tsk);#else static inline void kick_process (struct task_struct *tsk) { }#endif -extern void sched_fork (unsigned long clone_flags,struct task_struct *p); +extern int sched_fork (unsigned long clone_flags,struct task_struct *p);extern void sched_dead (struct task_struct *p);extern void proc_caches_init (void );
include/linux/sched/deadline.h new file +24 -0
@@ -0,0 +1,24 @@ +#ifndef _SCHED_DEADLINE_H +#define _SCHED_DEADLINE_H + +/* +* SCHED_DEADLINE tasks has negative priorities, reflecting +* the fact that any of them has higher prio than RT and +* NORMAL/BATCH tasks. +*/ + +#define MAX_DL_PRIO 0 + +static inline int dl_prio (int prio) +{ +if (unlikely (prio <MAX_DL_PRIO )) +return 1 ; +return 0 ; +} + +static inline int dl_task (struct task_struct *p) +{ +return dl_prio (p->prio); +} + +#endif /* _SCHED_DEADLINE_H */
include/uapi/linux/sched.h +1 -0
@@ -39,6 +39,7 @@#define SCHED_BATCH 3/* SCHED_ISO: reserved but not implemented yet */ #define SCHED_IDLE 5 +#define SCHED_DEADLINE 6/* Can be ORed in to make sure the process is reverted back to SCHED_NORMAL on fork */ #define SCHED_RESET_ON_FORK 0x40000000
kernel/fork.c +3 -1
@@ -1311,7 +1311,9 @@ static struct task_struct *copy_process(unsigned long clone_flags,#endif /* Perform scheduler related setup. Assign this task to a CPU. */ -sched_fork (clone_flags, p); + retval =sched_fork (clone_flags, p); +if (retval) + goto bad_fork_cleanup_policy; retval =perf_event_init_task (p);if (retval)
kernel/hrtimer.c +2 -1
@@ -46,6 +46,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/timer.h> #include <linux/freezer.h>
@@ -1610,7 +1611,7 @@ long hrtimer_nanosleep(struct timespec *rqtp, struct timespec __user *rmtp,unsigned long slack; slack = current->timer_slack_ns; -if (rt_task (current)) +if (dl_task (current) ||rt_task (current)) slack =0 ;hrtimer_init_on_stack (&t.timer, clockid, mode);
kernel/sched/Makefile +2 -1
@@ -11,7 +11,8 @@ ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y) CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer endif -obj-y += core.o proc.o clock.o cputime.o idle_task.o fair.o rt.o stop_task.o +obj-y += core.o proc.o clock.o cputime.o +obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o obj-y += wait.o completion.o obj-$(CONFIG_SMP) += cpupri.o obj-$(CONFIG_SCHED_AUTOGROUP) += auto_group.o
kernel/sched/core.c +94 -15
249 lines not on this page · raw message
kernel/sched/deadline.c new file +684 -0
684 lines not on this page · raw message
kernel/sched/sched.h +26 -0
@@ -2,6 +2,7 @@#include <linux/sched.h> #include <linux/sched/sysctl.h> #include <linux/sched/rt.h> +#include <linux/sched/deadline.h> #include <linux/mutex.h> #include <linux/spinlock.h> #include <linux/stop_machine.h>
@@ -91,11 +92,21 @@ static inline int rt_policy(int policy)return policy ==SCHED_FIFO || policy ==SCHED_RR ; } +static inline int dl_policy (int policy) +{ +return policy ==SCHED_DEADLINE ; +} +static inline int task_has_rt_policy (struct task_struct *p) {return rt_policy (p->policy); } +static inline int task_has_dl_policy (struct task_struct *p) +{ +return dl_policy (p->policy); +} +/* * This is the priority-queue data structure of the RT scheduling class: */
@@ -367,6 +378,15 @@ struct rt_rq {#endif }; +/* Deadline class' related fields in a runqueue */ +struct dl_rq { +/* runqueue is an rbtree, ordered by deadline */ +struct rb_root rb_root; +struct rb_node *rb_leftmost; + +unsigned long dl_nr_running; +}; +#ifdef CONFIG_SMP /*
@@ -435,6 +455,7 @@ struct rq {struct cfs_rq cfs;struct rt_rq rt; +struct dl_rq dl;#ifdef CONFIG_FAIR_GROUP_SCHED /* list of leaf cfs_rq on this cpu: */
@@ -991,6 +1012,7 @@ static const u32 prio_to_wmult[40] = {#else #define ENQUEUE_WAKING 0#endif +#define ENQUEUE_REPLENISH 8#define DEQUEUE_SLEEP 1
@@ -1046,6 +1068,7 @@ struct sched_class {for (class = sched_class_highest; class; class = class->next) extern const struct sched_class stop_sched_class; +extern const struct sched_class dl_sched_class;extern const struct sched_class rt_sched_class;extern const struct sched_class fair_sched_class;extern const struct sched_class idle_sched_class;
@@ -1081,6 +1104,8 @@ extern void resched_cpu(int cpu);extern struct rt_bandwidth def_rt_bandwidth;extern void init_rt_bandwidth (struct rt_bandwidth *rt_b,u64 period,u64 runtime); +extern void init_dl_task_timer (struct sched_dl_entity *dl_se); +extern void update_idle_cpu_load (struct rq *this_rq);extern void init_task_runnable_average (struct task_struct *p);
@@ -1357,6 +1382,7 @@ extern void print_rt_stats(struct seq_file *m, int cpu);extern void init_cfs_rq (struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq); +extern void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq);extern void cfs_bandwidth_usage_inc (void );extern void cfs_bandwidth_usage_dec (void );
kernel/sched/stop_task.c +1 -1
@@ -103,7 +103,7 @@ get_rr_interval_stop(struct rq *rq, struct task_struct *task)* Simple, special scheduling class for the per-CPU stop tasks: */ const struct sched_class stop_sched_class = { - .next = &rt_sched_class, + .next = &dl_sched_class, .enqueue_task = enqueue_task_stop, .dequeue_task = dequeue_task_stop,
PATCH 4/14 sched: SCHED_DEADLINE SMP-related data structures & logic.
In reply to Juri LelliIntroduces data structures relevant for implementing dynamic migration of -deadline tasks and the logic for checking if runqueues are overloaded with -deadline tasks and for choosing where a task should migrate, when it is the case.
Adds also dynamic migrations to SCHED_DEADLINE, so that tasks can be moved among CPUs when necessary. It is also possible to bind a task to a (set of) CPU(s), thus restricting its capability of migrating, or forbidding migrations at all.
The very same approach used in sched_rt is utilised:
- -deadline tasks are kept into CPU-specific runqueues,
- -deadline tasks are migrated among runqueues to achieve the following:
* on an M-CPU system the M earliest deadline ready tasks are always running;
* affinity/cpusets settings of all the -deadline tasks is always respected.
Therefore, this very special form of "load balancing" is done with an active method, i.e., the scheduler pushes or pulls tasks between runqueues when they are woken up and/or (de)scheduled.
IOW, every time a preemption occurs, the descheduled task might be sent to some other CPU (depending on its deadline) to continue executing (push). On the other hand, every time a CPU becomes idle, it might pull the second earliest deadline ready task from some other CPU.
To enforce this, a pull operation is always attempted before taking any scheduling decision (pre_schedule()), as well as a push one after each scheduling decision (post_schedule()). In addition, when a task arrives or wakes up, the best CPU where to resume it is selected taking into account its affinity mask, the system topology, but also its deadline. E.g., from the scheduling point of view, the best CPU where to wake up (and also where to push) a task is the one which is running the task with the latest deadline among the M executing ones.
In order to facilitate these decisions, per-runqueue "caching" of the deadlines of the currently running and of the first ready task is used. Queued but not running tasks are also parked in another rb-tree to speed-up pushes.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 9 +- kernel/sched/deadline.c | 937 ++++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 34 ++ 5 files changed, 966 insertions(+), 17 deletions(-)
include/linux/sched.h +1 -0
@@ -1174,6 +1174,7 @@ struct task_struct {struct list_head tasks;#ifdef CONFIG_SMP struct plist_node pushable_tasks; +struct rb_node pushable_dl_tasks;#endif struct mm_struct *mm, *active_mm;
kernel/sched/core.c +8 -1
@@ -1737,6 +1737,7 @@ int sched_fork(struct task_struct *p)#endif #ifdef CONFIG_SMP plist_node_init (&p->pushable_tasks,MAX_PRIO ); +RB_CLEAR_NODE (&p->pushable_dl_tasks);#endif put_cpu ();
@@ -5148,6 +5149,7 @@ static void free_rootdomain(struct rcu_head *rcu) struct root_domain *rd = container_of(rcu, struct root_domain, rcu);cpupri_cleanup (&rd->cpupri); +free_cpumask_var (rd->dlo_mask);free_cpumask_var (rd->rto_mask);free_cpumask_var (rd->online);free_cpumask_var (rd->span);
@@ -5199,8 +5201,10 @@ static int init_rootdomain(struct root_domain *rd) goto out;if (!alloc_cpumask_var (&rd->online,GFP_KERNEL )) goto free_span; -if (!alloc_cpumask_var (&rd->rto_mask,GFP_KERNEL )) +if (!alloc_cpumask_var (&rd->dlo_mask,GFP_KERNEL )) goto free_online; +if (!alloc_cpumask_var (&rd->rto_mask,GFP_KERNEL )) + goto free_dlo_mask;if (cpupri_init (&rd->cpupri) !=0 ) goto free_rto_mask;
@@ -5208,6 +5212,8 @@ static int init_rootdomain(struct root_domain *rd) free_rto_mask:free_cpumask_var (rd->rto_mask); +free_dlo_mask: +free_cpumask_var (rd->dlo_mask); free_online:free_cpumask_var (rd->online); free_span:
@@ -6542,6 +6548,7 @@ void __init sched_init_smp(void)free_cpumask_var (non_isolated_cpus);init_sched_rt_class (); + init_sched_dl_class(); }#else void __init sched_init_smp (void )
kernel/sched/deadline.c +922 -15
1,058 lines not on this page · raw message
kernel/sched/rt.c +1 -1
@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);
kernel/sched/sched.h +34 -0
@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP
@@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */
@@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Signature · 1 line
1.7.9.5
Steven Rostedt
maintainerRe: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 29 lines · On Thu, 7 Nov 2013 14:43:38 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/deadline.c +10
-0@@ -10,6 +10,7 @@* miss some of their deadlines), and won't affect any other task. * * Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Juri Lelli <[email protected]>, * Michael Trimarchi <[email protected]>, * Fabio Checconi <[email protected]> */ @@ -20,6 +21,15 @@ static inline int dl_time_before(u64 a, u64 b)return (s64)(a - b) <0 ; } +/* +* Tells if entity @a should preempt entity @b. +*/ +static inline +int dl_entity_preempt (struct sched_dl_entity *a,struct sched_dl_entity *b) +{ +return dl_time_before (a->deadline, b->deadline); +} +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) { return container_of(dl_se, struct task_struct, dl);
kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 6 lines
kernel/sched/deadline.c +3
-0@@ -53,8 +63,168 @@ static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) { dl_rq->rb_root =RB_ROOT ; + +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */
I'm curious to why you add the '-' to -deadline.
Quoted 503 lines
kernel/sched/deadline.c +418
-3+ dl_rq->earliest_dl.curr = dl_rq->earliest_dl.next =0 ; + + dl_rq->dl_nr_migratory =0 ; + dl_rq->overloaded =0 ; + dl_rq->pushable_dl_tasks_root =RB_ROOT ; +#endif +} + +#ifdef CONFIG_SMP + +static inline int dl_overloaded (struct rq *rq) +{ +return atomic_read (&rq->rd->dlo_count); +} + +static inline void dl_set_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +cpumask_set_cpu (rq->cpu, rq->rd->dlo_mask); +/* +* Must be visible before the overload count is +* set (as in sched_rt.c). +* +* Matched by the barrier in pull_dl_task(). +*/ +smp_wmb (); +atomic_inc (&rq->rd->dlo_count); +} + +static inline void dl_clear_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +atomic_dec (&rq->rd->dlo_count); +cpumask_clear_cpu (rq->cpu, rq->rd->dlo_mask); +} + +static void update_dl_migration (struct dl_rq *dl_rq) +{ +if (dl_rq->dl_nr_migratory && dl_rq->dl_nr_total >1 ) { +if (!dl_rq->overloaded) { +dl_set_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =1 ; + } + }else if (dl_rq->overloaded) { +dl_clear_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =0 ; + } +} + +static void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total++; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory++; + +update_dl_migration (dl_rq); +} + +static void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total--; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory--; + +update_dl_migration (dl_rq); +} + +/* +* The list of pushable -deadline task is not a plist, like in +* sched_rt.c, it is an rb-tree with tasks ordered by deadline. +*/ +static void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; +struct rb_node **link = &dl_rq->pushable_dl_tasks_root.rb_node; +struct rb_node *parent =NULL ; +struct task_struct *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&p->pushable_dl_tasks)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct task_struct, + pushable_dl_tasks); +if (dl_entity_preempt (&p->dl, &entry->dl)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->pushable_dl_tasks_leftmost = &p->pushable_dl_tasks; + +rb_link_node (&p->pushable_dl_tasks, parent, link); +rb_insert_color (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +} + +static void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; + +if (RB_EMPTY_NODE (&p->pushable_dl_tasks)) +return ; + +if (dl_rq->pushable_dl_tasks_leftmost == &p->pushable_dl_tasks) { +struct rb_node *next_node; + + next_node =rb_next (&p->pushable_dl_tasks); + dl_rq->pushable_dl_tasks_leftmost = next_node; + } + +rb_erase (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +RB_CLEAR_NODE (&p->pushable_dl_tasks); +} + +static inline int has_pushable_dl_tasks (struct rq *rq) +{ +return !RB_EMPTY_ROOT (&rq->dl.pushable_dl_tasks_root); +} + +static int push_dl_task (struct rq *rq); + +#else + +static inline +void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +} + +static inline +void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ } +#endif /* CONFIG_SMP */ +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags);static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,@@ -307,6 +477,14 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)check_preempt_curr_dl (rq, p,0 ); else resched_task(rq->curr); +#ifdef CONFIG_SMP +/* +* Queueing this task back might have overloaded rq, +* check if we need to kick someone away. +*/ +if (has_pushable_dl_tasks (rq)) +push_dl_task (rq); +#endif } unlock:raw_spin_unlock (&rq->lock);@@ -397,6 +575,100 @@ static void update_curr_dl(struct rq *rq) } } +#ifdef CONFIG_SMP + +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu); + +static inline u64 next_deadline (struct rq *rq) +{ +struct task_struct *next =pick_next_earliest_dl_task (rq, rq->cpu); + +if (next &&dl_prio (next->prio)) +return next->dl.deadline; +else +return 0 ; +} + +static void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +if (dl_rq->earliest_dl.curr ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.curr)) { +/* +* If the dl_rq had no -deadline tasks, or if the new task +* has shorter deadline than the current one on dl_rq, we +* know that the previous earliest becomes our next earliest, +* as the new task becomes the earliest itself. +*/ + dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; + dl_rq->earliest_dl.curr = deadline; + }else if (dl_rq->earliest_dl.next ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.next)) { +/* +* On the other hand, if the new -deadline task has a +* a later deadline than the earliest one on dl_rq, but +* it is earlier than the next (if any), we must +* recompute the next-earliest. +*/ + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +static void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* Since we may have removed our earliest (and/or next earliest) +* task we must recompute them. +*/ +if (!dl_rq->dl_nr_running) { + dl_rq->earliest_dl.curr =0 ; + dl_rq->earliest_dl.next =0 ; + }else { +struct rb_node *leftmost = dl_rq->rb_leftmost; +struct sched_dl_entity *entry; + + entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); + dl_rq->earliest_dl.curr = entry->deadline; + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +#else + +static inline void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} +static inline void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} + +#endif /* CONFIG_SMP */ + +static inline +void inc_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; +u64 deadline = dl_se->deadline; + +WARN_ON (!dl_prio (prio)); + dl_rq->dl_nr_running++; + +inc_dl_deadline (dl_rq, deadline); +inc_dl_migration (dl_se, dl_rq); +} + +static inline +void dec_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; + +WARN_ON (!dl_prio (prio)); +WARN_ON (!dl_rq->dl_nr_running); + dl_rq->dl_nr_running--; + +dec_dl_deadline (dl_rq, dl_se->deadline); +dec_dl_migration (dl_se, dl_rq); +} +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se);@@ -424,7 +696,7 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)rb_link_node (&dl_se->rb_node, parent, link);rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); - dl_rq->dl_nr_running++; +inc_dl_tasks (dl_se, dl_rq); }static void __dequeue_dl_entity (struct sched_dl_entity *dl_se)@@ -444,7 +716,7 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)rb_erase (&dl_se->rb_node, &dl_rq->rb_root);RB_CLEAR_NODE (&dl_se->rb_node); - dl_rq->dl_nr_running--; +dec_dl_tasks (dl_se, dl_rq); } static void@@ -482,12 +754,17 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)return ;enqueue_dl_entity (&p->dl, flags); + +if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); +inc_nr_running (rq); }static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {dequeue_dl_entity (&p->dl); +dequeue_pushable_dl_task (rq, p); }static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags)@@ -525,6 +802,77 @@ static void yield_task_dl(struct rq *rq)update_curr_dl (rq); } +#ifdef CONFIG_SMP + +static int find_later_rq (struct task_struct *task); +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask); + +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +struct task_struct *curr; +struct rq *rq; +int cpu; + + cpu =task_cpu (p); + +if (sd_flag !=SD_BALANCE_WAKE && sd_flag !=SD_BALANCE_FORK ) + goto out; + + rq =cpu_rq (cpu); + +rcu_read_lock (); + curr =ACCESS_ONCE (rq->curr);/* unlocked access */ + +/* +* If we are dealing with a -deadline task, we must +* decide where to wake it up. +* If it has a later deadline and the current task +* on this rq can't move (provided the waking task +* can!) we prefer to send it somewhere else. On the +* other hand, if it has a shorter deadline, we +* try to make it stay here, it might be important. +*/ +if (unlikely (dl_task (curr)) && + (curr->nr_cpus_allowed <2 || + !dl_entity_preempt (&p->dl, &curr->dl)) && + (p->nr_cpus_allowed >1 )) { +int target =find_later_rq (p); + +if (target !=-1 ) + cpu = target; + } +rcu_read_unlock (); + +out: +return cpu; +} + +static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p) +{ +/* +* Current can't be migrated, useless to reschedule, +* let's hope p can move out. +*/ +if (rq->curr->nr_cpus_allowed ==1 || +latest_cpu_find (rq->rd->span, rq->curr,NULL ) ==-1 ) +return ; + +/* +* p is migratable, so let's not schedule it and +* see if it is pushed or pulled somewhere else. +*/ +if (p->nr_cpus_allowed !=1 && +latest_cpu_find (rq->rd->span, p,NULL ) !=-1 ) +return ; + +resched_task (rq->curr); +} + +#endif /* CONFIG_SMP */ +/* * Only called when both the current and waking task are -deadline * tasks. @@ -532,8 +880,20 @@ static void yield_task_dl(struct rq *rq)static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +if (dl_entity_preempt (&p->dl, &rq->curr->dl)) {resched_task (rq->curr); +return ; + } + +#ifdef CONFIG_SMP +/* +* In the unlikely case current and p have the same deadline +* let us try to decide what's the best thing to do... +*/ +if ((s64)(p->dl.deadline - rq->curr->dl.deadline) ==0 && + !need_resched ()) +check_preempt_equal_dl (rq, p); +#endif /* CONFIG_SMP */ }#ifdef CONFIG_SCHED_HRTICK @@ -573,16 +933,29 @@ struct task_struct *pick_next_task_dl(struct rq *rq) p =dl_task_of (dl_se); p->se.exec_start =rq_clock_task (rq); + +/* Running task will never be pushed. */ +if (p) +dequeue_pushable_dl_task (rq, p); +#ifdef CONFIG_SCHED_HRTICK if (hrtick_enabled (rq))start_hrtick_dl (rq, p);#endif + +#ifdef CONFIG_SMP + rq->post_schedule =has_pushable_dl_tasks (rq); +#endif /* CONFIG_SMP */ +return p; }static void put_prev_task_dl (struct rq *rq,struct task_struct *p) {update_curr_dl (rq); + +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); }static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued)@@ -616,16 +989,517 @@ static void set_curr_task_dl(struct rq *rq)struct task_struct *p = rq->curr; p->se.exec_start =rq_clock_task (rq); + +/* You can't push away the running task */ +dequeue_pushable_dl_task (rq, p); +} + +#ifdef CONFIG_SMP + +/* Only try algorithms three times */ +#define DL_MAX_TRIES 3 + +static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_running (rq, p) && + (cpu <0 ||cpumask_test_cpu (cpu, &p->cpus_allowed)) && + (p->nr_cpus_allowed >1 )) +return 1 ; + +return 0 ; +} + +/* Returns the second earliest -deadline task, NULL otherwise */ +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu) +{ +struct rb_node *next_node = rq->dl.rb_leftmost; +struct sched_dl_entity *dl_se; +struct task_struct *p =NULL ; + +next_node: + next_node =rb_next (next_node); +if (next_node) { + dl_se = rb_entry(next_node, struct sched_dl_entity, rb_node); + p =dl_task_of (dl_se); + +if (pick_dl_task (rq, p, cpu)) +return p; + + goto next_node; + } + +return NULL ; +} + +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {
Ug, the above is hard to read. What about:
if (!best) {
if (!dl_rq->dl_nr_running) {
best = 1;
found = cpu;
} elsif (dl_time_before(...)) {
...
}
}Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
We should be able to incorporate cpuprio_find() to be dl aware too. That is, work for both -rt and -dl.
Quoted 260 lines
kernel/sched/deadline.c +258
-0+ max_dl = dl_rq->earliest_dl.curr; + found = cpu; + } + }else if (later_mask) +cpumask_clear_cpu (cpu, later_mask); + } + +return found; +} + +static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); + +static int find_later_rq (struct task_struct *task) +{ +struct sched_domain *sd; +struct cpumask *later_mask =__get_cpu_var (local_cpu_mask_dl); +int this_cpu =smp_processor_id (); +int best_cpu, cpu =task_cpu (task); + +/* Make sure the mask is initialized first */ +if (unlikely (!later_mask)) +return -1 ; + +if (task->nr_cpus_allowed ==1 ) +return -1 ; + + best_cpu =latest_cpu_find (task_rq (task)->rd->span, task, later_mask); +if (best_cpu ==-1 ) +return -1 ; + +/* +* If we are here, some target has been found, +* the most suitable of which is cached in best_cpu. +* This is, among the runqueues where the current tasks +* have later deadlines than the task's one, the rq +* with the latest possible one. +* +* Now we check how well this matches with task's +* affinity and system topology. +* +* The last cpu where the task run is our first +* guess, since it is most likely cache-hot there. +*/ +if (cpumask_test_cpu (cpu, later_mask)) +return cpu; +/* +* Check if this_cpu is to be skipped (i.e., it is +* not in the mask) or not. +*/ +if (!cpumask_test_cpu (this_cpu, later_mask)) + this_cpu =-1 ; + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { + +/* +* If possible, preempting this_cpu is +* cheaper than migrating. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + +/* +* Last chance: if best_cpu is valid and is +* in the mask, that becomes our choice. +*/ +if (best_cpu < nr_cpu_ids && +cpumask_test_cpu (best_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* At this point, all our guesses failed, we just return +* 'something', and let the caller sort the things out. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any (later_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* Locks the rq it finds */ +static struct rq *find_lock_later_rq (struct task_struct *task,struct rq *rq) +{ +struct rq *later_rq =NULL ; +int tries; +int cpu; + +for (tries =0 ; tries <DL_MAX_TRIES ; tries++) { + cpu =find_later_rq (task); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + later_rq =cpu_rq (cpu); + +/* Retry if something changed. */ +if (double_lock_balance (rq, later_rq)) { +if (unlikely (task_rq (task) != rq || + !cpumask_test_cpu (later_rq->cpu, + &task->cpus_allowed) || +task_running (rq, task) || !task->on_rq)) { +double_unlock_balance (rq, later_rq); + later_rq =NULL ; +break ; + } + } + +/* +* If the rq we found has no -deadline task, or +* its earliest one has a later deadline than our +* task, the rq is a good one. +*/ +if (!later_rq->dl.dl_nr_running || +dl_time_before (task->dl.deadline, + later_rq->dl.earliest_dl.curr)) +break ; + +/* Otherwise we try again. */ +double_unlock_balance (rq, later_rq); + later_rq =NULL ; + } + +return later_rq; } +static struct task_struct *pick_next_pushable_dl_task (struct rq *rq) +{ +struct task_struct *p; + +if (!has_pushable_dl_tasks (rq)) +return NULL ; + + p =rb_entry (rq->dl.pushable_dl_tasks_leftmost, + struct task_struct, pushable_dl_tasks); + +BUG_ON (rq->cpu !=task_cpu (p)); +BUG_ON (task_current (rq, p)); +BUG_ON (p->nr_cpus_allowed <=1 ); + +BUG_ON (!p->se.on_rq); +BUG_ON (!dl_task (p)); + +return p; +} + +/* +* See if the non running -deadline tasks on this rq +* can be sent to some other CPU where they can preempt +* and start executing. +*/ +static int push_dl_task (struct rq *rq) +{ +struct task_struct *next_task; +struct rq *later_rq; + +if (!rq->dl.overloaded) +return 0 ; + + next_task =pick_next_pushable_dl_task (rq); +if (!next_task) +return 0 ; + +retry: +if (unlikely (next_task == rq->curr)) { +WARN_ON (1 ); +return 0 ; + } + +/* +* If next_task preempts rq->curr, and rq->curr +* can move away, it makes sense to just reschedule +* without going further in pushing next_task. +*/ +if (dl_task (rq->curr) && +dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && + rq->curr->nr_cpus_allowed >1 ) { +resched_task (rq->curr); +return 0 ; + } + +/* We might release rq lock */ +get_task_struct (next_task); + +/* Will lock the rq it'll find */ + later_rq =find_lock_later_rq (next_task, rq); +if (!later_rq) { +struct task_struct *task; + +/* +* We must check all this again, since +* find_lock_later_rq releases rq->lock and it is +* then possible that next_task has migrated. +*/ + task =pick_next_pushable_dl_task (rq); +if (task_cpu (next_task) == rq->cpu && task == next_task) { +/* +* The task is still there. We don't try +* again, some other cpu will pull it when ready. +*/ +dequeue_pushable_dl_task (rq, next_task); + goto out; + } + +if (!task) +/* No more tasks */ + goto out; + +put_task_struct (next_task); + next_task = task; + goto retry; + } + +deactivate_task (rq, next_task,0 ); +set_task_cpu (next_task, later_rq->cpu); +activate_task (later_rq, next_task,0 ); + +resched_task (later_rq->curr); + +double_unlock_balance (rq, later_rq); + +out: +put_task_struct (next_task); + +return 1 ; +} + +static void push_dl_tasks (struct rq *rq) +{ +/* Terminates as it moves a -deadline task */ +while (push_dl_task (rq)) + ; +} + +static int pull_dl_task (struct rq *this_rq) +{ +int this_cpu = this_rq->cpu, ret =0 , cpu; +struct task_struct *p; +struct rq *src_rq; +u64 dmin =LONG_MAX ; + +if (likely (!dl_overloaded (this_rq))) +return 0 ; + +/* +* Match the barrier from dl_set_overloaded; this guarantees that if we +* see overloaded we must also see the dlo_mask bit. +*/
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1326
Quoted 10 lines
kernel/sched/deadline.c +10
-0+smp_rmb (); + +for_each_cpu (cpu, this_rq->rd->dlo_mask) { +if (this_cpu == cpu) +continue ; + + src_rq =cpu_rq (cpu); + +/* +* It looks racy, abd it is! However, as in sched_rt.c,
abd it is?
:-)
-- Steve
Quoted 330 lines
kernel/sched/deadline.c +213
-12+* we are fine with this. +*/ +if (this_rq->dl.dl_nr_running && +dl_time_before (this_rq->dl.earliest_dl.curr, + src_rq->dl.earliest_dl.next)) +continue ; + +/* Might drop this_rq->lock */ +double_lock_balance (this_rq, src_rq); + +/* +* If there are no more pullable tasks on the +* rq, we're done with it. +*/ +if (src_rq->dl.dl_nr_running <=1 ) + goto skip; + + p =pick_next_earliest_dl_task (src_rq, this_cpu); + +/* +* We found a task to be pulled if: +* - it preempts our current (if there's one), +* - it will preempt the last one we pulled (if any). +*/ +if (p &&dl_time_before (p->dl.deadline, dmin) && + (!this_rq->dl.dl_nr_running || +dl_time_before (p->dl.deadline, + this_rq->dl.earliest_dl.curr))) { +WARN_ON (p == src_rq->curr); +WARN_ON (!p->se.on_rq); + +/* +* Then we pull iff p has actually an earlier +* deadline than the current task of its runqueue. +*/ +if (dl_time_before (p->dl.deadline, + src_rq->curr->dl.deadline)) + goto skip; + + ret =1 ; + +deactivate_task (src_rq, p,0 ); +set_task_cpu (p, this_cpu); +activate_task (this_rq, p,0 ); + dmin = p->dl.deadline; + +/* Is there any other task even earlier? */ + } +skip: + double_unlock_balance(this_rq, src_rq); + } + +return ret; +} + +static void pre_schedule_dl (struct rq *rq,struct task_struct *prev) +{ +/* Try to pull other tasks here */ +if (dl_task (prev)) +pull_dl_task (rq); +} + +static void post_schedule_dl (struct rq *rq) +{ +push_dl_tasks (rq); +} + +/* +* Since the task is not running and a reschedule is not going to happen +* anytime soon on its runqueue, we try pushing it away now. +*/ +static void task_woken_dl (struct rq *rq,struct task_struct *p) +{ +if (!task_running (rq, p) && + !test_tsk_need_resched (rq->curr) && +has_pushable_dl_tasks (rq) && + p->nr_cpus_allowed >1 && +dl_task (rq->curr) && + (rq->curr->nr_cpus_allowed <2 || +dl_entity_preempt (&rq->curr->dl, &p->dl))) { +push_dl_tasks (rq); + } +} + +static void set_cpus_allowed_dl (struct task_struct *p, +const struct cpumask *new_mask) +{ +struct rq *rq; +int weight; + +BUG_ON (!dl_task (p)); + +/* +* Update only if the task is actually running (i.e., +* it is on the rq AND it is not throttled). +*/ +if (!on_dl_rq (&p->dl)) +return ; + + weight =cpumask_weight (new_mask); + +/* +* Only update if the process changes its state from whether it +* can migrate or not. +*/ +if ((p->nr_cpus_allowed >1 ) == (weight >1 )) +return ; + + rq =task_rq (p); + +/* +* The process used to be able to migrate OR it can now migrate +*/ +if (weight <=1 ) { +if (!task_current (rq, p)) +dequeue_pushable_dl_task (rq, p); +BUG_ON (!rq->dl.dl_nr_migratory); + rq->dl.dl_nr_migratory--; + }else { +if (!task_current (rq, p)) +enqueue_pushable_dl_task (rq, p); + rq->dl.dl_nr_migratory++; + } + +update_dl_migration (&rq->dl); +} + +/* Assumes rq->lock is held */ +static void rq_online_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_set_overload (rq); +} + +/* Assumes rq->lock is held */ +static void rq_offline_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_clear_overload (rq); +} + +void init_sched_dl_class (void ) +{ +unsigned int i; + +for_each_possible_cpu (i) +zalloc_cpumask_var_node (&per_cpu (local_cpu_mask_dl, i), +GFP_KERNEL ,cpu_to_node (i)); +} + +#endif /* CONFIG_SMP */ +static void switched_from_dl (struct rq *rq,struct task_struct *p) { -if (hrtimer_active (&p->dl.dl_timer)) +if (hrtimer_active (&p->dl.dl_timer) && !dl_policy (p->policy))hrtimer_try_to_cancel (&p->dl.dl_timer); + +#ifdef CONFIG_SMP +/* +* Since this might be the only -deadline task on the rq, +* this is the right place to try to pull some other one +* from an overloaded cpu, if any. +*/ +if (!rq->dl.dl_nr_running) +pull_dl_task (rq); +#endif } +/* +* When switching to -deadline, we may overload the rq, then +* we try to push someone off, if possible. +*/ static void switched_to_dl (struct rq *rq,struct task_struct *p) { +int check_resched =1 ; +/* * If p is throttled, don't consider the possibility * of preempting rq->curr, the check will be done right @@ -635,26 +1509,53 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ;if (!p->on_rq || rq->curr != p) { -if (task_has_dl_policy (rq->curr)) +#ifdef CONFIG_SMP +if (rq->dl.overloaded &&push_dl_task (rq) && rq !=task_rq (p)) +/* Only reschedule if pushing failed */ + check_resched =0 ; +#endif /* CONFIG_SMP */ +if (check_resched &&task_has_dl_policy (rq->curr))check_preempt_curr_dl (rq, p,0 ); -else -resched_task (rq->curr); } } +/* +* If the scheduling parameters of a -deadline task changed, +* a push or pull operation might be needed. +*/ static void prio_changed_dl (struct rq *rq,struct task_struct *p,int oldprio) { -switched_to_dl (rq, p); -} - +if (p->on_rq || rq->curr == p) {#ifdef CONFIG_SMP -static int -select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) -{ -return task_cpu (p); +/* +* This might be too much, but unfortunately +* we don't have the old deadline value, and +* we can't argue if the task is increasing +* or lowering its prio, so... +*/ +if (!rq->dl.overloaded) +pull_dl_task (rq); + +/* +* If we now have a earlier deadline task than p, +* then reschedule, provided p is still on this +* runqueue. +*/ +if (dl_time_before (rq->dl.earliest_dl.curr, p->dl.deadline) && + rq->curr == p) +resched_task (p); +#else +/* +* Again, we don't know if p has a earlier +* or later deadline, so let's blindly set a +* (maybe not needed) rescheduling point. +*/ +resched_task (p); +#endif /* CONFIG_SMP */ + }else +switched_to_dl (rq, p); } -#endif const struct sched_class dl_sched_class = { .next = &rt_sched_class,@@ -669,6 +1570,12 @@ const struct sched_class dl_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl, + .set_cpus_allowed = set_cpus_allowed_dl, + .rq_online = rq_online_dl, + .rq_offline = rq_offline_dl, + .pre_schedule = pre_schedule_dl, + .post_schedule = post_schedule_dl, + .task_woken = task_woken_dl, #endif .set_curr_task = set_curr_task_dl,kernel/sched/rt.c +1
-1@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);kernel/sched/sched.h +34
-0@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP @@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ @@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Re: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 33 lines · Steven Rostedt
Quoted 29 lines · On Thu, 7 Nov 2013 14:43:38 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/deadline.c +10
-0@@ -10,6 +10,7 @@* miss some of their deadlines), and won't affect any other task. * * Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Juri Lelli <[email protected]>, * Michael Trimarchi <[email protected]>, * Fabio Checconi <[email protected]> */ @@ -20,6 +21,15 @@ static inline int dl_time_before(u64 a, u64 b)return (s64)(a - b) <0 ; } +/* +* Tells if entity @a should preempt entity @b. +*/ +static inline +int dl_entity_preempt (struct sched_dl_entity *a,struct sched_dl_entity *b) +{ +return dl_time_before (a->deadline, b->deadline); +} +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) { return container_of(dl_se, struct task_struct, dl);@@ -53,8 +63,168 @@ static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)
Quoted 9 lines
kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 6 lines
kernel/sched/deadline.c +3
-0void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) { dl_rq->rb_root =RB_ROOT ; + +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */ I'm curious to why you add the '-' to -deadline.
I guess "deadline tasks" is too much generic; "SCHED_DEADLINE tasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.
Quoted 517 lines
Quoted 504 lines
kernel/sched/deadline.c +418
-3+ dl_rq->earliest_dl.curr = dl_rq->earliest_dl.next =0 ; + + dl_rq->dl_nr_migratory =0 ; + dl_rq->overloaded =0 ; + dl_rq->pushable_dl_tasks_root =RB_ROOT ; +#endif +} + +#ifdef CONFIG_SMP + +static inline int dl_overloaded (struct rq *rq) +{ +return atomic_read (&rq->rd->dlo_count); +} + +static inline void dl_set_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +cpumask_set_cpu (rq->cpu, rq->rd->dlo_mask); +/* +* Must be visible before the overload count is +* set (as in sched_rt.c). +* +* Matched by the barrier in pull_dl_task(). +*/ +smp_wmb (); +atomic_inc (&rq->rd->dlo_count); +} + +static inline void dl_clear_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +atomic_dec (&rq->rd->dlo_count); +cpumask_clear_cpu (rq->cpu, rq->rd->dlo_mask); +} + +static void update_dl_migration (struct dl_rq *dl_rq) +{ +if (dl_rq->dl_nr_migratory && dl_rq->dl_nr_total >1 ) { +if (!dl_rq->overloaded) { +dl_set_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =1 ; + } + }else if (dl_rq->overloaded) { +dl_clear_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =0 ; + } +} + +static void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total++; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory++; + +update_dl_migration (dl_rq); +} + +static void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total--; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory--; + +update_dl_migration (dl_rq); +} + +/* +* The list of pushable -deadline task is not a plist, like in +* sched_rt.c, it is an rb-tree with tasks ordered by deadline. +*/ +static void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; +struct rb_node **link = &dl_rq->pushable_dl_tasks_root.rb_node; +struct rb_node *parent =NULL ; +struct task_struct *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&p->pushable_dl_tasks)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct task_struct, + pushable_dl_tasks); +if (dl_entity_preempt (&p->dl, &entry->dl)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->pushable_dl_tasks_leftmost = &p->pushable_dl_tasks; + +rb_link_node (&p->pushable_dl_tasks, parent, link); +rb_insert_color (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +} + +static void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; + +if (RB_EMPTY_NODE (&p->pushable_dl_tasks)) +return ; + +if (dl_rq->pushable_dl_tasks_leftmost == &p->pushable_dl_tasks) { +struct rb_node *next_node; + + next_node =rb_next (&p->pushable_dl_tasks); + dl_rq->pushable_dl_tasks_leftmost = next_node; + } + +rb_erase (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +RB_CLEAR_NODE (&p->pushable_dl_tasks); +} + +static inline int has_pushable_dl_tasks (struct rq *rq) +{ +return !RB_EMPTY_ROOT (&rq->dl.pushable_dl_tasks_root); +} + +static int push_dl_task (struct rq *rq); + +#else + +static inline +void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +} + +static inline +void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ } +#endif /* CONFIG_SMP */ +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags);static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,@@ -307,6 +477,14 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)check_preempt_curr_dl (rq, p,0 ); else resched_task(rq->curr); +#ifdef CONFIG_SMP +/* +* Queueing this task back might have overloaded rq, +* check if we need to kick someone away. +*/ +if (has_pushable_dl_tasks (rq)) +push_dl_task (rq); +#endif } unlock:raw_spin_unlock (&rq->lock);@@ -397,6 +575,100 @@ static void update_curr_dl(struct rq *rq) } } +#ifdef CONFIG_SMP + +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu); + +static inline u64 next_deadline (struct rq *rq) +{ +struct task_struct *next =pick_next_earliest_dl_task (rq, rq->cpu); + +if (next &&dl_prio (next->prio)) +return next->dl.deadline; +else +return 0 ; +} + +static void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +if (dl_rq->earliest_dl.curr ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.curr)) { +/* +* If the dl_rq had no -deadline tasks, or if the new task +* has shorter deadline than the current one on dl_rq, we +* know that the previous earliest becomes our next earliest, +* as the new task becomes the earliest itself. +*/ + dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; + dl_rq->earliest_dl.curr = deadline; + }else if (dl_rq->earliest_dl.next ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.next)) { +/* +* On the other hand, if the new -deadline task has a +* a later deadline than the earliest one on dl_rq, but +* it is earlier than the next (if any), we must +* recompute the next-earliest. +*/ + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +static void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* Since we may have removed our earliest (and/or next earliest) +* task we must recompute them. +*/ +if (!dl_rq->dl_nr_running) { + dl_rq->earliest_dl.curr =0 ; + dl_rq->earliest_dl.next =0 ; + }else { +struct rb_node *leftmost = dl_rq->rb_leftmost; +struct sched_dl_entity *entry; + + entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); + dl_rq->earliest_dl.curr = entry->deadline; + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +#else + +static inline void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} +static inline void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} + +#endif /* CONFIG_SMP */ + +static inline +void inc_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; +u64 deadline = dl_se->deadline; + +WARN_ON (!dl_prio (prio)); + dl_rq->dl_nr_running++; + +inc_dl_deadline (dl_rq, deadline); +inc_dl_migration (dl_se, dl_rq); +} + +static inline +void dec_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; + +WARN_ON (!dl_prio (prio)); +WARN_ON (!dl_rq->dl_nr_running); + dl_rq->dl_nr_running--; + +dec_dl_deadline (dl_rq, dl_se->deadline); +dec_dl_migration (dl_se, dl_rq); +} +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se);@@ -424,7 +696,7 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)rb_link_node (&dl_se->rb_node, parent, link);rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); - dl_rq->dl_nr_running++; +inc_dl_tasks (dl_se, dl_rq); }static void __dequeue_dl_entity (struct sched_dl_entity *dl_se)@@ -444,7 +716,7 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)rb_erase (&dl_se->rb_node, &dl_rq->rb_root);RB_CLEAR_NODE (&dl_se->rb_node); - dl_rq->dl_nr_running--; +dec_dl_tasks (dl_se, dl_rq); } static void@@ -482,12 +754,17 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)return ;enqueue_dl_entity (&p->dl, flags); + +if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); +inc_nr_running (rq); }static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {dequeue_dl_entity (&p->dl); +dequeue_pushable_dl_task (rq, p); }static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags)@@ -525,6 +802,77 @@ static void yield_task_dl(struct rq *rq)update_curr_dl (rq); } +#ifdef CONFIG_SMP + +static int find_later_rq (struct task_struct *task); +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask); + +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +struct task_struct *curr; +struct rq *rq; +int cpu; + + cpu =task_cpu (p); + +if (sd_flag !=SD_BALANCE_WAKE && sd_flag !=SD_BALANCE_FORK ) + goto out; + + rq =cpu_rq (cpu); + +rcu_read_lock (); + curr =ACCESS_ONCE (rq->curr);/* unlocked access */ + +/* +* If we are dealing with a -deadline task, we must +* decide where to wake it up. +* If it has a later deadline and the current task +* on this rq can't move (provided the waking task +* can!) we prefer to send it somewhere else. On the +* other hand, if it has a shorter deadline, we +* try to make it stay here, it might be important. +*/ +if (unlikely (dl_task (curr)) && + (curr->nr_cpus_allowed <2 || + !dl_entity_preempt (&p->dl, &curr->dl)) && + (p->nr_cpus_allowed >1 )) { +int target =find_later_rq (p); + +if (target !=-1 ) + cpu = target; + } +rcu_read_unlock (); + +out: +return cpu; +} + +static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p) +{ +/* +* Current can't be migrated, useless to reschedule, +* let's hope p can move out. +*/ +if (rq->curr->nr_cpus_allowed ==1 || +latest_cpu_find (rq->rd->span, rq->curr,NULL ) ==-1 ) +return ; + +/* +* p is migratable, so let's not schedule it and +* see if it is pushed or pulled somewhere else. +*/ +if (p->nr_cpus_allowed !=1 && +latest_cpu_find (rq->rd->span, p,NULL ) !=-1 ) +return ; + +resched_task (rq->curr); +} + +#endif /* CONFIG_SMP */ +/* * Only called when both the current and waking task are -deadline * tasks. @@ -532,8 +880,20 @@ static void yield_task_dl(struct rq *rq)static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +if (dl_entity_preempt (&p->dl, &rq->curr->dl)) {resched_task (rq->curr); +return ; + } + +#ifdef CONFIG_SMP +/* +* In the unlikely case current and p have the same deadline +* let us try to decide what's the best thing to do... +*/ +if ((s64)(p->dl.deadline - rq->curr->dl.deadline) ==0 && + !need_resched ()) +check_preempt_equal_dl (rq, p); +#endif /* CONFIG_SMP */ }#ifdef CONFIG_SCHED_HRTICK @@ -573,16 +933,29 @@ struct task_struct *pick_next_task_dl(struct rq *rq) p =dl_task_of (dl_se); p->se.exec_start =rq_clock_task (rq); + +/* Running task will never be pushed. */ +if (p) +dequeue_pushable_dl_task (rq, p); +#ifdef CONFIG_SCHED_HRTICK if (hrtick_enabled (rq))start_hrtick_dl (rq, p);#endif + +#ifdef CONFIG_SMP + rq->post_schedule =has_pushable_dl_tasks (rq); +#endif /* CONFIG_SMP */ +return p; }static void put_prev_task_dl (struct rq *rq,struct task_struct *p) {update_curr_dl (rq); + +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); }static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued)@@ -616,16 +989,517 @@ static void set_curr_task_dl(struct rq *rq)struct task_struct *p = rq->curr; p->se.exec_start =rq_clock_task (rq); + +/* You can't push away the running task */ +dequeue_pushable_dl_task (rq, p); +} + +#ifdef CONFIG_SMP + +/* Only try algorithms three times */ +#define DL_MAX_TRIES 3 + +static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_running (rq, p) && + (cpu <0 ||cpumask_test_cpu (cpu, &p->cpus_allowed)) && + (p->nr_cpus_allowed >1 )) +return 1 ; + +return 0 ; +} + +/* Returns the second earliest -deadline task, NULL otherwise */ +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu) +{ +struct rb_node *next_node = rq->dl.rb_leftmost; +struct sched_dl_entity *dl_se; +struct task_struct *p =NULL ; + +next_node: + next_node =rb_next (next_node); +if (next_node) { + dl_se = rb_entry(next_node, struct sched_dl_entity, rb_node); + p =dl_task_of (dl_se); + +if (pick_dl_task (rq, p, cpu)) +return p; + + goto next_node; + } + +return NULL ; +} + +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {Ug, the above is hard to read. What about:
Quoted 9 lines
if (!best) {
if (!dl_rq->dl_nr_running) {
best = 1;
found = cpu;
} elsif (dl_time_before(...)) {
...
}
}This is completely removed in 13/14. I don't like it either, but since we end up removing this mess, do you think we still have to fix this here?
Quoted 4 lines
Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
Well, in 13/14 we introduce a free_cpus mask. A CPU is considered free if it doesn't have any -deadline task running. We can modify that excluding also CPUs running RT tasks, but I have to think a bit if we can do this also from -rt code.
Quoted 3 lines
We should be able to incorporate
cpuprio_find()to be dl aware too. That is, work for both -rt and -dl.
Like checking if a -dl task is running on the cpu chosen for pushing an -rt task, and continue searching in that case.
Quoted 266 lines
Quoted 266 lines
kernel/sched/deadline.c +262
-0+ max_dl = dl_rq->earliest_dl.curr; + found = cpu; + } + }else if (later_mask) +cpumask_clear_cpu (cpu, later_mask); + } + +return found; +} + +static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); + +static int find_later_rq (struct task_struct *task) +{ +struct sched_domain *sd; +struct cpumask *later_mask =__get_cpu_var (local_cpu_mask_dl); +int this_cpu =smp_processor_id (); +int best_cpu, cpu =task_cpu (task); + +/* Make sure the mask is initialized first */ +if (unlikely (!later_mask)) +return -1 ; + +if (task->nr_cpus_allowed ==1 ) +return -1 ; + + best_cpu =latest_cpu_find (task_rq (task)->rd->span, task, later_mask); +if (best_cpu ==-1 ) +return -1 ; + +/* +* If we are here, some target has been found, +* the most suitable of which is cached in best_cpu. +* This is, among the runqueues where the current tasks +* have later deadlines than the task's one, the rq +* with the latest possible one. +* +* Now we check how well this matches with task's +* affinity and system topology. +* +* The last cpu where the task run is our first +* guess, since it is most likely cache-hot there. +*/ +if (cpumask_test_cpu (cpu, later_mask)) +return cpu; +/* +* Check if this_cpu is to be skipped (i.e., it is +* not in the mask) or not. +*/ +if (!cpumask_test_cpu (this_cpu, later_mask)) + this_cpu =-1 ; + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { + +/* +* If possible, preempting this_cpu is +* cheaper than migrating. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + +/* +* Last chance: if best_cpu is valid and is +* in the mask, that becomes our choice. +*/ +if (best_cpu < nr_cpu_ids && +cpumask_test_cpu (best_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* At this point, all our guesses failed, we just return +* 'something', and let the caller sort the things out. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any (later_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* Locks the rq it finds */ +static struct rq *find_lock_later_rq (struct task_struct *task,struct rq *rq) +{ +struct rq *later_rq =NULL ; +int tries; +int cpu; + +for (tries =0 ; tries <DL_MAX_TRIES ; tries++) { + cpu =find_later_rq (task); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + later_rq =cpu_rq (cpu); + +/* Retry if something changed. */ +if (double_lock_balance (rq, later_rq)) { +if (unlikely (task_rq (task) != rq || + !cpumask_test_cpu (later_rq->cpu, + &task->cpus_allowed) || +task_running (rq, task) || !task->on_rq)) { +double_unlock_balance (rq, later_rq); + later_rq =NULL ; +break ; + } + } + +/* +* If the rq we found has no -deadline task, or +* its earliest one has a later deadline than our +* task, the rq is a good one. +*/ +if (!later_rq->dl.dl_nr_running || +dl_time_before (task->dl.deadline, + later_rq->dl.earliest_dl.curr)) +break ; + +/* Otherwise we try again. */ +double_unlock_balance (rq, later_rq); + later_rq =NULL ; + } + +return later_rq; } +static struct task_struct *pick_next_pushable_dl_task (struct rq *rq) +{ +struct task_struct *p; + +if (!has_pushable_dl_tasks (rq)) +return NULL ; + + p =rb_entry (rq->dl.pushable_dl_tasks_leftmost, + struct task_struct, pushable_dl_tasks); + +BUG_ON (rq->cpu !=task_cpu (p)); +BUG_ON (task_current (rq, p)); +BUG_ON (p->nr_cpus_allowed <=1 ); + +BUG_ON (!p->se.on_rq); +BUG_ON (!dl_task (p)); + +return p; +} + +/* +* See if the non running -deadline tasks on this rq +* can be sent to some other CPU where they can preempt +* and start executing. +*/ +static int push_dl_task (struct rq *rq) +{ +struct task_struct *next_task; +struct rq *later_rq; + +if (!rq->dl.overloaded) +return 0 ; + + next_task =pick_next_pushable_dl_task (rq); +if (!next_task) +return 0 ; + +retry: +if (unlikely (next_task == rq->curr)) { +WARN_ON (1 ); +return 0 ; + } + +/* +* If next_task preempts rq->curr, and rq->curr +* can move away, it makes sense to just reschedule +* without going further in pushing next_task. +*/ +if (dl_task (rq->curr) && +dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && + rq->curr->nr_cpus_allowed >1 ) { +resched_task (rq->curr); +return 0 ; + } + +/* We might release rq lock */ +get_task_struct (next_task); + +/* Will lock the rq it'll find */ + later_rq =find_lock_later_rq (next_task, rq); +if (!later_rq) { +struct task_struct *task; + +/* +* We must check all this again, since +* find_lock_later_rq releases rq->lock and it is +* then possible that next_task has migrated. +*/ + task =pick_next_pushable_dl_task (rq); +if (task_cpu (next_task) == rq->cpu && task == next_task) { +/* +* The task is still there. We don't try +* again, some other cpu will pull it when ready. +*/ +dequeue_pushable_dl_task (rq, next_task); + goto out; + } + +if (!task) +/* No more tasks */ + goto out; + +put_task_struct (next_task); + next_task = task; + goto retry; + } + +deactivate_task (rq, next_task,0 ); +set_task_cpu (next_task, later_rq->cpu); +activate_task (later_rq, next_task,0 ); + +resched_task (later_rq->curr); + +double_unlock_balance (rq, later_rq); + +out: +put_task_struct (next_task); + +return 1 ; +} + +static void push_dl_tasks (struct rq *rq) +{ +/* Terminates as it moves a -deadline task */ +while (push_dl_task (rq)) + ; +} + +static int pull_dl_task (struct rq *this_rq) +{ +int this_cpu = this_rq->cpu, ret =0 , cpu; +struct task_struct *p; +struct rq *src_rq; +u64 dmin =LONG_MAX ; + +if (likely (!dl_overloaded (this_rq))) +return 0 ; + +/* +* Match the barrier from dl_set_overloaded; this guarantees that if we +* see overloaded we must also see the dlo_mask bit. +*/ +smp_rmb (); + +for_each_cpu (cpu, this_rq->rd->dlo_mask) { +if (this_cpu == cpu)
Quoted 9 lines
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1326
Quoted 6 lines
kernel/sched/deadline.c +6
-0+continue ; + + src_rq =cpu_rq (cpu); + +/* +* It looks racy, abd it is! However, as in sched_rt.c, abd it is?
Oops!
Thanks,
- Juri
Quoted 332 lines
Quoted 331 lines
kernel/sched/deadline.c +213
-12+* we are fine with this. +*/ +if (this_rq->dl.dl_nr_running && +dl_time_before (this_rq->dl.earliest_dl.curr, + src_rq->dl.earliest_dl.next)) +continue ; + +/* Might drop this_rq->lock */ +double_lock_balance (this_rq, src_rq); + +/* +* If there are no more pullable tasks on the +* rq, we're done with it. +*/ +if (src_rq->dl.dl_nr_running <=1 ) + goto skip; + + p =pick_next_earliest_dl_task (src_rq, this_cpu); + +/* +* We found a task to be pulled if: +* - it preempts our current (if there's one), +* - it will preempt the last one we pulled (if any). +*/ +if (p &&dl_time_before (p->dl.deadline, dmin) && + (!this_rq->dl.dl_nr_running || +dl_time_before (p->dl.deadline, + this_rq->dl.earliest_dl.curr))) { +WARN_ON (p == src_rq->curr); +WARN_ON (!p->se.on_rq); + +/* +* Then we pull iff p has actually an earlier +* deadline than the current task of its runqueue. +*/ +if (dl_time_before (p->dl.deadline, + src_rq->curr->dl.deadline)) + goto skip; + + ret =1 ; + +deactivate_task (src_rq, p,0 ); +set_task_cpu (p, this_cpu); +activate_task (this_rq, p,0 ); + dmin = p->dl.deadline; + +/* Is there any other task even earlier? */ + } +skip: + double_unlock_balance(this_rq, src_rq); + } + +return ret; +} + +static void pre_schedule_dl (struct rq *rq,struct task_struct *prev) +{ +/* Try to pull other tasks here */ +if (dl_task (prev)) +pull_dl_task (rq); +} + +static void post_schedule_dl (struct rq *rq) +{ +push_dl_tasks (rq); +} + +/* +* Since the task is not running and a reschedule is not going to happen +* anytime soon on its runqueue, we try pushing it away now. +*/ +static void task_woken_dl (struct rq *rq,struct task_struct *p) +{ +if (!task_running (rq, p) && + !test_tsk_need_resched (rq->curr) && +has_pushable_dl_tasks (rq) && + p->nr_cpus_allowed >1 && +dl_task (rq->curr) && + (rq->curr->nr_cpus_allowed <2 || +dl_entity_preempt (&rq->curr->dl, &p->dl))) { +push_dl_tasks (rq); + } +} + +static void set_cpus_allowed_dl (struct task_struct *p, +const struct cpumask *new_mask) +{ +struct rq *rq; +int weight; + +BUG_ON (!dl_task (p)); + +/* +* Update only if the task is actually running (i.e., +* it is on the rq AND it is not throttled). +*/ +if (!on_dl_rq (&p->dl)) +return ; + + weight =cpumask_weight (new_mask); + +/* +* Only update if the process changes its state from whether it +* can migrate or not. +*/ +if ((p->nr_cpus_allowed >1 ) == (weight >1 )) +return ; + + rq =task_rq (p); + +/* +* The process used to be able to migrate OR it can now migrate +*/ +if (weight <=1 ) { +if (!task_current (rq, p)) +dequeue_pushable_dl_task (rq, p); +BUG_ON (!rq->dl.dl_nr_migratory); + rq->dl.dl_nr_migratory--; + }else { +if (!task_current (rq, p)) +enqueue_pushable_dl_task (rq, p); + rq->dl.dl_nr_migratory++; + } + +update_dl_migration (&rq->dl); +} + +/* Assumes rq->lock is held */ +static void rq_online_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_set_overload (rq); +} + +/* Assumes rq->lock is held */ +static void rq_offline_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_clear_overload (rq); +} + +void init_sched_dl_class (void ) +{ +unsigned int i; + +for_each_possible_cpu (i) +zalloc_cpumask_var_node (&per_cpu (local_cpu_mask_dl, i), +GFP_KERNEL ,cpu_to_node (i)); +} + +#endif /* CONFIG_SMP */ +static void switched_from_dl (struct rq *rq,struct task_struct *p) { -if (hrtimer_active (&p->dl.dl_timer)) +if (hrtimer_active (&p->dl.dl_timer) && !dl_policy (p->policy))hrtimer_try_to_cancel (&p->dl.dl_timer); + +#ifdef CONFIG_SMP +/* +* Since this might be the only -deadline task on the rq, +* this is the right place to try to pull some other one +* from an overloaded cpu, if any. +*/ +if (!rq->dl.dl_nr_running) +pull_dl_task (rq); +#endif } +/* +* When switching to -deadline, we may overload the rq, then +* we try to push someone off, if possible. +*/ static void switched_to_dl (struct rq *rq,struct task_struct *p) { +int check_resched =1 ; +/* * If p is throttled, don't consider the possibility * of preempting rq->curr, the check will be done right @@ -635,26 +1509,53 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ;if (!p->on_rq || rq->curr != p) { -if (task_has_dl_policy (rq->curr)) +#ifdef CONFIG_SMP +if (rq->dl.overloaded &&push_dl_task (rq) && rq !=task_rq (p)) +/* Only reschedule if pushing failed */ + check_resched =0 ; +#endif /* CONFIG_SMP */ +if (check_resched &&task_has_dl_policy (rq->curr))check_preempt_curr_dl (rq, p,0 ); -else -resched_task (rq->curr); } } +/* +* If the scheduling parameters of a -deadline task changed, +* a push or pull operation might be needed. +*/ static void prio_changed_dl (struct rq *rq,struct task_struct *p,int oldprio) { -switched_to_dl (rq, p); -} - +if (p->on_rq || rq->curr == p) {#ifdef CONFIG_SMP -static int -select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) -{ -return task_cpu (p); +/* +* This might be too much, but unfortunately +* we don't have the old deadline value, and +* we can't argue if the task is increasing +* or lowering its prio, so... +*/ +if (!rq->dl.overloaded) +pull_dl_task (rq); + +/* +* If we now have a earlier deadline task than p, +* then reschedule, provided p is still on this +* runqueue. +*/ +if (dl_time_before (rq->dl.earliest_dl.curr, p->dl.deadline) && + rq->curr == p) +resched_task (p); +#else +/* +* Again, we don't know if p has a earlier +* or later deadline, so let's blindly set a +* (maybe not needed) rescheduling point. +*/ +resched_task (p); +#endif /* CONFIG_SMP */ + }else +switched_to_dl (rq, p); } -#endif const struct sched_class dl_sched_class = { .next = &rt_sched_class,@@ -669,6 +1570,12 @@ const struct sched_class dl_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl, + .set_cpus_allowed = set_cpus_allowed_dl, + .rq_online = rq_online_dl, + .rq_offline = rq_offline_dl, + .pre_schedule = pre_schedule_dl, + .post_schedule = post_schedule_dl, + .task_woken = task_woken_dl, #endif .set_curr_task = set_curr_task_dl,kernel/sched/rt.c +1
-1@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);kernel/sched/sched.h +34
-0@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP @@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ @@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Steven Rostedt
maintainerRe: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
On Thu, 21 Nov 2013 15:13:28 +0100
Juri Lelli <[email protected]> wrote:
;
Quoted 10 lines
Quoted 6 lines
kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 3 lines
kernel/sched/deadline.c +3
-0+ +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */ I'm curious to why you add the '-' to -deadline.
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.
Yeah, keep the -deadline, as that makes it more searchable.
Quoted 36 lines
Quoted 26 lines
Quoted 13 lines
kernel/sched/deadline.c +13
-0+static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {Ug, the above is hard to read. What about:
Quoted 9 lines
if (!best) { if (!dl_rq->dl_nr_running) { best = 1; found = cpu; } elsif (dl_time_before(...)) { ... } }
Quoted 2 lines
This is completely removed in 13/14. I don't like it either, but since we end up removing this mess, do you think we still have to fix this here?
OK, if it gets removed later (I haven't hit patch 13 yet), then it should be fine.
Quoted 10 lines
Quoted 5 lines
Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
Well, in 13/14 we introduce a
free_cpusmask. A CPU is considered free if it doesn't have any -deadline task running. We can modify that excluding also CPUs running RT tasks, but I have to think a bit if we can do this also from -rt code.
Quoted 6 lines
Quoted 3 lines
We should be able to incorporate
cpuprio_find()to be dl aware too. That is, work for both -rt and -dl.Like checking if a -dl task is running on the cpu chosen for pushing an -rt task, and continue searching in that case.
Yep, and also, it could perhaps be used for knowing where to push a -dl task too.
-- Steve
Re: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
In reply to Juri LelliQuoted 44 lines · Juri Lelli
Quoted 33 lines · Steven Rostedt
Quoted 29 lines · On Thu, 7 Nov 2013 14:43:38 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/deadline.c +10
-0@@ -10,6 +10,7 @@* miss some of their deadlines), and won't affect any other task. * * Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Juri Lelli <[email protected]>, * Michael Trimarchi <[email protected]>, * Fabio Checconi <[email protected]> */ @@ -20,6 +21,15 @@ static inline int dl_time_before(u64 a, u64 b)return (s64)(a - b) <0 ; } +/* +* Tells if entity @a should preempt entity @b. +*/ +static inline +int dl_entity_preempt (struct sched_dl_entity *a,struct sched_dl_entity *b) +{ +return dl_time_before (a->deadline, b->deadline); +} +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) { return container_of(dl_se, struct task_struct, dl);@@ -53,8 +63,168 @@ static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)Quoted 9 lines
kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 6 lines
kernel/sched/deadline.c +3
-0void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) { dl_rq->rb_root =RB_ROOT ; + +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */ I'm curious to why you add the '-' to -deadline.
Quoted 3 lines
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.
My guess was that "-deadline" was an abbreviation for tasks that had missed their deadline, just so you know. ;-)
Thanx, Paul
Quoted 1,161 lines
Quoted 516 lines
Quoted 503 lines
kernel/sched/deadline.c +418
-3+ dl_rq->earliest_dl.curr = dl_rq->earliest_dl.next =0 ; + + dl_rq->dl_nr_migratory =0 ; + dl_rq->overloaded =0 ; + dl_rq->pushable_dl_tasks_root =RB_ROOT ; +#endif +} + +#ifdef CONFIG_SMP + +static inline int dl_overloaded (struct rq *rq) +{ +return atomic_read (&rq->rd->dlo_count); +} + +static inline void dl_set_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +cpumask_set_cpu (rq->cpu, rq->rd->dlo_mask); +/* +* Must be visible before the overload count is +* set (as in sched_rt.c). +* +* Matched by the barrier in pull_dl_task(). +*/ +smp_wmb (); +atomic_inc (&rq->rd->dlo_count); +} + +static inline void dl_clear_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +atomic_dec (&rq->rd->dlo_count); +cpumask_clear_cpu (rq->cpu, rq->rd->dlo_mask); +} + +static void update_dl_migration (struct dl_rq *dl_rq) +{ +if (dl_rq->dl_nr_migratory && dl_rq->dl_nr_total >1 ) { +if (!dl_rq->overloaded) { +dl_set_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =1 ; + } + }else if (dl_rq->overloaded) { +dl_clear_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =0 ; + } +} + +static void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total++; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory++; + +update_dl_migration (dl_rq); +} + +static void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total--; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory--; + +update_dl_migration (dl_rq); +} + +/* +* The list of pushable -deadline task is not a plist, like in +* sched_rt.c, it is an rb-tree with tasks ordered by deadline. +*/ +static void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; +struct rb_node **link = &dl_rq->pushable_dl_tasks_root.rb_node; +struct rb_node *parent =NULL ; +struct task_struct *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&p->pushable_dl_tasks)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct task_struct, + pushable_dl_tasks); +if (dl_entity_preempt (&p->dl, &entry->dl)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->pushable_dl_tasks_leftmost = &p->pushable_dl_tasks; + +rb_link_node (&p->pushable_dl_tasks, parent, link); +rb_insert_color (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +} + +static void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; + +if (RB_EMPTY_NODE (&p->pushable_dl_tasks)) +return ; + +if (dl_rq->pushable_dl_tasks_leftmost == &p->pushable_dl_tasks) { +struct rb_node *next_node; + + next_node =rb_next (&p->pushable_dl_tasks); + dl_rq->pushable_dl_tasks_leftmost = next_node; + } + +rb_erase (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +RB_CLEAR_NODE (&p->pushable_dl_tasks); +} + +static inline int has_pushable_dl_tasks (struct rq *rq) +{ +return !RB_EMPTY_ROOT (&rq->dl.pushable_dl_tasks_root); +} + +static int push_dl_task (struct rq *rq); + +#else + +static inline +void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +} + +static inline +void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ } +#endif /* CONFIG_SMP */ +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags);static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,@@ -307,6 +477,14 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)check_preempt_curr_dl (rq, p,0 ); else resched_task(rq->curr); +#ifdef CONFIG_SMP +/* +* Queueing this task back might have overloaded rq, +* check if we need to kick someone away. +*/ +if (has_pushable_dl_tasks (rq)) +push_dl_task (rq); +#endif } unlock:raw_spin_unlock (&rq->lock);@@ -397,6 +575,100 @@ static void update_curr_dl(struct rq *rq) } } +#ifdef CONFIG_SMP + +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu); + +static inline u64 next_deadline (struct rq *rq) +{ +struct task_struct *next =pick_next_earliest_dl_task (rq, rq->cpu); + +if (next &&dl_prio (next->prio)) +return next->dl.deadline; +else +return 0 ; +} + +static void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +if (dl_rq->earliest_dl.curr ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.curr)) { +/* +* If the dl_rq had no -deadline tasks, or if the new task +* has shorter deadline than the current one on dl_rq, we +* know that the previous earliest becomes our next earliest, +* as the new task becomes the earliest itself. +*/ + dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; + dl_rq->earliest_dl.curr = deadline; + }else if (dl_rq->earliest_dl.next ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.next)) { +/* +* On the other hand, if the new -deadline task has a +* a later deadline than the earliest one on dl_rq, but +* it is earlier than the next (if any), we must +* recompute the next-earliest. +*/ + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +static void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* Since we may have removed our earliest (and/or next earliest) +* task we must recompute them. +*/ +if (!dl_rq->dl_nr_running) { + dl_rq->earliest_dl.curr =0 ; + dl_rq->earliest_dl.next =0 ; + }else { +struct rb_node *leftmost = dl_rq->rb_leftmost; +struct sched_dl_entity *entry; + + entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); + dl_rq->earliest_dl.curr = entry->deadline; + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +#else + +static inline void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} +static inline void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} + +#endif /* CONFIG_SMP */ + +static inline +void inc_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; +u64 deadline = dl_se->deadline; + +WARN_ON (!dl_prio (prio)); + dl_rq->dl_nr_running++; + +inc_dl_deadline (dl_rq, deadline); +inc_dl_migration (dl_se, dl_rq); +} + +static inline +void dec_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; + +WARN_ON (!dl_prio (prio)); +WARN_ON (!dl_rq->dl_nr_running); + dl_rq->dl_nr_running--; + +dec_dl_deadline (dl_rq, dl_se->deadline); +dec_dl_migration (dl_se, dl_rq); +} +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se);@@ -424,7 +696,7 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)rb_link_node (&dl_se->rb_node, parent, link);rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); - dl_rq->dl_nr_running++; +inc_dl_tasks (dl_se, dl_rq); }static void __dequeue_dl_entity (struct sched_dl_entity *dl_se)@@ -444,7 +716,7 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)rb_erase (&dl_se->rb_node, &dl_rq->rb_root);RB_CLEAR_NODE (&dl_se->rb_node); - dl_rq->dl_nr_running--; +dec_dl_tasks (dl_se, dl_rq); } static void@@ -482,12 +754,17 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)return ;enqueue_dl_entity (&p->dl, flags); + +if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); +inc_nr_running (rq); }static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {dequeue_dl_entity (&p->dl); +dequeue_pushable_dl_task (rq, p); }static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags)@@ -525,6 +802,77 @@ static void yield_task_dl(struct rq *rq)update_curr_dl (rq); } +#ifdef CONFIG_SMP + +static int find_later_rq (struct task_struct *task); +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask); + +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +struct task_struct *curr; +struct rq *rq; +int cpu; + + cpu =task_cpu (p); + +if (sd_flag !=SD_BALANCE_WAKE && sd_flag !=SD_BALANCE_FORK ) + goto out; + + rq =cpu_rq (cpu); + +rcu_read_lock (); + curr =ACCESS_ONCE (rq->curr);/* unlocked access */ + +/* +* If we are dealing with a -deadline task, we must +* decide where to wake it up. +* If it has a later deadline and the current task +* on this rq can't move (provided the waking task +* can!) we prefer to send it somewhere else. On the +* other hand, if it has a shorter deadline, we +* try to make it stay here, it might be important. +*/ +if (unlikely (dl_task (curr)) && + (curr->nr_cpus_allowed <2 || + !dl_entity_preempt (&p->dl, &curr->dl)) && + (p->nr_cpus_allowed >1 )) { +int target =find_later_rq (p); + +if (target !=-1 ) + cpu = target; + } +rcu_read_unlock (); + +out: +return cpu; +} + +static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p) +{ +/* +* Current can't be migrated, useless to reschedule, +* let's hope p can move out. +*/ +if (rq->curr->nr_cpus_allowed ==1 || +latest_cpu_find (rq->rd->span, rq->curr,NULL ) ==-1 ) +return ; + +/* +* p is migratable, so let's not schedule it and +* see if it is pushed or pulled somewhere else. +*/ +if (p->nr_cpus_allowed !=1 && +latest_cpu_find (rq->rd->span, p,NULL ) !=-1 ) +return ; + +resched_task (rq->curr); +} + +#endif /* CONFIG_SMP */ +/* * Only called when both the current and waking task are -deadline * tasks. @@ -532,8 +880,20 @@ static void yield_task_dl(struct rq *rq)static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +if (dl_entity_preempt (&p->dl, &rq->curr->dl)) {resched_task (rq->curr); +return ; + } + +#ifdef CONFIG_SMP +/* +* In the unlikely case current and p have the same deadline +* let us try to decide what's the best thing to do... +*/ +if ((s64)(p->dl.deadline - rq->curr->dl.deadline) ==0 && + !need_resched ()) +check_preempt_equal_dl (rq, p); +#endif /* CONFIG_SMP */ }#ifdef CONFIG_SCHED_HRTICK @@ -573,16 +933,29 @@ struct task_struct *pick_next_task_dl(struct rq *rq) p =dl_task_of (dl_se); p->se.exec_start =rq_clock_task (rq); + +/* Running task will never be pushed. */ +if (p) +dequeue_pushable_dl_task (rq, p); +#ifdef CONFIG_SCHED_HRTICK if (hrtick_enabled (rq))start_hrtick_dl (rq, p);#endif + +#ifdef CONFIG_SMP + rq->post_schedule =has_pushable_dl_tasks (rq); +#endif /* CONFIG_SMP */ +return p; }static void put_prev_task_dl (struct rq *rq,struct task_struct *p) {update_curr_dl (rq); + +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); }static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued)@@ -616,16 +989,517 @@ static void set_curr_task_dl(struct rq *rq)struct task_struct *p = rq->curr; p->se.exec_start =rq_clock_task (rq); + +/* You can't push away the running task */ +dequeue_pushable_dl_task (rq, p); +} + +#ifdef CONFIG_SMP + +/* Only try algorithms three times */ +#define DL_MAX_TRIES 3 + +static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_running (rq, p) && + (cpu <0 ||cpumask_test_cpu (cpu, &p->cpus_allowed)) && + (p->nr_cpus_allowed >1 )) +return 1 ; + +return 0 ; +} + +/* Returns the second earliest -deadline task, NULL otherwise */ +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu) +{ +struct rb_node *next_node = rq->dl.rb_leftmost; +struct sched_dl_entity *dl_se; +struct task_struct *p =NULL ; + +next_node: + next_node =rb_next (next_node); +if (next_node) { + dl_se = rb_entry(next_node, struct sched_dl_entity, rb_node); + p =dl_task_of (dl_se); + +if (pick_dl_task (rq, p, cpu)) +return p; + + goto next_node; + } + +return NULL ; +} + +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {Ug, the above is hard to read. What about:
Quoted 9 lines
if (!best) { if (!dl_rq->dl_nr_running) { best = 1; found = cpu; } elsif (dl_time_before(...)) { ... } }This is completely removed in 13/14. I don't like it either, but since we end up removing this mess, do you think we still have to fix this here?
Quoted 4 lines
Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
Well, in 13/14 we introduce a
free_cpusmask. A CPU is considered free if it doesn't have any -deadline task running. We can modify that excluding also CPUs running RT tasks, but I have to think a bit if we can do this also from -rt code.Quoted 3 lines
We should be able to incorporate
cpuprio_find()to be dl aware too. That is, work for both -rt and -dl.Like checking if a -dl task is running on the cpu chosen for pushing an -rt task, and continue searching in that case.
Quoted 266 lines
Quoted 266 lines
kernel/sched/deadline.c +262
-0+ max_dl = dl_rq->earliest_dl.curr; + found = cpu; + } + }else if (later_mask) +cpumask_clear_cpu (cpu, later_mask); + } + +return found; +} + +static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); + +static int find_later_rq (struct task_struct *task) +{ +struct sched_domain *sd; +struct cpumask *later_mask =__get_cpu_var (local_cpu_mask_dl); +int this_cpu =smp_processor_id (); +int best_cpu, cpu =task_cpu (task); + +/* Make sure the mask is initialized first */ +if (unlikely (!later_mask)) +return -1 ; + +if (task->nr_cpus_allowed ==1 ) +return -1 ; + + best_cpu =latest_cpu_find (task_rq (task)->rd->span, task, later_mask); +if (best_cpu ==-1 ) +return -1 ; + +/* +* If we are here, some target has been found, +* the most suitable of which is cached in best_cpu. +* This is, among the runqueues where the current tasks +* have later deadlines than the task's one, the rq +* with the latest possible one. +* +* Now we check how well this matches with task's +* affinity and system topology. +* +* The last cpu where the task run is our first +* guess, since it is most likely cache-hot there. +*/ +if (cpumask_test_cpu (cpu, later_mask)) +return cpu; +/* +* Check if this_cpu is to be skipped (i.e., it is +* not in the mask) or not. +*/ +if (!cpumask_test_cpu (this_cpu, later_mask)) + this_cpu =-1 ; + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { + +/* +* If possible, preempting this_cpu is +* cheaper than migrating. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + +/* +* Last chance: if best_cpu is valid and is +* in the mask, that becomes our choice. +*/ +if (best_cpu < nr_cpu_ids && +cpumask_test_cpu (best_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* At this point, all our guesses failed, we just return +* 'something', and let the caller sort the things out. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any (later_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* Locks the rq it finds */ +static struct rq *find_lock_later_rq (struct task_struct *task,struct rq *rq) +{ +struct rq *later_rq =NULL ; +int tries; +int cpu; + +for (tries =0 ; tries <DL_MAX_TRIES ; tries++) { + cpu =find_later_rq (task); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + later_rq =cpu_rq (cpu); + +/* Retry if something changed. */ +if (double_lock_balance (rq, later_rq)) { +if (unlikely (task_rq (task) != rq || + !cpumask_test_cpu (later_rq->cpu, + &task->cpus_allowed) || +task_running (rq, task) || !task->on_rq)) { +double_unlock_balance (rq, later_rq); + later_rq =NULL ; +break ; + } + } + +/* +* If the rq we found has no -deadline task, or +* its earliest one has a later deadline than our +* task, the rq is a good one. +*/ +if (!later_rq->dl.dl_nr_running || +dl_time_before (task->dl.deadline, + later_rq->dl.earliest_dl.curr)) +break ; + +/* Otherwise we try again. */ +double_unlock_balance (rq, later_rq); + later_rq =NULL ; + } + +return later_rq; } +static struct task_struct *pick_next_pushable_dl_task (struct rq *rq) +{ +struct task_struct *p; + +if (!has_pushable_dl_tasks (rq)) +return NULL ; + + p =rb_entry (rq->dl.pushable_dl_tasks_leftmost, + struct task_struct, pushable_dl_tasks); + +BUG_ON (rq->cpu !=task_cpu (p)); +BUG_ON (task_current (rq, p)); +BUG_ON (p->nr_cpus_allowed <=1 ); + +BUG_ON (!p->se.on_rq); +BUG_ON (!dl_task (p)); + +return p; +} + +/* +* See if the non running -deadline tasks on this rq +* can be sent to some other CPU where they can preempt +* and start executing. +*/ +static int push_dl_task (struct rq *rq) +{ +struct task_struct *next_task; +struct rq *later_rq; + +if (!rq->dl.overloaded) +return 0 ; + + next_task =pick_next_pushable_dl_task (rq); +if (!next_task) +return 0 ; + +retry: +if (unlikely (next_task == rq->curr)) { +WARN_ON (1 ); +return 0 ; + } + +/* +* If next_task preempts rq->curr, and rq->curr +* can move away, it makes sense to just reschedule +* without going further in pushing next_task. +*/ +if (dl_task (rq->curr) && +dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && + rq->curr->nr_cpus_allowed >1 ) { +resched_task (rq->curr); +return 0 ; + } + +/* We might release rq lock */ +get_task_struct (next_task); + +/* Will lock the rq it'll find */ + later_rq =find_lock_later_rq (next_task, rq); +if (!later_rq) { +struct task_struct *task; + +/* +* We must check all this again, since +* find_lock_later_rq releases rq->lock and it is +* then possible that next_task has migrated. +*/ + task =pick_next_pushable_dl_task (rq); +if (task_cpu (next_task) == rq->cpu && task == next_task) { +/* +* The task is still there. We don't try +* again, some other cpu will pull it when ready. +*/ +dequeue_pushable_dl_task (rq, next_task); + goto out; + } + +if (!task) +/* No more tasks */ + goto out; + +put_task_struct (next_task); + next_task = task; + goto retry; + } + +deactivate_task (rq, next_task,0 ); +set_task_cpu (next_task, later_rq->cpu); +activate_task (later_rq, next_task,0 ); + +resched_task (later_rq->curr); + +double_unlock_balance (rq, later_rq); + +out: +put_task_struct (next_task); + +return 1 ; +} + +static void push_dl_tasks (struct rq *rq) +{ +/* Terminates as it moves a -deadline task */ +while (push_dl_task (rq)) + ; +} + +static int pull_dl_task (struct rq *this_rq) +{ +int this_cpu = this_rq->cpu, ret =0 , cpu; +struct task_struct *p; +struct rq *src_rq; +u64 dmin =LONG_MAX ; + +if (likely (!dl_overloaded (this_rq))) +return 0 ; + +/* +* Match the barrier from dl_set_overloaded; this guarantees that if we +* see overloaded we must also see the dlo_mask bit. +*/ +smp_rmb (); + +for_each_cpu (cpu, this_rq->rd->dlo_mask) { +if (this_cpu == cpu)Quoted 9 lines
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1326
Quoted 6 lines
kernel/sched/deadline.c +6
-0+continue ; + + src_rq =cpu_rq (cpu); + +/* +* It looks racy, abd it is! However, as in sched_rt.c, abd it is?
Oops!
Thanks,
- Juri
Quoted 332 lines
Quoted 331 lines
kernel/sched/deadline.c +213
-12+* we are fine with this. +*/ +if (this_rq->dl.dl_nr_running && +dl_time_before (this_rq->dl.earliest_dl.curr, + src_rq->dl.earliest_dl.next)) +continue ; + +/* Might drop this_rq->lock */ +double_lock_balance (this_rq, src_rq); + +/* +* If there are no more pullable tasks on the +* rq, we're done with it. +*/ +if (src_rq->dl.dl_nr_running <=1 ) + goto skip; + + p =pick_next_earliest_dl_task (src_rq, this_cpu); + +/* +* We found a task to be pulled if: +* - it preempts our current (if there's one), +* - it will preempt the last one we pulled (if any). +*/ +if (p &&dl_time_before (p->dl.deadline, dmin) && + (!this_rq->dl.dl_nr_running || +dl_time_before (p->dl.deadline, + this_rq->dl.earliest_dl.curr))) { +WARN_ON (p == src_rq->curr); +WARN_ON (!p->se.on_rq); + +/* +* Then we pull iff p has actually an earlier +* deadline than the current task of its runqueue. +*/ +if (dl_time_before (p->dl.deadline, + src_rq->curr->dl.deadline)) + goto skip; + + ret =1 ; + +deactivate_task (src_rq, p,0 ); +set_task_cpu (p, this_cpu); +activate_task (this_rq, p,0 ); + dmin = p->dl.deadline; + +/* Is there any other task even earlier? */ + } +skip: + double_unlock_balance(this_rq, src_rq); + } + +return ret; +} + +static void pre_schedule_dl (struct rq *rq,struct task_struct *prev) +{ +/* Try to pull other tasks here */ +if (dl_task (prev)) +pull_dl_task (rq); +} + +static void post_schedule_dl (struct rq *rq) +{ +push_dl_tasks (rq); +} + +/* +* Since the task is not running and a reschedule is not going to happen +* anytime soon on its runqueue, we try pushing it away now. +*/ +static void task_woken_dl (struct rq *rq,struct task_struct *p) +{ +if (!task_running (rq, p) && + !test_tsk_need_resched (rq->curr) && +has_pushable_dl_tasks (rq) && + p->nr_cpus_allowed >1 && +dl_task (rq->curr) && + (rq->curr->nr_cpus_allowed <2 || +dl_entity_preempt (&rq->curr->dl, &p->dl))) { +push_dl_tasks (rq); + } +} + +static void set_cpus_allowed_dl (struct task_struct *p, +const struct cpumask *new_mask) +{ +struct rq *rq; +int weight; + +BUG_ON (!dl_task (p)); + +/* +* Update only if the task is actually running (i.e., +* it is on the rq AND it is not throttled). +*/ +if (!on_dl_rq (&p->dl)) +return ; + + weight =cpumask_weight (new_mask); + +/* +* Only update if the process changes its state from whether it +* can migrate or not. +*/ +if ((p->nr_cpus_allowed >1 ) == (weight >1 )) +return ; + + rq =task_rq (p); + +/* +* The process used to be able to migrate OR it can now migrate +*/ +if (weight <=1 ) { +if (!task_current (rq, p)) +dequeue_pushable_dl_task (rq, p); +BUG_ON (!rq->dl.dl_nr_migratory); + rq->dl.dl_nr_migratory--; + }else { +if (!task_current (rq, p)) +enqueue_pushable_dl_task (rq, p); + rq->dl.dl_nr_migratory++; + } + +update_dl_migration (&rq->dl); +} + +/* Assumes rq->lock is held */ +static void rq_online_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_set_overload (rq); +} + +/* Assumes rq->lock is held */ +static void rq_offline_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_clear_overload (rq); +} + +void init_sched_dl_class (void ) +{ +unsigned int i; + +for_each_possible_cpu (i) +zalloc_cpumask_var_node (&per_cpu (local_cpu_mask_dl, i), +GFP_KERNEL ,cpu_to_node (i)); +} + +#endif /* CONFIG_SMP */ +static void switched_from_dl (struct rq *rq,struct task_struct *p) { -if (hrtimer_active (&p->dl.dl_timer)) +if (hrtimer_active (&p->dl.dl_timer) && !dl_policy (p->policy))hrtimer_try_to_cancel (&p->dl.dl_timer); + +#ifdef CONFIG_SMP +/* +* Since this might be the only -deadline task on the rq, +* this is the right place to try to pull some other one +* from an overloaded cpu, if any. +*/ +if (!rq->dl.dl_nr_running) +pull_dl_task (rq); +#endif } +/* +* When switching to -deadline, we may overload the rq, then +* we try to push someone off, if possible. +*/ static void switched_to_dl (struct rq *rq,struct task_struct *p) { +int check_resched =1 ; +/* * If p is throttled, don't consider the possibility * of preempting rq->curr, the check will be done right @@ -635,26 +1509,53 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ;if (!p->on_rq || rq->curr != p) { -if (task_has_dl_policy (rq->curr)) +#ifdef CONFIG_SMP +if (rq->dl.overloaded &&push_dl_task (rq) && rq !=task_rq (p)) +/* Only reschedule if pushing failed */ + check_resched =0 ; +#endif /* CONFIG_SMP */ +if (check_resched &&task_has_dl_policy (rq->curr))check_preempt_curr_dl (rq, p,0 ); -else -resched_task (rq->curr); } } +/* +* If the scheduling parameters of a -deadline task changed, +* a push or pull operation might be needed. +*/ static void prio_changed_dl (struct rq *rq,struct task_struct *p,int oldprio) { -switched_to_dl (rq, p); -} - +if (p->on_rq || rq->curr == p) {#ifdef CONFIG_SMP -static int -select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) -{ -return task_cpu (p); +/* +* This might be too much, but unfortunately +* we don't have the old deadline value, and +* we can't argue if the task is increasing +* or lowering its prio, so... +*/ +if (!rq->dl.overloaded) +pull_dl_task (rq); + +/* +* If we now have a earlier deadline task than p, +* then reschedule, provided p is still on this +* runqueue. +*/ +if (dl_time_before (rq->dl.earliest_dl.curr, p->dl.deadline) && + rq->curr == p) +resched_task (p); +#else +/* +* Again, we don't know if p has a earlier +* or later deadline, so let's blindly set a +* (maybe not needed) rescheduling point. +*/ +resched_task (p); +#endif /* CONFIG_SMP */ + }else +switched_to_dl (rq, p); } -#endif const struct sched_class dl_sched_class = { .next = &rt_sched_class,@@ -669,6 +1570,12 @@ const struct sched_class dl_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl, + .set_cpus_allowed = set_cpus_allowed_dl, + .rq_online = rq_online_dl, + .rq_offline = rq_offline_dl, + .pre_schedule = pre_schedule_dl, + .post_schedule = post_schedule_dl, + .task_woken = task_woken_dl, #endif .set_curr_task = set_curr_task_dl,kernel/sched/rt.c +1
-1@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);kernel/sched/sched.h +34
-0@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP @@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ @@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Re: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Juri Lelli
Quoted 42 lines · Steven Rostedt
Quoted 29 lines · On Thu, 7 Nov 2013 14:43:38 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/deadline.c +10
-0@@ -10,6 +10,7 @@* miss some of their deadlines), and won't affect any other task. * * Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Juri Lelli <[email protected]>, * Michael Trimarchi <[email protected]>, * Fabio Checconi <[email protected]> */ @@ -20,6 +21,15 @@ static inline int dl_time_before(u64 a, u64 b)return (s64)(a - b) <0 ; } +/* +* Tells if entity @a should preempt entity @b. +*/ +static inline +int dl_entity_preempt (struct sched_dl_entity *a,struct sched_dl_entity *b) +{ +return dl_time_before (a->deadline, b->deadline); +} +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) { return container_of(dl_se, struct task_struct, dl);kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 6 lines
kernel/sched/deadline.c +3
-0@@ -53,8 +63,168 @@ static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) { dl_rq->rb_root =RB_ROOT ; + +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */ I'm curious to why you add the '-' to -deadline.
Quoted 7 lines
Quoted 3 lines
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.My guess was that "-deadline" was an abbreviation for tasks that had missed their deadline, just so you know. ;-)
Argh! ;)
Actually, thinking about it twice: if we call "-rt tasks" things that are governed by code that resides in sched/rt.c, we could agree on "-deadline tasks" for things that pass through sched/deadline.c.
Thanks,
- Juri
Quoted 1,162 lines
Quoted 1,161 lines
Quoted 516 lines
Quoted 503 lines
kernel/sched/deadline.c +418
-3+ dl_rq->earliest_dl.curr = dl_rq->earliest_dl.next =0 ; + + dl_rq->dl_nr_migratory =0 ; + dl_rq->overloaded =0 ; + dl_rq->pushable_dl_tasks_root =RB_ROOT ; +#endif +} + +#ifdef CONFIG_SMP + +static inline int dl_overloaded (struct rq *rq) +{ +return atomic_read (&rq->rd->dlo_count); +} + +static inline void dl_set_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +cpumask_set_cpu (rq->cpu, rq->rd->dlo_mask); +/* +* Must be visible before the overload count is +* set (as in sched_rt.c). +* +* Matched by the barrier in pull_dl_task(). +*/ +smp_wmb (); +atomic_inc (&rq->rd->dlo_count); +} + +static inline void dl_clear_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +atomic_dec (&rq->rd->dlo_count); +cpumask_clear_cpu (rq->cpu, rq->rd->dlo_mask); +} + +static void update_dl_migration (struct dl_rq *dl_rq) +{ +if (dl_rq->dl_nr_migratory && dl_rq->dl_nr_total >1 ) { +if (!dl_rq->overloaded) { +dl_set_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =1 ; + } + }else if (dl_rq->overloaded) { +dl_clear_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =0 ; + } +} + +static void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total++; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory++; + +update_dl_migration (dl_rq); +} + +static void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total--; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory--; + +update_dl_migration (dl_rq); +} + +/* +* The list of pushable -deadline task is not a plist, like in +* sched_rt.c, it is an rb-tree with tasks ordered by deadline. +*/ +static void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; +struct rb_node **link = &dl_rq->pushable_dl_tasks_root.rb_node; +struct rb_node *parent =NULL ; +struct task_struct *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&p->pushable_dl_tasks)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct task_struct, + pushable_dl_tasks); +if (dl_entity_preempt (&p->dl, &entry->dl)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->pushable_dl_tasks_leftmost = &p->pushable_dl_tasks; + +rb_link_node (&p->pushable_dl_tasks, parent, link); +rb_insert_color (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +} + +static void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; + +if (RB_EMPTY_NODE (&p->pushable_dl_tasks)) +return ; + +if (dl_rq->pushable_dl_tasks_leftmost == &p->pushable_dl_tasks) { +struct rb_node *next_node; + + next_node =rb_next (&p->pushable_dl_tasks); + dl_rq->pushable_dl_tasks_leftmost = next_node; + } + +rb_erase (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +RB_CLEAR_NODE (&p->pushable_dl_tasks); +} + +static inline int has_pushable_dl_tasks (struct rq *rq) +{ +return !RB_EMPTY_ROOT (&rq->dl.pushable_dl_tasks_root); +} + +static int push_dl_task (struct rq *rq); + +#else + +static inline +void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +} + +static inline +void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ } +#endif /* CONFIG_SMP */ +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags);static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,@@ -307,6 +477,14 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)check_preempt_curr_dl (rq, p,0 ); else resched_task(rq->curr); +#ifdef CONFIG_SMP +/* +* Queueing this task back might have overloaded rq, +* check if we need to kick someone away. +*/ +if (has_pushable_dl_tasks (rq)) +push_dl_task (rq); +#endif } unlock:raw_spin_unlock (&rq->lock);@@ -397,6 +575,100 @@ static void update_curr_dl(struct rq *rq) } } +#ifdef CONFIG_SMP + +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu); + +static inline u64 next_deadline (struct rq *rq) +{ +struct task_struct *next =pick_next_earliest_dl_task (rq, rq->cpu); + +if (next &&dl_prio (next->prio)) +return next->dl.deadline; +else +return 0 ; +} + +static void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +if (dl_rq->earliest_dl.curr ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.curr)) { +/* +* If the dl_rq had no -deadline tasks, or if the new task +* has shorter deadline than the current one on dl_rq, we +* know that the previous earliest becomes our next earliest, +* as the new task becomes the earliest itself. +*/ + dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; + dl_rq->earliest_dl.curr = deadline; + }else if (dl_rq->earliest_dl.next ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.next)) { +/* +* On the other hand, if the new -deadline task has a +* a later deadline than the earliest one on dl_rq, but +* it is earlier than the next (if any), we must +* recompute the next-earliest. +*/ + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +static void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* Since we may have removed our earliest (and/or next earliest) +* task we must recompute them. +*/ +if (!dl_rq->dl_nr_running) { + dl_rq->earliest_dl.curr =0 ; + dl_rq->earliest_dl.next =0 ; + }else { +struct rb_node *leftmost = dl_rq->rb_leftmost; +struct sched_dl_entity *entry; + + entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); + dl_rq->earliest_dl.curr = entry->deadline; + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +#else + +static inline void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} +static inline void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} + +#endif /* CONFIG_SMP */ + +static inline +void inc_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; +u64 deadline = dl_se->deadline; + +WARN_ON (!dl_prio (prio)); + dl_rq->dl_nr_running++; + +inc_dl_deadline (dl_rq, deadline); +inc_dl_migration (dl_se, dl_rq); +} + +static inline +void dec_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; + +WARN_ON (!dl_prio (prio)); +WARN_ON (!dl_rq->dl_nr_running); + dl_rq->dl_nr_running--; + +dec_dl_deadline (dl_rq, dl_se->deadline); +dec_dl_migration (dl_se, dl_rq); +} +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se);@@ -424,7 +696,7 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)rb_link_node (&dl_se->rb_node, parent, link);rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); - dl_rq->dl_nr_running++; +inc_dl_tasks (dl_se, dl_rq); }static void __dequeue_dl_entity (struct sched_dl_entity *dl_se)@@ -444,7 +716,7 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)rb_erase (&dl_se->rb_node, &dl_rq->rb_root);RB_CLEAR_NODE (&dl_se->rb_node); - dl_rq->dl_nr_running--; +dec_dl_tasks (dl_se, dl_rq); } static void@@ -482,12 +754,17 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)return ;enqueue_dl_entity (&p->dl, flags); + +if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); +inc_nr_running (rq); }static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {dequeue_dl_entity (&p->dl); +dequeue_pushable_dl_task (rq, p); }static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags)@@ -525,6 +802,77 @@ static void yield_task_dl(struct rq *rq)update_curr_dl (rq); } +#ifdef CONFIG_SMP + +static int find_later_rq (struct task_struct *task); +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask); + +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +struct task_struct *curr; +struct rq *rq; +int cpu; + + cpu =task_cpu (p); + +if (sd_flag !=SD_BALANCE_WAKE && sd_flag !=SD_BALANCE_FORK ) + goto out; + + rq =cpu_rq (cpu); + +rcu_read_lock (); + curr =ACCESS_ONCE (rq->curr);/* unlocked access */ + +/* +* If we are dealing with a -deadline task, we must +* decide where to wake it up. +* If it has a later deadline and the current task +* on this rq can't move (provided the waking task +* can!) we prefer to send it somewhere else. On the +* other hand, if it has a shorter deadline, we +* try to make it stay here, it might be important. +*/ +if (unlikely (dl_task (curr)) && + (curr->nr_cpus_allowed <2 || + !dl_entity_preempt (&p->dl, &curr->dl)) && + (p->nr_cpus_allowed >1 )) { +int target =find_later_rq (p); + +if (target !=-1 ) + cpu = target; + } +rcu_read_unlock (); + +out: +return cpu; +} + +static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p) +{ +/* +* Current can't be migrated, useless to reschedule, +* let's hope p can move out. +*/ +if (rq->curr->nr_cpus_allowed ==1 || +latest_cpu_find (rq->rd->span, rq->curr,NULL ) ==-1 ) +return ; + +/* +* p is migratable, so let's not schedule it and +* see if it is pushed or pulled somewhere else. +*/ +if (p->nr_cpus_allowed !=1 && +latest_cpu_find (rq->rd->span, p,NULL ) !=-1 ) +return ; + +resched_task (rq->curr); +} + +#endif /* CONFIG_SMP */ +/* * Only called when both the current and waking task are -deadline * tasks. @@ -532,8 +880,20 @@ static void yield_task_dl(struct rq *rq)static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +if (dl_entity_preempt (&p->dl, &rq->curr->dl)) {resched_task (rq->curr); +return ; + } + +#ifdef CONFIG_SMP +/* +* In the unlikely case current and p have the same deadline +* let us try to decide what's the best thing to do... +*/ +if ((s64)(p->dl.deadline - rq->curr->dl.deadline) ==0 && + !need_resched ()) +check_preempt_equal_dl (rq, p); +#endif /* CONFIG_SMP */ }#ifdef CONFIG_SCHED_HRTICK @@ -573,16 +933,29 @@ struct task_struct *pick_next_task_dl(struct rq *rq) p =dl_task_of (dl_se); p->se.exec_start =rq_clock_task (rq); + +/* Running task will never be pushed. */ +if (p) +dequeue_pushable_dl_task (rq, p); +#ifdef CONFIG_SCHED_HRTICK if (hrtick_enabled (rq))start_hrtick_dl (rq, p);#endif + +#ifdef CONFIG_SMP + rq->post_schedule =has_pushable_dl_tasks (rq); +#endif /* CONFIG_SMP */ +return p; }static void put_prev_task_dl (struct rq *rq,struct task_struct *p) {update_curr_dl (rq); + +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); }static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued)@@ -616,16 +989,517 @@ static void set_curr_task_dl(struct rq *rq)struct task_struct *p = rq->curr; p->se.exec_start =rq_clock_task (rq); + +/* You can't push away the running task */ +dequeue_pushable_dl_task (rq, p); +} + +#ifdef CONFIG_SMP + +/* Only try algorithms three times */ +#define DL_MAX_TRIES 3 + +static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_running (rq, p) && + (cpu <0 ||cpumask_test_cpu (cpu, &p->cpus_allowed)) && + (p->nr_cpus_allowed >1 )) +return 1 ; + +return 0 ; +} + +/* Returns the second earliest -deadline task, NULL otherwise */ +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu) +{ +struct rb_node *next_node = rq->dl.rb_leftmost; +struct sched_dl_entity *dl_se; +struct task_struct *p =NULL ; + +next_node: + next_node =rb_next (next_node); +if (next_node) { + dl_se = rb_entry(next_node, struct sched_dl_entity, rb_node); + p =dl_task_of (dl_se); + +if (pick_dl_task (rq, p, cpu)) +return p; + + goto next_node; + } + +return NULL ; +} + +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {Ug, the above is hard to read. What about:
Quoted 9 lines
if (!best) { if (!dl_rq->dl_nr_running) { best = 1; found = cpu; } elsif (dl_time_before(...)) { ... } }This is completely removed in 13/14. I don't like it either, but since we end up removing this mess, do you think we still have to fix this here?
Quoted 4 lines
Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
Well, in 13/14 we introduce a
free_cpusmask. A CPU is considered free if it doesn't have any -deadline task running. We can modify that excluding also CPUs running RT tasks, but I have to think a bit if we can do this also from -rt code.Quoted 3 lines
We should be able to incorporate
cpuprio_find()to be dl aware too. That is, work for both -rt and -dl.Like checking if a -dl task is running on the cpu chosen for pushing an -rt task, and continue searching in that case.
Quoted 266 lines
Quoted 266 lines
kernel/sched/deadline.c +262
-0+ max_dl = dl_rq->earliest_dl.curr; + found = cpu; + } + }else if (later_mask) +cpumask_clear_cpu (cpu, later_mask); + } + +return found; +} + +static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); + +static int find_later_rq (struct task_struct *task) +{ +struct sched_domain *sd; +struct cpumask *later_mask =__get_cpu_var (local_cpu_mask_dl); +int this_cpu =smp_processor_id (); +int best_cpu, cpu =task_cpu (task); + +/* Make sure the mask is initialized first */ +if (unlikely (!later_mask)) +return -1 ; + +if (task->nr_cpus_allowed ==1 ) +return -1 ; + + best_cpu =latest_cpu_find (task_rq (task)->rd->span, task, later_mask); +if (best_cpu ==-1 ) +return -1 ; + +/* +* If we are here, some target has been found, +* the most suitable of which is cached in best_cpu. +* This is, among the runqueues where the current tasks +* have later deadlines than the task's one, the rq +* with the latest possible one. +* +* Now we check how well this matches with task's +* affinity and system topology. +* +* The last cpu where the task run is our first +* guess, since it is most likely cache-hot there. +*/ +if (cpumask_test_cpu (cpu, later_mask)) +return cpu; +/* +* Check if this_cpu is to be skipped (i.e., it is +* not in the mask) or not. +*/ +if (!cpumask_test_cpu (this_cpu, later_mask)) + this_cpu =-1 ; + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { + +/* +* If possible, preempting this_cpu is +* cheaper than migrating. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + +/* +* Last chance: if best_cpu is valid and is +* in the mask, that becomes our choice. +*/ +if (best_cpu < nr_cpu_ids && +cpumask_test_cpu (best_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* At this point, all our guesses failed, we just return +* 'something', and let the caller sort the things out. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any (later_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* Locks the rq it finds */ +static struct rq *find_lock_later_rq (struct task_struct *task,struct rq *rq) +{ +struct rq *later_rq =NULL ; +int tries; +int cpu; + +for (tries =0 ; tries <DL_MAX_TRIES ; tries++) { + cpu =find_later_rq (task); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + later_rq =cpu_rq (cpu); + +/* Retry if something changed. */ +if (double_lock_balance (rq, later_rq)) { +if (unlikely (task_rq (task) != rq || + !cpumask_test_cpu (later_rq->cpu, + &task->cpus_allowed) || +task_running (rq, task) || !task->on_rq)) { +double_unlock_balance (rq, later_rq); + later_rq =NULL ; +break ; + } + } + +/* +* If the rq we found has no -deadline task, or +* its earliest one has a later deadline than our +* task, the rq is a good one. +*/ +if (!later_rq->dl.dl_nr_running || +dl_time_before (task->dl.deadline, + later_rq->dl.earliest_dl.curr)) +break ; + +/* Otherwise we try again. */ +double_unlock_balance (rq, later_rq); + later_rq =NULL ; + } + +return later_rq; } +static struct task_struct *pick_next_pushable_dl_task (struct rq *rq) +{ +struct task_struct *p; + +if (!has_pushable_dl_tasks (rq)) +return NULL ; + + p =rb_entry (rq->dl.pushable_dl_tasks_leftmost, + struct task_struct, pushable_dl_tasks); + +BUG_ON (rq->cpu !=task_cpu (p)); +BUG_ON (task_current (rq, p)); +BUG_ON (p->nr_cpus_allowed <=1 ); + +BUG_ON (!p->se.on_rq); +BUG_ON (!dl_task (p)); + +return p; +} + +/* +* See if the non running -deadline tasks on this rq +* can be sent to some other CPU where they can preempt +* and start executing. +*/ +static int push_dl_task (struct rq *rq) +{ +struct task_struct *next_task; +struct rq *later_rq; + +if (!rq->dl.overloaded) +return 0 ; + + next_task =pick_next_pushable_dl_task (rq); +if (!next_task) +return 0 ; + +retry: +if (unlikely (next_task == rq->curr)) { +WARN_ON (1 ); +return 0 ; + } + +/* +* If next_task preempts rq->curr, and rq->curr +* can move away, it makes sense to just reschedule +* without going further in pushing next_task. +*/ +if (dl_task (rq->curr) && +dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && + rq->curr->nr_cpus_allowed >1 ) { +resched_task (rq->curr); +return 0 ; + } + +/* We might release rq lock */ +get_task_struct (next_task); + +/* Will lock the rq it'll find */ + later_rq =find_lock_later_rq (next_task, rq); +if (!later_rq) { +struct task_struct *task; + +/* +* We must check all this again, since +* find_lock_later_rq releases rq->lock and it is +* then possible that next_task has migrated. +*/ + task =pick_next_pushable_dl_task (rq); +if (task_cpu (next_task) == rq->cpu && task == next_task) { +/* +* The task is still there. We don't try +* again, some other cpu will pull it when ready. +*/ +dequeue_pushable_dl_task (rq, next_task); + goto out; + } + +if (!task) +/* No more tasks */ + goto out; + +put_task_struct (next_task); + next_task = task; + goto retry; + } + +deactivate_task (rq, next_task,0 ); +set_task_cpu (next_task, later_rq->cpu); +activate_task (later_rq, next_task,0 ); + +resched_task (later_rq->curr); + +double_unlock_balance (rq, later_rq); + +out: +put_task_struct (next_task); + +return 1 ; +} + +static void push_dl_tasks (struct rq *rq) +{ +/* Terminates as it moves a -deadline task */ +while (push_dl_task (rq)) + ; +} + +static int pull_dl_task (struct rq *this_rq) +{ +int this_cpu = this_rq->cpu, ret =0 , cpu; +struct task_struct *p; +struct rq *src_rq; +u64 dmin =LONG_MAX ; + +if (likely (!dl_overloaded (this_rq))) +return 0 ; + +/* +* Match the barrier from dl_set_overloaded; this guarantees that if we +* see overloaded we must also see the dlo_mask bit. +*/ +smp_rmb (); + +for_each_cpu (cpu, this_rq->rd->dlo_mask) { +if (this_cpu == cpu)Quoted 9 lines
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1326
Quoted 6 lines
kernel/sched/deadline.c +6
-0+continue ; + + src_rq =cpu_rq (cpu); + +/* +* It looks racy, abd it is! However, as in sched_rt.c, abd it is?
Oops!
Thanks,
- Juri
Quoted 332 lines
Quoted 331 lines
kernel/sched/deadline.c +213
-12+* we are fine with this. +*/ +if (this_rq->dl.dl_nr_running && +dl_time_before (this_rq->dl.earliest_dl.curr, + src_rq->dl.earliest_dl.next)) +continue ; + +/* Might drop this_rq->lock */ +double_lock_balance (this_rq, src_rq); + +/* +* If there are no more pullable tasks on the +* rq, we're done with it. +*/ +if (src_rq->dl.dl_nr_running <=1 ) + goto skip; + + p =pick_next_earliest_dl_task (src_rq, this_cpu); + +/* +* We found a task to be pulled if: +* - it preempts our current (if there's one), +* - it will preempt the last one we pulled (if any). +*/ +if (p &&dl_time_before (p->dl.deadline, dmin) && + (!this_rq->dl.dl_nr_running || +dl_time_before (p->dl.deadline, + this_rq->dl.earliest_dl.curr))) { +WARN_ON (p == src_rq->curr); +WARN_ON (!p->se.on_rq); + +/* +* Then we pull iff p has actually an earlier +* deadline than the current task of its runqueue. +*/ +if (dl_time_before (p->dl.deadline, + src_rq->curr->dl.deadline)) + goto skip; + + ret =1 ; + +deactivate_task (src_rq, p,0 ); +set_task_cpu (p, this_cpu); +activate_task (this_rq, p,0 ); + dmin = p->dl.deadline; + +/* Is there any other task even earlier? */ + } +skip: + double_unlock_balance(this_rq, src_rq); + } + +return ret; +} + +static void pre_schedule_dl (struct rq *rq,struct task_struct *prev) +{ +/* Try to pull other tasks here */ +if (dl_task (prev)) +pull_dl_task (rq); +} + +static void post_schedule_dl (struct rq *rq) +{ +push_dl_tasks (rq); +} + +/* +* Since the task is not running and a reschedule is not going to happen +* anytime soon on its runqueue, we try pushing it away now. +*/ +static void task_woken_dl (struct rq *rq,struct task_struct *p) +{ +if (!task_running (rq, p) && + !test_tsk_need_resched (rq->curr) && +has_pushable_dl_tasks (rq) && + p->nr_cpus_allowed >1 && +dl_task (rq->curr) && + (rq->curr->nr_cpus_allowed <2 || +dl_entity_preempt (&rq->curr->dl, &p->dl))) { +push_dl_tasks (rq); + } +} + +static void set_cpus_allowed_dl (struct task_struct *p, +const struct cpumask *new_mask) +{ +struct rq *rq; +int weight; + +BUG_ON (!dl_task (p)); + +/* +* Update only if the task is actually running (i.e., +* it is on the rq AND it is not throttled). +*/ +if (!on_dl_rq (&p->dl)) +return ; + + weight =cpumask_weight (new_mask); + +/* +* Only update if the process changes its state from whether it +* can migrate or not. +*/ +if ((p->nr_cpus_allowed >1 ) == (weight >1 )) +return ; + + rq =task_rq (p); + +/* +* The process used to be able to migrate OR it can now migrate +*/ +if (weight <=1 ) { +if (!task_current (rq, p)) +dequeue_pushable_dl_task (rq, p); +BUG_ON (!rq->dl.dl_nr_migratory); + rq->dl.dl_nr_migratory--; + }else { +if (!task_current (rq, p)) +enqueue_pushable_dl_task (rq, p); + rq->dl.dl_nr_migratory++; + } + +update_dl_migration (&rq->dl); +} + +/* Assumes rq->lock is held */ +static void rq_online_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_set_overload (rq); +} + +/* Assumes rq->lock is held */ +static void rq_offline_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_clear_overload (rq); +} + +void init_sched_dl_class (void ) +{ +unsigned int i; + +for_each_possible_cpu (i) +zalloc_cpumask_var_node (&per_cpu (local_cpu_mask_dl, i), +GFP_KERNEL ,cpu_to_node (i)); +} + +#endif /* CONFIG_SMP */ +static void switched_from_dl (struct rq *rq,struct task_struct *p) { -if (hrtimer_active (&p->dl.dl_timer)) +if (hrtimer_active (&p->dl.dl_timer) && !dl_policy (p->policy))hrtimer_try_to_cancel (&p->dl.dl_timer); + +#ifdef CONFIG_SMP +/* +* Since this might be the only -deadline task on the rq, +* this is the right place to try to pull some other one +* from an overloaded cpu, if any. +*/ +if (!rq->dl.dl_nr_running) +pull_dl_task (rq); +#endif } +/* +* When switching to -deadline, we may overload the rq, then +* we try to push someone off, if possible. +*/ static void switched_to_dl (struct rq *rq,struct task_struct *p) { +int check_resched =1 ; +/* * If p is throttled, don't consider the possibility * of preempting rq->curr, the check will be done right @@ -635,26 +1509,53 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ;if (!p->on_rq || rq->curr != p) { -if (task_has_dl_policy (rq->curr)) +#ifdef CONFIG_SMP +if (rq->dl.overloaded &&push_dl_task (rq) && rq !=task_rq (p)) +/* Only reschedule if pushing failed */ + check_resched =0 ; +#endif /* CONFIG_SMP */ +if (check_resched &&task_has_dl_policy (rq->curr))check_preempt_curr_dl (rq, p,0 ); -else -resched_task (rq->curr); } } +/* +* If the scheduling parameters of a -deadline task changed, +* a push or pull operation might be needed. +*/ static void prio_changed_dl (struct rq *rq,struct task_struct *p,int oldprio) { -switched_to_dl (rq, p); -} - +if (p->on_rq || rq->curr == p) {#ifdef CONFIG_SMP -static int -select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) -{ -return task_cpu (p); +/* +* This might be too much, but unfortunately +* we don't have the old deadline value, and +* we can't argue if the task is increasing +* or lowering its prio, so... +*/ +if (!rq->dl.overloaded) +pull_dl_task (rq); + +/* +* If we now have a earlier deadline task than p, +* then reschedule, provided p is still on this +* runqueue. +*/ +if (dl_time_before (rq->dl.earliest_dl.curr, p->dl.deadline) && + rq->curr == p) +resched_task (p); +#else +/* +* Again, we don't know if p has a earlier +* or later deadline, so let's blindly set a +* (maybe not needed) rescheduling point. +*/ +resched_task (p); +#endif /* CONFIG_SMP */ + }else +switched_to_dl (rq, p); } -#endif const struct sched_class dl_sched_class = { .next = &rt_sched_class,@@ -669,6 +1570,12 @@ const struct sched_class dl_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl, + .set_cpus_allowed = set_cpus_allowed_dl, + .rq_online = rq_online_dl, + .rq_offline = rq_offline_dl, + .pre_schedule = pre_schedule_dl, + .post_schedule = post_schedule_dl, + .task_woken = task_woken_dl, #endif .set_curr_task = set_curr_task_dl,kernel/sched/rt.c +1
-1@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);kernel/sched/sched.h +34
-0@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP @@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ @@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Re: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 54 lines · Juri Lelli
Quoted 45 lines · Paul E. McKenney
Quoted 44 lines · Juri Lelli
Quoted 42 lines · Steven Rostedt
Quoted 29 lines · On Thu, 7 Nov 2013 14:43:38 +0100 Juri Lelli <[email protected]> wrote:
kernel/sched/deadline.c +10
-0@@ -10,6 +10,7 @@* miss some of their deadlines), and won't affect any other task. * * Copyright (C) 2012 Dario Faggioli <[email protected]>, +* Juri Lelli <[email protected]>, * Michael Trimarchi <[email protected]>, * Fabio Checconi <[email protected]> */ @@ -20,6 +21,15 @@ static inline int dl_time_before(u64 a, u64 b)return (s64)(a - b) <0 ; } +/* +* Tells if entity @a should preempt entity @b. +*/ +static inline +int dl_entity_preempt (struct sched_dl_entity *a,struct sched_dl_entity *b) +{ +return dl_time_before (a->deadline, b->deadline); +} +static inline struct task_struct *dl_task_of (struct sched_dl_entity *dl_se) { return container_of(dl_se, struct task_struct, dl);kernel/sched/deadline.c static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq) · line 68
Quoted 6 lines
kernel/sched/deadline.c +3
-0@@ -53,8 +63,168 @@ static inline int is_leftmost(struct task_struct *p, struct dl_rq *dl_rq)void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq) { dl_rq->rb_root =RB_ROOT ; + +#ifdef CONFIG_SMP +/* zero means no -deadline tasks */ I'm curious to why you add the '-' to -deadline.
Quoted 7 lines
Quoted 3 lines
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.My guess was that "-deadline" was an abbreviation for tasks that had missed their deadline, just so you know. ;-)
Quoted 5 lines
Argh! ;)
Actually, thinking about it twice: if we call "-rt tasks" things that are governed by code that resides in
sched/rt.c, we could agree on "-deadline tasks" for things that pass throughsched/deadline.c.
Actually, "SCHED_DEADLINE" isn't -that- much longer than "-deadline" and it does have the virtue of being unambiguous.
Thanx, Paul
Quoted 1,168 lines
Thanks,
- Juri
Quoted 1,162 lines
Quoted 1,161 lines
Quoted 516 lines
Quoted 503 lines
kernel/sched/deadline.c +418
-3+ dl_rq->earliest_dl.curr = dl_rq->earliest_dl.next =0 ; + + dl_rq->dl_nr_migratory =0 ; + dl_rq->overloaded =0 ; + dl_rq->pushable_dl_tasks_root =RB_ROOT ; +#endif +} + +#ifdef CONFIG_SMP + +static inline int dl_overloaded (struct rq *rq) +{ +return atomic_read (&rq->rd->dlo_count); +} + +static inline void dl_set_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +cpumask_set_cpu (rq->cpu, rq->rd->dlo_mask); +/* +* Must be visible before the overload count is +* set (as in sched_rt.c). +* +* Matched by the barrier in pull_dl_task(). +*/ +smp_wmb (); +atomic_inc (&rq->rd->dlo_count); +} + +static inline void dl_clear_overload (struct rq *rq) +{ +if (!rq->online) +return ; + +atomic_dec (&rq->rd->dlo_count); +cpumask_clear_cpu (rq->cpu, rq->rd->dlo_mask); +} + +static void update_dl_migration (struct dl_rq *dl_rq) +{ +if (dl_rq->dl_nr_migratory && dl_rq->dl_nr_total >1 ) { +if (!dl_rq->overloaded) { +dl_set_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =1 ; + } + }else if (dl_rq->overloaded) { +dl_clear_overload (rq_of_dl_rq (dl_rq)); + dl_rq->overloaded =0 ; + } +} + +static void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total++; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory++; + +update_dl_migration (dl_rq); +} + +static void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +struct task_struct *p =dl_task_of (dl_se); + dl_rq = &rq_of_dl_rq (dl_rq)->dl; + + dl_rq->dl_nr_total--; +if (p->nr_cpus_allowed >1 ) + dl_rq->dl_nr_migratory--; + +update_dl_migration (dl_rq); +} + +/* +* The list of pushable -deadline task is not a plist, like in +* sched_rt.c, it is an rb-tree with tasks ordered by deadline. +*/ +static void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; +struct rb_node **link = &dl_rq->pushable_dl_tasks_root.rb_node; +struct rb_node *parent =NULL ; +struct task_struct *entry; +int leftmost =1 ; + +BUG_ON (!RB_EMPTY_NODE (&p->pushable_dl_tasks)); + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct task_struct, + pushable_dl_tasks); +if (dl_entity_preempt (&p->dl, &entry->dl)) + link = &parent->rb_left; +else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + dl_rq->pushable_dl_tasks_leftmost = &p->pushable_dl_tasks; + +rb_link_node (&p->pushable_dl_tasks, parent, link); +rb_insert_color (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +} + +static void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +struct dl_rq *dl_rq = &rq->dl; + +if (RB_EMPTY_NODE (&p->pushable_dl_tasks)) +return ; + +if (dl_rq->pushable_dl_tasks_leftmost == &p->pushable_dl_tasks) { +struct rb_node *next_node; + + next_node =rb_next (&p->pushable_dl_tasks); + dl_rq->pushable_dl_tasks_leftmost = next_node; + } + +rb_erase (&p->pushable_dl_tasks, &dl_rq->pushable_dl_tasks_root); +RB_CLEAR_NODE (&p->pushable_dl_tasks); +} + +static inline int has_pushable_dl_tasks (struct rq *rq) +{ +return !RB_EMPTY_ROOT (&rq->dl.pushable_dl_tasks_root); +} + +static int push_dl_task (struct rq *rq); + +#else + +static inline +void enqueue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void dequeue_pushable_dl_task (struct rq *rq,struct task_struct *p) +{ +} + +static inline +void inc_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +} + +static inline +void dec_dl_migration (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ } +#endif /* CONFIG_SMP */ +static void enqueue_task_dl (struct rq *rq,struct task_struct *p,int flags);static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags); static void check_preempt_curr_dl(struct rq *rq, struct task_struct *p,@@ -307,6 +477,14 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)check_preempt_curr_dl (rq, p,0 ); else resched_task(rq->curr); +#ifdef CONFIG_SMP +/* +* Queueing this task back might have overloaded rq, +* check if we need to kick someone away. +*/ +if (has_pushable_dl_tasks (rq)) +push_dl_task (rq); +#endif } unlock:raw_spin_unlock (&rq->lock);@@ -397,6 +575,100 @@ static void update_curr_dl(struct rq *rq) } } +#ifdef CONFIG_SMP + +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu); + +static inline u64 next_deadline (struct rq *rq) +{ +struct task_struct *next =pick_next_earliest_dl_task (rq, rq->cpu); + +if (next &&dl_prio (next->prio)) +return next->dl.deadline; +else +return 0 ; +} + +static void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +if (dl_rq->earliest_dl.curr ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.curr)) { +/* +* If the dl_rq had no -deadline tasks, or if the new task +* has shorter deadline than the current one on dl_rq, we +* know that the previous earliest becomes our next earliest, +* as the new task becomes the earliest itself. +*/ + dl_rq->earliest_dl.next = dl_rq->earliest_dl.curr; + dl_rq->earliest_dl.curr = deadline; + }else if (dl_rq->earliest_dl.next ==0 || +dl_time_before (deadline, dl_rq->earliest_dl.next)) { +/* +* On the other hand, if the new -deadline task has a +* a later deadline than the earliest one on dl_rq, but +* it is earlier than the next (if any), we must +* recompute the next-earliest. +*/ + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +static void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) +{ +struct rq *rq =rq_of_dl_rq (dl_rq); + +/* +* Since we may have removed our earliest (and/or next earliest) +* task we must recompute them. +*/ +if (!dl_rq->dl_nr_running) { + dl_rq->earliest_dl.curr =0 ; + dl_rq->earliest_dl.next =0 ; + }else { +struct rb_node *leftmost = dl_rq->rb_leftmost; +struct sched_dl_entity *entry; + + entry = rb_entry(leftmost, struct sched_dl_entity, rb_node); + dl_rq->earliest_dl.curr = entry->deadline; + dl_rq->earliest_dl.next =next_deadline (rq); + } +} + +#else + +static inline void inc_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} +static inline void dec_dl_deadline (struct dl_rq *dl_rq,u64 deadline) {} + +#endif /* CONFIG_SMP */ + +static inline +void inc_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; +u64 deadline = dl_se->deadline; + +WARN_ON (!dl_prio (prio)); + dl_rq->dl_nr_running++; + +inc_dl_deadline (dl_rq, deadline); +inc_dl_migration (dl_se, dl_rq); +} + +static inline +void dec_dl_tasks (struct sched_dl_entity *dl_se,struct dl_rq *dl_rq) +{ +int prio =dl_task_of (dl_se)->prio; + +WARN_ON (!dl_prio (prio)); +WARN_ON (!dl_rq->dl_nr_running); + dl_rq->dl_nr_running--; + +dec_dl_deadline (dl_rq, dl_se->deadline); +dec_dl_migration (dl_se, dl_rq); +} +static void __enqueue_dl_entity (struct sched_dl_entity *dl_se) {struct dl_rq *dl_rq =dl_rq_of_se (dl_se);@@ -424,7 +696,7 @@ static void __enqueue_dl_entity(struct sched_dl_entity *dl_se)rb_link_node (&dl_se->rb_node, parent, link);rb_insert_color (&dl_se->rb_node, &dl_rq->rb_root); - dl_rq->dl_nr_running++; +inc_dl_tasks (dl_se, dl_rq); }static void __dequeue_dl_entity (struct sched_dl_entity *dl_se)@@ -444,7 +716,7 @@ static void __dequeue_dl_entity(struct sched_dl_entity *dl_se)rb_erase (&dl_se->rb_node, &dl_rq->rb_root);RB_CLEAR_NODE (&dl_se->rb_node); - dl_rq->dl_nr_running--; +dec_dl_tasks (dl_se, dl_rq); } static void@@ -482,12 +754,17 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)return ;enqueue_dl_entity (&p->dl, flags); + +if (!task_current (rq, p) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); +inc_nr_running (rq); }static void __dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags) {dequeue_dl_entity (&p->dl); +dequeue_pushable_dl_task (rq, p); }static void dequeue_task_dl (struct rq *rq,struct task_struct *p,int flags)@@ -525,6 +802,77 @@ static void yield_task_dl(struct rq *rq)update_curr_dl (rq); } +#ifdef CONFIG_SMP + +static int find_later_rq (struct task_struct *task); +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask); + +static int +select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) +{ +struct task_struct *curr; +struct rq *rq; +int cpu; + + cpu =task_cpu (p); + +if (sd_flag !=SD_BALANCE_WAKE && sd_flag !=SD_BALANCE_FORK ) + goto out; + + rq =cpu_rq (cpu); + +rcu_read_lock (); + curr =ACCESS_ONCE (rq->curr);/* unlocked access */ + +/* +* If we are dealing with a -deadline task, we must +* decide where to wake it up. +* If it has a later deadline and the current task +* on this rq can't move (provided the waking task +* can!) we prefer to send it somewhere else. On the +* other hand, if it has a shorter deadline, we +* try to make it stay here, it might be important. +*/ +if (unlikely (dl_task (curr)) && + (curr->nr_cpus_allowed <2 || + !dl_entity_preempt (&p->dl, &curr->dl)) && + (p->nr_cpus_allowed >1 )) { +int target =find_later_rq (p); + +if (target !=-1 ) + cpu = target; + } +rcu_read_unlock (); + +out: +return cpu; +} + +static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p) +{ +/* +* Current can't be migrated, useless to reschedule, +* let's hope p can move out. +*/ +if (rq->curr->nr_cpus_allowed ==1 || +latest_cpu_find (rq->rd->span, rq->curr,NULL ) ==-1 ) +return ; + +/* +* p is migratable, so let's not schedule it and +* see if it is pushed or pulled somewhere else. +*/ +if (p->nr_cpus_allowed !=1 && +latest_cpu_find (rq->rd->span, p,NULL ) !=-1 ) +return ; + +resched_task (rq->curr); +} + +#endif /* CONFIG_SMP */ +/* * Only called when both the current and waking task are -deadline * tasks. @@ -532,8 +880,20 @@ static void yield_task_dl(struct rq *rq)static void check_preempt_curr_dl (struct rq *rq,struct task_struct *p,int flags) { -if (dl_time_before (p->dl.deadline, rq->curr->dl.deadline)) +if (dl_entity_preempt (&p->dl, &rq->curr->dl)) {resched_task (rq->curr); +return ; + } + +#ifdef CONFIG_SMP +/* +* In the unlikely case current and p have the same deadline +* let us try to decide what's the best thing to do... +*/ +if ((s64)(p->dl.deadline - rq->curr->dl.deadline) ==0 && + !need_resched ()) +check_preempt_equal_dl (rq, p); +#endif /* CONFIG_SMP */ }#ifdef CONFIG_SCHED_HRTICK @@ -573,16 +933,29 @@ struct task_struct *pick_next_task_dl(struct rq *rq) p =dl_task_of (dl_se); p->se.exec_start =rq_clock_task (rq); + +/* Running task will never be pushed. */ +if (p) +dequeue_pushable_dl_task (rq, p); +#ifdef CONFIG_SCHED_HRTICK if (hrtick_enabled (rq))start_hrtick_dl (rq, p);#endif + +#ifdef CONFIG_SMP + rq->post_schedule =has_pushable_dl_tasks (rq); +#endif /* CONFIG_SMP */ +return p; }static void put_prev_task_dl (struct rq *rq,struct task_struct *p) {update_curr_dl (rq); + +if (on_dl_rq (&p->dl) && p->nr_cpus_allowed >1 ) +enqueue_pushable_dl_task (rq, p); }static void task_tick_dl (struct rq *rq,struct task_struct *p,int queued)@@ -616,16 +989,517 @@ static void set_curr_task_dl(struct rq *rq)struct task_struct *p = rq->curr; p->se.exec_start =rq_clock_task (rq); + +/* You can't push away the running task */ +dequeue_pushable_dl_task (rq, p); +} + +#ifdef CONFIG_SMP + +/* Only try algorithms three times */ +#define DL_MAX_TRIES 3 + +static int pick_dl_task (struct rq *rq,struct task_struct *p,int cpu) +{ +if (!task_running (rq, p) && + (cpu <0 ||cpumask_test_cpu (cpu, &p->cpus_allowed)) && + (p->nr_cpus_allowed >1 )) +return 1 ; + +return 0 ; +} + +/* Returns the second earliest -deadline task, NULL otherwise */ +static struct task_struct *pick_next_earliest_dl_task (struct rq *rq,int cpu) +{ +struct rb_node *next_node = rq->dl.rb_leftmost; +struct sched_dl_entity *dl_se; +struct task_struct *p =NULL ; + +next_node: + next_node =rb_next (next_node); +if (next_node) { + dl_se = rb_entry(next_node, struct sched_dl_entity, rb_node); + p =dl_task_of (dl_se); + +if (pick_dl_task (rq, p, cpu)) +return p; + + goto next_node; + } + +return NULL ; +} + +static int latest_cpu_find (struct cpumask *span, +struct task_struct *task, +struct cpumask *later_mask) +{ +const struct sched_dl_entity *dl_se = &task->dl; +int cpu, found =-1 , best =0 ; +u64 max_dl =0 ; + +for_each_cpu (cpu, span) { +struct rq *rq =cpu_rq (cpu); +struct dl_rq *dl_rq = &rq->dl; + + if (cpumask_test_cpu(cpu, &task->cpus_allowed) &&kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1056
Quoted 10 lines
kernel/sched/deadline.c +10
-0+ (!dl_rq->dl_nr_running || dl_time_before(dl_se->deadline, + dl_rq->earliest_dl.curr))) { +if (later_mask) +cpumask_set_cpu (cpu, later_mask); +if (!best && !dl_rq->dl_nr_running) { + best =1 ; + found = cpu; + }else if (!best && +dl_time_before (max_dl, + dl_rq->earliest_dl.curr)) {Ug, the above is hard to read. What about:
Quoted 9 lines
if (!best) { if (!dl_rq->dl_nr_running) { best = 1; found = cpu; } elsif (dl_time_before(...)) { ... } }This is completely removed in 13/14. I don't like it either, but since we end up removing this mess, do you think we still have to fix this here?
Quoted 4 lines
Also, I would think dl should be nice to rt as well. There may be a idle CPU or a non rt task, and this could pick a CPU running an RT task. Worse yet, that RT task may be pinned to that CPU.
Well, in 13/14 we introduce a
free_cpusmask. A CPU is considered free if it doesn't have any -deadline task running. We can modify that excluding also CPUs running RT tasks, but I have to think a bit if we can do this also from -rt code.Quoted 3 lines
We should be able to incorporate
cpuprio_find()to be dl aware too. That is, work for both -rt and -dl.Like checking if a -dl task is running on the cpu chosen for pushing an -rt task, and continue searching in that case.
Quoted 266 lines
Quoted 266 lines
kernel/sched/deadline.c +262
-0+ max_dl = dl_rq->earliest_dl.curr; + found = cpu; + } + }else if (later_mask) +cpumask_clear_cpu (cpu, later_mask); + } + +return found; +} + +static DEFINE_PER_CPU(cpumask_var_t, local_cpu_mask_dl); + +static int find_later_rq (struct task_struct *task) +{ +struct sched_domain *sd; +struct cpumask *later_mask =__get_cpu_var (local_cpu_mask_dl); +int this_cpu =smp_processor_id (); +int best_cpu, cpu =task_cpu (task); + +/* Make sure the mask is initialized first */ +if (unlikely (!later_mask)) +return -1 ; + +if (task->nr_cpus_allowed ==1 ) +return -1 ; + + best_cpu =latest_cpu_find (task_rq (task)->rd->span, task, later_mask); +if (best_cpu ==-1 ) +return -1 ; + +/* +* If we are here, some target has been found, +* the most suitable of which is cached in best_cpu. +* This is, among the runqueues where the current tasks +* have later deadlines than the task's one, the rq +* with the latest possible one. +* +* Now we check how well this matches with task's +* affinity and system topology. +* +* The last cpu where the task run is our first +* guess, since it is most likely cache-hot there. +*/ +if (cpumask_test_cpu (cpu, later_mask)) +return cpu; +/* +* Check if this_cpu is to be skipped (i.e., it is +* not in the mask) or not. +*/ +if (!cpumask_test_cpu (this_cpu, later_mask)) + this_cpu =-1 ; + +rcu_read_lock (); +for_each_domain (cpu, sd) { +if (sd->flags &SD_WAKE_AFFINE ) { + +/* +* If possible, preempting this_cpu is +* cheaper than migrating. +*/ +if (this_cpu !=-1 && +cpumask_test_cpu (this_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return this_cpu; + } + +/* +* Last chance: if best_cpu is valid and is +* in the mask, that becomes our choice. +*/ +if (best_cpu < nr_cpu_ids && +cpumask_test_cpu (best_cpu,sched_domain_span (sd))) { +rcu_read_unlock (); +return best_cpu; + } + } + } +rcu_read_unlock (); + +/* +* At this point, all our guesses failed, we just return +* 'something', and let the caller sort the things out. +*/ +if (this_cpu !=-1 ) +return this_cpu; + + cpu =cpumask_any (later_mask); +if (cpu < nr_cpu_ids) +return cpu; + +return -1 ; +} + +/* Locks the rq it finds */ +static struct rq *find_lock_later_rq (struct task_struct *task,struct rq *rq) +{ +struct rq *later_rq =NULL ; +int tries; +int cpu; + +for (tries =0 ; tries <DL_MAX_TRIES ; tries++) { + cpu =find_later_rq (task); + +if ((cpu ==-1 ) || (cpu == rq->cpu)) +break ; + + later_rq =cpu_rq (cpu); + +/* Retry if something changed. */ +if (double_lock_balance (rq, later_rq)) { +if (unlikely (task_rq (task) != rq || + !cpumask_test_cpu (later_rq->cpu, + &task->cpus_allowed) || +task_running (rq, task) || !task->on_rq)) { +double_unlock_balance (rq, later_rq); + later_rq =NULL ; +break ; + } + } + +/* +* If the rq we found has no -deadline task, or +* its earliest one has a later deadline than our +* task, the rq is a good one. +*/ +if (!later_rq->dl.dl_nr_running || +dl_time_before (task->dl.deadline, + later_rq->dl.earliest_dl.curr)) +break ; + +/* Otherwise we try again. */ +double_unlock_balance (rq, later_rq); + later_rq =NULL ; + } + +return later_rq; } +static struct task_struct *pick_next_pushable_dl_task (struct rq *rq) +{ +struct task_struct *p; + +if (!has_pushable_dl_tasks (rq)) +return NULL ; + + p =rb_entry (rq->dl.pushable_dl_tasks_leftmost, + struct task_struct, pushable_dl_tasks); + +BUG_ON (rq->cpu !=task_cpu (p)); +BUG_ON (task_current (rq, p)); +BUG_ON (p->nr_cpus_allowed <=1 ); + +BUG_ON (!p->se.on_rq); +BUG_ON (!dl_task (p)); + +return p; +} + +/* +* See if the non running -deadline tasks on this rq +* can be sent to some other CPU where they can preempt +* and start executing. +*/ +static int push_dl_task (struct rq *rq) +{ +struct task_struct *next_task; +struct rq *later_rq; + +if (!rq->dl.overloaded) +return 0 ; + + next_task =pick_next_pushable_dl_task (rq); +if (!next_task) +return 0 ; + +retry: +if (unlikely (next_task == rq->curr)) { +WARN_ON (1 ); +return 0 ; + } + +/* +* If next_task preempts rq->curr, and rq->curr +* can move away, it makes sense to just reschedule +* without going further in pushing next_task. +*/ +if (dl_task (rq->curr) && +dl_time_before (next_task->dl.deadline, rq->curr->dl.deadline) && + rq->curr->nr_cpus_allowed >1 ) { +resched_task (rq->curr); +return 0 ; + } + +/* We might release rq lock */ +get_task_struct (next_task); + +/* Will lock the rq it'll find */ + later_rq =find_lock_later_rq (next_task, rq); +if (!later_rq) { +struct task_struct *task; + +/* +* We must check all this again, since +* find_lock_later_rq releases rq->lock and it is +* then possible that next_task has migrated. +*/ + task =pick_next_pushable_dl_task (rq); +if (task_cpu (next_task) == rq->cpu && task == next_task) { +/* +* The task is still there. We don't try +* again, some other cpu will pull it when ready. +*/ +dequeue_pushable_dl_task (rq, next_task); + goto out; + } + +if (!task) +/* No more tasks */ + goto out; + +put_task_struct (next_task); + next_task = task; + goto retry; + } + +deactivate_task (rq, next_task,0 ); +set_task_cpu (next_task, later_rq->cpu); +activate_task (later_rq, next_task,0 ); + +resched_task (later_rq->curr); + +double_unlock_balance (rq, later_rq); + +out: +put_task_struct (next_task); + +return 1 ; +} + +static void push_dl_tasks (struct rq *rq) +{ +/* Terminates as it moves a -deadline task */ +while (push_dl_task (rq)) + ; +} + +static int pull_dl_task (struct rq *this_rq) +{ +int this_cpu = this_rq->cpu, ret =0 , cpu; +struct task_struct *p; +struct rq *src_rq; +u64 dmin =LONG_MAX ; + +if (likely (!dl_overloaded (this_rq))) +return 0 ; + +/* +* Match the barrier from dl_set_overloaded; this guarantees that if we +* see overloaded we must also see the dlo_mask bit. +*/ +smp_rmb (); + +for_each_cpu (cpu, this_rq->rd->dlo_mask) { +if (this_cpu == cpu)Quoted 9 lines
kernel/sched/deadline.c static void set_curr_task_dl(struct rq *rq) · line 1326
Quoted 6 lines
kernel/sched/deadline.c +6
-0+continue ; + + src_rq =cpu_rq (cpu); + +/* +* It looks racy, abd it is! However, as in sched_rt.c, abd it is?
Oops!
Thanks,
- Juri
Quoted 332 lines
Quoted 331 lines
kernel/sched/deadline.c +213
-12+* we are fine with this. +*/ +if (this_rq->dl.dl_nr_running && +dl_time_before (this_rq->dl.earliest_dl.curr, + src_rq->dl.earliest_dl.next)) +continue ; + +/* Might drop this_rq->lock */ +double_lock_balance (this_rq, src_rq); + +/* +* If there are no more pullable tasks on the +* rq, we're done with it. +*/ +if (src_rq->dl.dl_nr_running <=1 ) + goto skip; + + p =pick_next_earliest_dl_task (src_rq, this_cpu); + +/* +* We found a task to be pulled if: +* - it preempts our current (if there's one), +* - it will preempt the last one we pulled (if any). +*/ +if (p &&dl_time_before (p->dl.deadline, dmin) && + (!this_rq->dl.dl_nr_running || +dl_time_before (p->dl.deadline, + this_rq->dl.earliest_dl.curr))) { +WARN_ON (p == src_rq->curr); +WARN_ON (!p->se.on_rq); + +/* +* Then we pull iff p has actually an earlier +* deadline than the current task of its runqueue. +*/ +if (dl_time_before (p->dl.deadline, + src_rq->curr->dl.deadline)) + goto skip; + + ret =1 ; + +deactivate_task (src_rq, p,0 ); +set_task_cpu (p, this_cpu); +activate_task (this_rq, p,0 ); + dmin = p->dl.deadline; + +/* Is there any other task even earlier? */ + } +skip: + double_unlock_balance(this_rq, src_rq); + } + +return ret; +} + +static void pre_schedule_dl (struct rq *rq,struct task_struct *prev) +{ +/* Try to pull other tasks here */ +if (dl_task (prev)) +pull_dl_task (rq); +} + +static void post_schedule_dl (struct rq *rq) +{ +push_dl_tasks (rq); +} + +/* +* Since the task is not running and a reschedule is not going to happen +* anytime soon on its runqueue, we try pushing it away now. +*/ +static void task_woken_dl (struct rq *rq,struct task_struct *p) +{ +if (!task_running (rq, p) && + !test_tsk_need_resched (rq->curr) && +has_pushable_dl_tasks (rq) && + p->nr_cpus_allowed >1 && +dl_task (rq->curr) && + (rq->curr->nr_cpus_allowed <2 || +dl_entity_preempt (&rq->curr->dl, &p->dl))) { +push_dl_tasks (rq); + } +} + +static void set_cpus_allowed_dl (struct task_struct *p, +const struct cpumask *new_mask) +{ +struct rq *rq; +int weight; + +BUG_ON (!dl_task (p)); + +/* +* Update only if the task is actually running (i.e., +* it is on the rq AND it is not throttled). +*/ +if (!on_dl_rq (&p->dl)) +return ; + + weight =cpumask_weight (new_mask); + +/* +* Only update if the process changes its state from whether it +* can migrate or not. +*/ +if ((p->nr_cpus_allowed >1 ) == (weight >1 )) +return ; + + rq =task_rq (p); + +/* +* The process used to be able to migrate OR it can now migrate +*/ +if (weight <=1 ) { +if (!task_current (rq, p)) +dequeue_pushable_dl_task (rq, p); +BUG_ON (!rq->dl.dl_nr_migratory); + rq->dl.dl_nr_migratory--; + }else { +if (!task_current (rq, p)) +enqueue_pushable_dl_task (rq, p); + rq->dl.dl_nr_migratory++; + } + +update_dl_migration (&rq->dl); +} + +/* Assumes rq->lock is held */ +static void rq_online_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_set_overload (rq); +} + +/* Assumes rq->lock is held */ +static void rq_offline_dl (struct rq *rq) +{ +if (rq->dl.overloaded) +dl_clear_overload (rq); +} + +void init_sched_dl_class (void ) +{ +unsigned int i; + +for_each_possible_cpu (i) +zalloc_cpumask_var_node (&per_cpu (local_cpu_mask_dl, i), +GFP_KERNEL ,cpu_to_node (i)); +} + +#endif /* CONFIG_SMP */ +static void switched_from_dl (struct rq *rq,struct task_struct *p) { -if (hrtimer_active (&p->dl.dl_timer)) +if (hrtimer_active (&p->dl.dl_timer) && !dl_policy (p->policy))hrtimer_try_to_cancel (&p->dl.dl_timer); + +#ifdef CONFIG_SMP +/* +* Since this might be the only -deadline task on the rq, +* this is the right place to try to pull some other one +* from an overloaded cpu, if any. +*/ +if (!rq->dl.dl_nr_running) +pull_dl_task (rq); +#endif } +/* +* When switching to -deadline, we may overload the rq, then +* we try to push someone off, if possible. +*/ static void switched_to_dl (struct rq *rq,struct task_struct *p) { +int check_resched =1 ; +/* * If p is throttled, don't consider the possibility * of preempting rq->curr, the check will be done right @@ -635,26 +1509,53 @@ static void switched_to_dl(struct rq *rq, struct task_struct *p)return ;if (!p->on_rq || rq->curr != p) { -if (task_has_dl_policy (rq->curr)) +#ifdef CONFIG_SMP +if (rq->dl.overloaded &&push_dl_task (rq) && rq !=task_rq (p)) +/* Only reschedule if pushing failed */ + check_resched =0 ; +#endif /* CONFIG_SMP */ +if (check_resched &&task_has_dl_policy (rq->curr))check_preempt_curr_dl (rq, p,0 ); -else -resched_task (rq->curr); } } +/* +* If the scheduling parameters of a -deadline task changed, +* a push or pull operation might be needed. +*/ static void prio_changed_dl (struct rq *rq,struct task_struct *p,int oldprio) { -switched_to_dl (rq, p); -} - +if (p->on_rq || rq->curr == p) {#ifdef CONFIG_SMP -static int -select_task_rq_dl (struct task_struct *p,int sd_flag,int flags) -{ -return task_cpu (p); +/* +* This might be too much, but unfortunately +* we don't have the old deadline value, and +* we can't argue if the task is increasing +* or lowering its prio, so... +*/ +if (!rq->dl.overloaded) +pull_dl_task (rq); + +/* +* If we now have a earlier deadline task than p, +* then reschedule, provided p is still on this +* runqueue. +*/ +if (dl_time_before (rq->dl.earliest_dl.curr, p->dl.deadline) && + rq->curr == p) +resched_task (p); +#else +/* +* Again, we don't know if p has a earlier +* or later deadline, so let's blindly set a +* (maybe not needed) rescheduling point. +*/ +resched_task (p); +#endif /* CONFIG_SMP */ + }else +switched_to_dl (rq, p); } -#endif const struct sched_class dl_sched_class = { .next = &rt_sched_class,@@ -669,6 +1570,12 @@ const struct sched_class dl_sched_class = {#ifdef CONFIG_SMP .select_task_rq = select_task_rq_dl, + .set_cpus_allowed = set_cpus_allowed_dl, + .rq_online = rq_online_dl, + .rq_offline = rq_offline_dl, + .pre_schedule = pre_schedule_dl, + .post_schedule = post_schedule_dl, + .task_woken = task_woken_dl, #endif .set_curr_task = set_curr_task_dl,kernel/sched/rt.c +1
-1@@ -1720,7 +1720,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);kernel/sched/sched.h +34
-0@@ -383,6 +383,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP @@ -403,6 +428,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */ @@ -1063,6 +1095,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
Steven Rostedt
maintainerRe: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 8 lines · On Thu, 21 Nov 2013 08:26:31 -0800 "Paul E. McKenney"…
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.My guess was that "-deadline" was an abbreviation for tasks that had missed their deadline, just so you know. ;-)
Quoted 9 lines
Quoted 5 lines
Argh! ;)
Actually, thinking about it twice: if we call "-rt tasks" things that are governed by code that resides in
sched/rt.c, we could agree on "-deadline tasks" for things that pass throughsched/deadline.c.Actually, "
SCHED_DEADLINE" isn't -that- much longer than "-deadline" and it does have the virtue of being unambiguous.
-sdl ?
Unless you are a fan of "Saturday Day Live!" (for the older generation that can't stay awake for SNL)
-- Steve
Re: [PATCH 04/14] sched: SCHED_DEADLINE SMP-related data structures & logic.
Quoted 13 lines · Steven Rostedt
Quoted 10 lines · On Thu, 21 Nov 2013 08:26:31 -0800 "Paul E. McKenney"…
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
I guess "deadline tasks" is too much generic; "
SCHED_DEADLINEtasks" too long; "DL" or "-dl" can be associated to "download". Nothing special in the end, just tought it was a reasonable abbreviation.My guess was that "-deadline" was an abbreviation for tasks that had missed their deadline, just so you know. ;-)
Argh! ;)
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Actually, thinking about it twice: if we call "-rt tasks" things that are governed by code that resides in
sched/rt.c, we could agree on "-deadline tasks" for things that pass throughsched/deadline.c.Actually, "
SCHED_DEADLINE" isn't -that- much longer than "-deadline" and it does have the virtue of being unambiguous.-sdl ?
I will let you guys hash this one out.
Quoted 2 lines
Unless you are a fan of "Saturday Day Live!" (for the older generation that can't stay awake for SNL)
Isn't that what DVRs are for? ;-)
Thanx, Paul
tip-bot for Juri Lelli
merge bot[tip:sched/core] sched/deadline: Add SCHED_DEADLINE SMP-related data structures & logic
In reply to Juri Lellisched/deadline: Add SCHED_DEADLINE SMP-related data structures & logic
Commit-ID: 1baca4ce16b8cc7d4f50be1f7914799af30a2861 Gitweb: http://git.kernel.org/tip/1baca4ce16b8cc7d4f50be1f7914799af30a2861 Author: Juri Lelli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:38 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:07 +0100
sched/deadline: Add SCHED_DEADLINE SMP-related data structures & logic
Introduces data structures relevant for implementing dynamic migration of -deadline tasks and the logic for checking if runqueues are overloaded with -deadline tasks and for choosing where a task should migrate, when it is the case.
Adds also dynamic migrations to SCHED_DEADLINE, so that tasks can be moved among CPUs when necessary. It is also possible to bind a task to a (set of) CPU(s), thus restricting its capability of migrating, or forbidding migrations at all.
The very same approach used in sched_rt is utilised:
- -deadline tasks are kept into CPU-specific runqueues,
- -deadline tasks are migrated among runqueues to achieve the following:
* on an M-CPU system the M earliest deadline ready tasks are always running;
* affinity/cpusets settings of all the -deadline tasks is always respected.
Therefore, this very special form of "load balancing" is done with an active method, i.e., the scheduler pushes or pulls tasks between runqueues when they are woken up and/or (de)scheduled.
IOW, every time a preemption occurs, the descheduled task might be sent to some other CPU (depending on its deadline) to continue executing (push). On the other hand, every time a CPU becomes idle, it might pull the second earliest deadline ready task from some other CPU.
To enforce this, a pull operation is always attempted before taking any scheduling decision (pre_schedule()), as well as a push one after each scheduling decision (post_schedule()). In addition, when a task arrives or wakes up, the best CPU where to resume it is selected taking into account its affinity mask, the system topology, but also its deadline. E.g., from the scheduling point of view, the best CPU where to wake up (and also where to push) a task is the one which is running the task with the latest deadline among the M executing ones.
In order to facilitate these decisions, per-runqueue "caching" of the deadlines of the currently running and of the first ready task is used. Queued but not running tasks are also parked in another rb-tree to speed-up pushes.
--- include/linux/sched.h | 1 + kernel/sched/core.c | 9 +- kernel/sched/deadline.c | 934 +++++++++++++++++++++++++++++++++++++++++++++++- kernel/sched/rt.c | 2 +- kernel/sched/sched.h | 34 ++ 5 files changed, 963 insertions(+), 17 deletions(-)
include/linux/sched.h +1 -0
@@ -1201,6 +1201,7 @@ struct task_struct {struct list_head tasks;#ifdef CONFIG_SMP struct plist_node pushable_tasks; +struct rb_node pushable_dl_tasks;#endif struct mm_struct *mm, *active_mm;
kernel/sched/core.c +8 -1
@@ -1848,6 +1848,7 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)init_task_preempt_count (p);#ifdef CONFIG_SMP plist_node_init (&p->pushable_tasks,MAX_PRIO ); +RB_CLEAR_NODE (&p->pushable_dl_tasks);#endif put_cpu ();
@@ -5040,6 +5041,7 @@ static void free_rootdomain(struct rcu_head *rcu) struct root_domain *rd = container_of(rcu, struct root_domain, rcu);cpupri_cleanup (&rd->cpupri); +free_cpumask_var (rd->dlo_mask);free_cpumask_var (rd->rto_mask);free_cpumask_var (rd->online);free_cpumask_var (rd->span);
@@ -5091,8 +5093,10 @@ static int init_rootdomain(struct root_domain *rd) goto out;if (!alloc_cpumask_var (&rd->online,GFP_KERNEL )) goto free_span; -if (!alloc_cpumask_var (&rd->rto_mask,GFP_KERNEL )) +if (!alloc_cpumask_var (&rd->dlo_mask,GFP_KERNEL )) goto free_online; +if (!alloc_cpumask_var (&rd->rto_mask,GFP_KERNEL )) + goto free_dlo_mask;if (cpupri_init (&rd->cpupri) !=0 ) goto free_rto_mask;
@@ -5100,6 +5104,8 @@ static int init_rootdomain(struct root_domain *rd) free_rto_mask:free_cpumask_var (rd->rto_mask); +free_dlo_mask: +free_cpumask_var (rd->dlo_mask); free_online:free_cpumask_var (rd->online); free_span:
@@ -6451,6 +6457,7 @@ void __init sched_init_smp(void)free_cpumask_var (non_isolated_cpus);init_sched_rt_class (); + init_sched_dl_class(); }#else void __init sched_init_smp (void )
kernel/sched/deadline.c +919 -15
1,055 lines not on this page · raw message
kernel/sched/rt.c +1 -1
@@ -1738,7 +1738,7 @@ static void task_woken_rt(struct rq *rq, struct task_struct *p) !test_tsk_need_resched (rq->curr) &&has_pushable_tasks (rq) && p->nr_cpus_allowed >1 && - rt_task(rq->curr) && + (dl_task (rq->curr) ||rt_task (rq->curr)) && (rq->curr->nr_cpus_allowed <2 || rq->curr->prio <= p->prio)) push_rt_tasks(rq);
kernel/sched/sched.h +34 -0
@@ -385,6 +385,31 @@ struct dl_rq {struct rb_node *rb_leftmost;unsigned long dl_nr_running; + +#ifdef CONFIG_SMP +/* +* Deadline values of the currently executing and the +* earliest ready task on this rq. Caching these facilitates +* the decision wether or not a ready but not running task +* should migrate somewhere else. +*/ +struct { +u64 curr; +u64 next; + } earliest_dl; + +unsigned long dl_nr_migratory; +unsigned long dl_nr_total; +int overloaded; + +/* +* Tasks on this rq that can be pushed away. They are kept in +* an rb-tree, ordered by tasks' deadlines, with caching +* of the leftmost (earliest deadline) element. +*/ +struct rb_root pushable_dl_tasks_root; +struct rb_node *pushable_dl_tasks_leftmost; +#endif };#ifdef CONFIG_SMP
@@ -405,6 +430,13 @@ struct root_domain {cpumask_var_t online;/* +* The bit corresponding to a CPU gets set here if such CPU has more +* than one runnable -deadline task (as it is below for RT tasks). +*/ +cpumask_var_t dlo_mask; +atomic_t dlo_count; + +/* * The "RT overload" flag: it gets set if a CPU has more than * one runnable RT task. */
@@ -1095,6 +1127,8 @@ static inline void idle_balance(int cpu, struct rq *rq)extern void sysrq_sched_debug_show (void );extern void sched_init_granularity (void );extern void update_max_interval (void ); + +extern void init_sched_dl_class (void );extern void init_sched_rt_class (void );extern void init_sched_fair_class (void );
PATCH 5/14 sched: SCHED_DEADLINE avg_update accounting.
In reply to Juri LelliFrom: Dario Faggioli <[email protected]>
Make the core scheduler and load balancer aware of the load produced by -deadline tasks, by updating the moving average like for sched_rt.
--- kernel/sched/deadline.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/deadline.c +2 -0
@@ -562,6 +562,8 @@ static void update_curr_dl(struct rq *rq) curr->se.exec_start =rq_clock_task (rq);cpuacct_charge (curr, delta_exec); +sched_rt_avg_update (rq, delta_exec); + dl_se->runtime -= delta_exec;if (dl_runtime_exceeded (rq, dl_se)) {__dequeue_task_dl (rq, curr,0 );
Signature · 1 line
1.7.9.5
tip-bot for Dario Faggioli
merge bot[tip:sched/core] sched/deadline: Add SCHED_DEADLINE avg_update accounting
sched/deadline: Add SCHED_DEADLINE avg_update accounting
Commit-ID: 239be4a982154ea0c979fca5846349bb68973aed Gitweb: http://git.kernel.org/tip/239be4a982154ea0c979fca5846349bb68973aed Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:39 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:08 +0100
sched/deadline: Add SCHED_DEADLINE avg_update accounting
Make the core scheduler and load balancer aware of the load produced by -deadline tasks, by updating the moving average like for sched_rt.
--- kernel/sched/deadline.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/sched/deadline.c +2 -0
@@ -564,6 +564,8 @@ static void update_curr_dl(struct rq *rq) curr->se.exec_start =rq_clock_task (rq);cpuacct_charge (curr, delta_exec); +sched_rt_avg_update (rq, delta_exec); + dl_se->runtime -= delta_exec;if (dl_runtime_exceeded (rq, dl_se)) {__dequeue_task_dl (rq, curr,0 );
PATCH 6/14 sched: add period support for -deadline tasks.
In reply to Juri LelliFrom: Harald Gustafsson <[email protected]>
Make it possible to specify a period (different or equal than deadline) for -deadline tasks. Relative deadlines (D_i) are used on task arrivals to generate new scheduling (absolute) deadlines as "d = t + D_i", and periods (P_i) to postpone the scheduling deadlines as "d = d + P_i" when the budget is zero. This is in general useful to model (and schedule) tasks that have slow activation rates (long periods), but have to be scheduled soon once activated (short deadlines).
--- include/linux/sched.h | 1 + kernel/sched/core.c | 15 ++++++++++++--- kernel/sched/deadline.c | 10 +++++++--- 3 files changed, 20 insertions(+), 6 deletions(-)
include/linux/sched.h +1 -0
@@ -1068,6 +1068,7 @@ struct sched_dl_entity {*/ u64 dl_runtime;/* maximum runtime for each instance */ u64 dl_deadline;/* relative deadline of each instance */ +u64 dl_period;/* separation of two instances (period) */ /* * Actual scheduling parameters. Initialized with the values above,
kernel/sched/core.c +12 -3
@@ -1617,6 +1617,7 @@ static void __sched_fork(struct task_struct *p)hrtimer_init (&p->dl.dl_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); p->dl.dl_runtime = p->dl.runtime =0 ; p->dl.dl_deadline = p->dl.deadline =0 ; + p->dl.dl_period =0 ; p->dl.flags =0 ;INIT_LIST_HEAD (&p->rt.run_list);
@@ -3304,6 +3305,10 @@ __setparam_dl(struct task_struct *p, const struct sched_param2 *param2)init_dl_task_timer (dl_se); dl_se->dl_runtime = param2->sched_runtime; dl_se->dl_deadline = param2->sched_deadline; +if (param2->sched_period !=0 ) + dl_se->dl_period = param2->sched_period; +else + dl_se->dl_period = dl_se->dl_deadline; dl_se->flags = param2->sched_flags; dl_se->dl_throttled =0 ; dl_se->dl_new =1 ;
@@ -3317,19 +3322,23 @@ __getparam_dl(struct task_struct *p, struct sched_param2 *param2) param2->sched_priority = p->rt_priority; param2->sched_runtime = dl_se->dl_runtime; param2->sched_deadline = dl_se->dl_deadline; + param2->sched_period = dl_se->dl_period; param2->sched_flags = dl_se->flags; }/* * This function validates the new parameters of a -deadline task. * We ask for the deadline not being zero, and greater or equal -* than the runtime. +* than the runtime, as well as the period of being zero or +* greater than deadline. */ static bool __checkparam_dl (const struct sched_param2 *prm) { -return prm && (&prm->sched_deadline) !=0 && - (s64)(&prm->sched_deadline - &prm->sched_runtime) >=0 ; +return prm && prm->sched_deadline !=0 && + (prm->sched_period ==0 || + (s64)(prm->sched_period - prm->sched_deadline) >=0 ) && + (s64)(prm->sched_deadline - prm->sched_runtime) >=0 ; }/*
kernel/sched/deadline.c +7 -3
@@ -289,7 +289,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* arbitrary large. */ while (dl_se->runtime <=0 ) { - dl_se->deadline += dl_se->dl_deadline; + dl_se->deadline += dl_se->dl_period; dl_se->runtime += dl_se->dl_runtime; }
@@ -329,9 +329,13 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* * This function returns true if: * -* runtime / (deadline - t) > dl_runtime / dl_deadline , +* runtime / (deadline - t) > dl_runtime / dl_period , * * IOW we can't recycle current parameters. +* +* Notice that the bandwidth check is done against the period. For +* task with deadline equal to period this is the same of using +* dl_deadline instead of dl_period in the equation above. */ static bool dl_entity_overflow (struct sched_dl_entity *dl_se,u64 t) {
@@ -355,7 +359,7 @@ static bool dl_entity_overflow(struct sched_dl_entity *dl_se, u64 t)* of anything below microseconds resolution is actually fiction * (but still we want to give the user that illusion >;). */ - left = (dl_se->dl_deadline >>10 ) * (dl_se->runtime >>10 ); + left = (dl_se->dl_period >>10 ) * (dl_se->runtime >>10 ); right = ((dl_se->deadline - t) >>10 ) * (dl_se->dl_runtime >>10 );return dl_time_before (right, left);
Signature · 1 line
1.7.9.5
tip-bot for Harald Gustafsson
merge bot[tip:sched/core] sched/deadline: Add period support for SCHED_DEADLINE tasks
sched/deadline: Add period support for SCHED_DEADLINE tasks
Commit-ID: 755378a47192a3d1f7c3a8ca6c15c1cf76de0af2 Gitweb: http://git.kernel.org/tip/755378a47192a3d1f7c3a8ca6c15c1cf76de0af2 Author: Harald Gustafsson <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:40 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:09 +0100
sched/deadline: Add period support for SCHED_DEADLINE tasks
Make it possible to specify a period (different or equal than deadline) for -deadline tasks. Relative deadlines (D_i) are used on task arrivals to generate new scheduling (absolute) deadlines as "d = t + D_i", and periods (P_i) to postpone the scheduling deadlines as "d = d + P_i" when the budget is zero.
This is in general useful to model (and schedule) tasks that have slow activation rates (long periods), but have to be scheduled soon once activated (short deadlines).
--- include/linux/sched.h | 1 + kernel/sched/core.c | 10 ++++++++-- kernel/sched/deadline.c | 10 +++++++--- 3 files changed, 16 insertions(+), 5 deletions(-)
include/linux/sched.h +1 -0
@@ -1102,6 +1102,7 @@ struct sched_dl_entity {*/ u64 dl_runtime;/* maximum runtime for each instance */ u64 dl_deadline;/* relative deadline of each instance */ +u64 dl_period;/* separation of two instances (period) */ /* * Actual scheduling parameters. Initialized with the values above,
kernel/sched/core.c +8 -2
@@ -1723,6 +1723,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)hrtimer_init (&p->dl.dl_timer,CLOCK_MONOTONIC ,HRTIMER_MODE_REL ); p->dl.dl_runtime = p->dl.runtime =0 ; p->dl.dl_deadline = p->dl.deadline =0 ; + p->dl.dl_period =0 ; p->dl.flags =0 ;INIT_LIST_HEAD (&p->rt.run_list);
@@ -3026,6 +3027,7 @@ __setparam_dl(struct task_struct *p, const struct sched_attr *attr)init_dl_task_timer (dl_se); dl_se->dl_runtime = attr->sched_runtime; dl_se->dl_deadline = attr->sched_deadline; + dl_se->dl_period = attr->sched_period ?: dl_se->dl_deadline; dl_se->flags = attr->sched_flags; dl_se->dl_throttled =0 ; dl_se->dl_new =1 ;
@@ -3067,19 +3069,23 @@ __getparam_dl(struct task_struct *p, struct sched_attr *attr) attr->sched_priority = p->rt_priority; attr->sched_runtime = dl_se->dl_runtime; attr->sched_deadline = dl_se->dl_deadline; + attr->sched_period = dl_se->dl_period; attr->sched_flags = dl_se->flags; }/* * This function validates the new parameters of a -deadline task. * We ask for the deadline not being zero, and greater or equal -* than the runtime. +* than the runtime, as well as the period of being zero or +* greater than deadline. */ static bool __checkparam_dl (const struct sched_attr *attr) {return attr && attr->sched_deadline !=0 && - (s64)(attr->sched_deadline - attr->sched_runtime) >=0 ; + (attr->sched_period ==0 || + (s64)(attr->sched_period - attr->sched_deadline) >=0 ) && + (s64)(attr->sched_deadline - attr->sched_runtime ) >=0 ; }/*
kernel/sched/deadline.c +7 -3
@@ -289,7 +289,7 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* arbitrary large. */ while (dl_se->runtime <=0 ) { - dl_se->deadline += dl_se->dl_deadline; + dl_se->deadline += dl_se->dl_period; dl_se->runtime += dl_se->dl_runtime; }
@@ -329,9 +329,13 @@ static void replenish_dl_entity(struct sched_dl_entity *dl_se)* * This function returns true if: * -* runtime / (deadline - t) > dl_runtime / dl_deadline , +* runtime / (deadline - t) > dl_runtime / dl_period , * * IOW we can't recycle current parameters. +* +* Notice that the bandwidth check is done against the period. For +* task with deadline equal to period this is the same of using +* dl_deadline instead of dl_period in the equation above. */ static bool dl_entity_overflow (struct sched_dl_entity *dl_se,u64 t) {
@@ -355,7 +359,7 @@ static bool dl_entity_overflow(struct sched_dl_entity *dl_se, u64 t)* of anything below microseconds resolution is actually fiction * (but still we want to give the user that illusion >;). */ - left = (dl_se->dl_deadline >>10 ) * (dl_se->runtime >>10 ); + left = (dl_se->dl_period >>10 ) * (dl_se->runtime >>10 ); right = ((dl_se->deadline - t) >>10 ) * (dl_se->dl_runtime >>10 );return dl_time_before (right, left);
PATCH 7/14 sched: add schedstats for -deadline tasks.
In reply to Juri LelliFrom: Dario Faggioli <[email protected]>
Add some typical sched-debug output to dl_rq(s) and some schedstats to -deadline tasks. This helps spotting problems with queue and dequeue operations (incorrect ordering) and gives hints about system status (one can have an idea if the system is overloaded, if tasks are missing their deadlines and the entity of such anomalies).
--- include/linux/sched.h | 13 +++++++++++++ kernel/sched/deadline.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/debug.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 9 ++++++++- 4 files changed, 114 insertions(+), 1 deletion(-)
include/linux/sched.h +13 -0
@@ -1058,6 +1058,15 @@ struct sched_rt_entity {#endif }; +#ifdef CONFIG_SCHEDSTATS +struct sched_stats_dl { +u64 last_dmiss; +u64 last_rorun; +u64 dmiss_max; +u64 rorun_max; +}; +#endif +struct sched_dl_entity {struct rb_node rb_node;
@@ -1097,6 +1106,10 @@ struct sched_dl_entity {* own bandwidth to be enforced, thus we need one timer per task. */ struct hrtimer dl_timer; + +#ifdef CONFIG_SCHEDSTATS +struct sched_stats_dl stats; +#endif };struct rcu_node ;
kernel/sched/deadline.c +47 -0
68 lines not on this page · raw message
kernel/sched/debug.c +46 -0
64 lines not on this page · raw message
kernel/sched/sched.h +8 -1
@@ -384,6 +384,8 @@ struct dl_rq {unsigned long dl_nr_running; +u64 exec_clock; +#ifdef CONFIG_SMP /* * Deadline values of the currently executing and the
@@ -410,6 +412,11 @@ struct dl_rq {#endif }; +#ifdef CONFIG_SCHED_DEBUG +struct sched_dl_entity *__pick_dl_last_entity (struct dl_rq *dl_rq); +void print_dl_stats (struct seq_file *m,int cpu); +#endif +#ifdef CONFIG_SMP /*
@@ -1366,7 +1373,7 @@ extern void print_rt_stats(struct seq_file *m, int cpu);extern void init_cfs_rq (struct cfs_rq *cfs_rq);extern void init_rt_rq (struct rt_rq *rt_rq,struct rq *rq); -extern void init_dl_rq (struct dl_rq *rt_rq,struct rq *rq); +extern void init_dl_rq (struct dl_rq *dl_rq,struct rq *rq);extern void account_cfs_bandwidth_used (int enabled,int was_enabled);
Signature · 1 line
1.7.9.5
PATCH 8/14 sched: add latency tracing for -deadline tasks.
In reply to Juri LelliFrom: Dario Faggioli <[email protected]>
It is very likely that systems that wants/needs to use the new SCHED_DEADLINE policy also want to have the scheduling latency of the -deadline tasks under control.
For this reason a new version of the scheduling wakeup latency, called "wakeup_dl", is introduced.
As a consequence of applying this patch there will be three wakeup latency tracer:
* "wakeup", that deals with all tasks in the system;
* "wakeup_rt", that deals with -rt and -deadline tasks only;
* "wakeup_dl", that deals with -deadline tasks only.
--- kernel/trace/trace_sched_wakeup.c | 44 +++++++++++++++++++++++++++++++++---- kernel/trace/trace_selftest.c | 28 +++++++++++++---------- 2 files changed, 57 insertions(+), 15 deletions(-)
kernel/trace/trace_sched_wakeup.c +40 -4
@@ -27,6 +27,7 @@ static int wakeup_cpu;static int wakeup_current_cpu;static unsigned wakeup_prio =-1 ;static int wakeup_rt; +static int wakeup_dl;static arch_spinlock_t wakeup_lock = (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;
@@ -472,9 +473,17 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)tracing_record_cmdline (p);tracing_record_cmdline (current); -if ((wakeup_rt && !rt_task (p)) || - p->prio >= wakeup_prio || - p->prio >= current->prio) +/* +* Semantic is like this: +* - wakeup tracer handles all tasks in the system, independently +* from their scheduling class; +* - wakeup_rt tracer handles tasks belonging to sched_dl and +* sched_rt class; +* - wakeup_dl handles tasks belonging to sched_dl class only. +*/ +if ((wakeup_dl && !dl_task (p)) || + (wakeup_rt && !dl_task (p) && !rt_task (p)) || + (p->prio >= wakeup_prio || p->prio >= current->prio))return ; pc =preempt_count ();
@@ -486,7 +495,7 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || p->prio >= wakeup_prio) +if (!tracer_enabled || (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;/* reset the trace */
@@ -597,16 +606,25 @@ static int __wakeup_tracer_init(struct trace_array *tr)static int wakeup_tracer_init (struct trace_array *tr) { + wakeup_dl =0 ; wakeup_rt =0 ;return __wakeup_tracer_init (tr); }static int wakeup_rt_tracer_init (struct trace_array *tr) { + wakeup_dl =0 ; wakeup_rt =1 ;return __wakeup_tracer_init (tr); } +static int wakeup_dl_tracer_init (struct trace_array *tr) +{ + wakeup_dl =1 ; + wakeup_rt =0 ; +return __wakeup_tracer_init (tr); +} +static void wakeup_tracer_reset (struct trace_array *tr) {int lat_flag = save_flags &TRACE_ITER_LATENCY_FMT ;
@@ -674,6 +692,20 @@ static struct tracer wakeup_rt_tracer __read_mostly = .use_max_tr = true, }; +static struct tracer wakeup_dl_tracer__read_mostly = +{ + .name ="wakeup_dl" , + .init = wakeup_dl_tracer_init, + .reset = wakeup_tracer_reset, + .start = wakeup_tracer_start, + .stop = wakeup_tracer_stop, + .wait_pipe = poll_wait_pipe, + .print_max =1 , +#ifdef CONFIG_FTRACE_SELFTEST + .selftest = trace_selftest_startup_wakeup, +#endif +}; +__init static int init_wakeup_tracer (void ) {int ret;
@@ -686,6 +718,10 @@ __init static int init_wakeup_tracer(void)if (ret)return ret; + ret =register_tracer (&wakeup_dl_tracer); +if (ret) +return ret; +return 0 ; }core_initcall (init_wakeup_tracer);
kernel/trace/trace_selftest.c +17 -11
@@ -1022,11 +1022,17 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)#ifdef CONFIG_SCHED_TRACER static int trace_wakeup_test_thread (void *data) { -/* Make this a RT thread, doesn't need to be too high */ -static const struct sched_param param = { .sched_priority =5 }; +/* Make this a -deadline thread */ +struct sched_param2 paramx = { + .sched_priority =0 , + .sched_runtime =100000ULL , + .sched_deadline =10000000ULL , + .sched_period =10000000ULL + .sched_flags =0 + };struct completion *x = data; -sched_setscheduler (current,SCHED_FIFO , ¶m); +sched_setscheduler2 (current,SCHED_DEADLINE , ¶mx);/* Make it know we have a new prio */ complete (x);
@@ -1040,8 +1046,8 @@ static int trace_wakeup_test_thread(void *data)/* we are awake, now wait to disappear */ while (!kthread_should_stop ()) {/* -* This is an RT task, do short sleeps to let -* others run. +* This will likely be the system top priority +* task, do short sleeps to let others run. */ msleep (100 ); }
@@ -1054,21 +1060,21 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr) {unsigned long save_max = tracing_max_latency;struct task_struct *p; -struct completion isrt; +struct completion is_ready;unsigned long count;int ret; -init_completion (&isrt); +init_completion (&is_ready); -/* create a high prio thread */ - p =kthread_run (trace_wakeup_test_thread, &isrt,"ftrace-test" ); +/* create a -deadline thread */ + p =kthread_run (trace_wakeup_test_thread, &is_ready,"ftrace-test" );if (IS_ERR (p)) {printk (KERN_CONT "Failed to create ftrace wakeup test thread " );return -1 ; } -/* make sure the thread is running at an RT prio */ -wait_for_completion (&isrt); +/* make sure the thread is running at -deadline policy */ +wait_for_completion (&is_ready);/* start the tracing */ ret =tracer_init (trace, tr);
Signature · 1 line
1.7.9.5
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 15 lines · On Thu, 7 Nov 2013 14:43:42 +0100 Juri Lelli <[email protected]> wrote:
kernel/trace/trace_sched_wakeup.c +11
-0+/* +* Semantic is like this: +* - wakeup tracer handles all tasks in the system, independently +* from their scheduling class; +* - wakeup_rt tracer handles tasks belonging to sched_dl and +* sched_rt class; +* - wakeup_dl handles tasks belonging to sched_dl class only. +*/ +if ((wakeup_dl && !dl_task (p)) || + (wakeup_rt && !dl_task (p) && !rt_task (p)) || + (p->prio >= wakeup_prio || p->prio >= current->prio))return ; pc =preempt_count ();
kernel/trace/trace_sched_wakeup.c probe_wakeup(void *ignore, struct task_struct *p, int success) · line 499
Quoted 6 lines
kernel/trace/trace_sched_wakeup.c +1
-1@@ -486,7 +495,7 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || p->prio >= wakeup_prio) +if (!tracer_enabled || (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;
We probably want to add a "tracing_dl" variable, and do the test like this:
if (!tracer_enabled || tracing_dl || (!dl_task(p) && p->prio >= wakeup_prio))
and for the first if statement too. Otherwise if two dl tasks are running on two different CPUs, the second will override the first. Once you start tracing a dl_task, you shouldn't bother tracing another task until that one wakes up.
if (dl_task(p)) tracing_dl = 1; else tracing_dl = 0;
-- Steve
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 37 lines · Steven Rostedt
Quoted 15 lines · On Thu, 7 Nov 2013 14:43:42 +0100 Juri Lelli <[email protected]> wrote:
(quoted diff) +11
-0+ /* + * Semantic is like this: + * - wakeup tracer handles all tasks in the system, independently + * from their scheduling class; + * - wakeup_rt tracer handles tasks belonging to sched_dl and + * sched_rt class; + * - wakeup_dl handles tasks belonging to sched_dl class only. + */ + if ((wakeup_dl && !dl_task(p)) || + (wakeup_rt && !dl_task(p) && !rt_task(p)) || + (p->prio >= wakeup_prio || p->prio >= current->prio)) return; pc = preempt_count();Review probe_wakeup(void *ignore, struct task_struct *p, int success) · line 499
Quoted 6 lines
(quoted diff) +1
-1@@ -486,7 +495,7 @@ probe_wakeup(void *ignore, struct task_struct *p, int success) arch_spin_lock(&wakeup_lock); /* check for races. */ - if (!tracer_enabled || p->prio >= wakeup_prio) + if (!tracer_enabled || (!dl_task(p) && p->prio >= wakeup_prio)) goto out_locked;We probably want to add a "
tracing_dl" variable, and do the test like this:if (!tracer_enabled || tracing_dl || (!dl_task(p) && p->prio >= wakeup_prio))and for the first if statement too. Otherwise if two dl tasks are running on two different CPUs, the second will override the first. Once you start tracing a
dl_task, you shouldn't bother tracing another task until that one wakes up.
Quoted 5 lines
if (dl_task(p)) tracing_dl = 1; else tracing_dl = 0;
Ok, this fixes the build error:
--------------------------------------
kernel/trace/trace_selftest.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/trace/trace_selftest.c +6 -6
@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data) {/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 , + .sched_flags =0 , .sched_runtime =100000ULL , .sched_deadline =10000000ULL , .sched_period =10000000ULL - .sched_flags =0 };struct completion *x = data; -sched_setscheduler2 (current,SCHED_DEADLINE , ¶mx); +sched_setscheduler2 (current,SCHED_DEADLINE , ¶m);/* Make it know we have a new prio */ complete (x);
@@ -1088,19 +1088,19 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr)while (p->on_rq) {/* -* Sleep to make sure the RT thread is asleep too. +* Sleep to make sure the -deadline thread is asleep too. * On virtual machines we can't rely on timings, * but we want to make sure this test still works. */ msleep (100 ); } -init_completion (&isrt); +init_completion (&is_ready);wake_up_process (p);/* Wait for the task to wake up */ -wait_for_completion (&isrt); +wait_for_completion (&is_ready);/* stop the tracing. */ tracing_stop ();
--------------------------------
And this should implement what you were asking for:
--------------------------------
kernel/trace/trace_sched_wakeup.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
kernel/trace/trace_sched_wakeup.c +24 -4
@@ -28,6 +28,7 @@ static int wakeup_current_cpu;static unsigned wakeup_prio =-1 ;static int wakeup_rt;static int wakeup_dl; +static int tracing_dl =0 ;static arch_spinlock_t wakeup_lock = (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;
@@ -438,6 +439,7 @@ static void __wakeup_reset(struct trace_array *tr) { wakeup_cpu =-1 ; wakeup_prio =-1 ; + tracing_dl =0 ;if (wakeup_task)put_task_struct (wakeup_task);
@@ -481,9 +483,9 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)* sched_rt class; * - wakeup_dl handles tasks belonging to sched_dl class only. */ -if ((wakeup_dl && !dl_task (p)) || +if (tracing_dl || (wakeup_dl && !dl_task (p)) || (wakeup_rt && !dl_task (p) && !rt_task (p)) || - (p->prio >= wakeup_prio || p->prio >= current->prio)) + (!dl_task (p) && (p->prio >= wakeup_prio || p->prio >= current->prio)))return ; pc =preempt_count ();
@@ -495,7 +497,8 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || (!dl_task (p) && p->prio >= wakeup_prio)) +if (!tracer_enabled || tracing_dl || + (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;/* reset the trace */
@@ -505,6 +508,15 @@ probe_wakeup(void *ignore, struct task_struct *p, int success) wakeup_current_cpu = wakeup_cpu; wakeup_prio = p->prio; +/* +* Once you start tracing a -deadline task, don't bother tracing +* another task until the first one wakes up. +*/ +if (dl_task (p)) + tracing_dl =1 ; +else + tracing_dl =0 ; + wakeup_task = p;get_task_struct (wakeup_task);
@@ -700,10 +712,18 @@ static struct tracer wakeup_dl_tracer __read_mostly = .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, .wait_pipe = poll_wait_pipe, - .print_max = 1, + .print_max = true, + .print_header = wakeup_print_header, + .print_line = wakeup_print_line, + .flags = &tracer_flags, + .set_flag = wakeup_set_flag, + .flag_changed = wakeup_flag_changed, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_wakeup, #endif + .open = wakeup_trace_open, + .close = wakeup_trace_close, + .use_max_tr = true, };__init static int init_wakeup_tracer (void )
-----------------------------------
Makes sense? :)
Thanks,
- Juri
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:
kernel/trace/trace_selftest.c +0 -0
Quoted 5 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +0
-0@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data)
kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 10 lines
kernel/trace/trace_selftest.c +2
-2{/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 , + .sched_flags =0 , .sched_runtime =100000ULL , .sched_deadline =10000000ULL , .sched_period =10000000ULL - .sched_flags =0
Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Quoted 13 lines
kernel/trace/trace_selftest.c +0
-1};struct completion *x = data; -------------------------------- kernel/trace/trace_sched_wakeup.c |28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)kernel/trace/trace_sched_wakeup.c +0
-0
kernel/trace/trace_sched_wakeup.c static int wakeup_current_cpu; · line 31
Quoted 4 lines
kernel/trace/trace_sched_wakeup.c +1
-0@@ -28,6 +28,7 @@ static int wakeup_current_cpu;static unsigned wakeup_prio =-1 ;static int wakeup_rt;static int wakeup_dl; +static int tracing_dl =0 ;
Get rid of the ' = 0', its implicit to all static and global variables that are not given any value.
Quoted 35 lines
kernel/trace/trace_sched_wakeup.c +5
-3static arch_spinlock_t wakeup_lock = (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;@@ -438,6 +439,7 @@ static void __wakeup_reset(struct trace_array *tr) { wakeup_cpu =-1 ; wakeup_prio =-1 ; + tracing_dl =0 ;if (wakeup_task)put_task_struct (wakeup_task);@@ -481,9 +483,9 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)* sched_rt class; * - wakeup_dl handles tasks belonging to sched_dl class only. */ -if ((wakeup_dl && !dl_task (p)) || +if (tracing_dl || (wakeup_dl && !dl_task (p)) || (wakeup_rt && !dl_task (p) && !rt_task (p)) || - (p->prio >= wakeup_prio || p->prio >= current->prio)) + (!dl_task (p) && (p->prio >= wakeup_prio || p->prio >= current->prio)))return ; pc =preempt_count ();@@ -495,7 +497,8 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || (!dl_task (p) && p->prio >= wakeup_prio)) +if (!tracer_enabled || tracing_dl || + (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;/* reset the trace */ @@ -505,6 +508,15 @@ probe_wakeup(void *ignore, struct task_struct *p, int success) wakeup_current_cpu = wakeup_cpu;
kernel/trace/trace_sched_wakeup.c probe_wakeup(void *ignore, struct task_struct *p, int success) · line 518
Quoted 10 lines
kernel/trace/trace_sched_wakeup.c +8
-0wakeup_prio = p->prio; +/* +* Once you start tracing a -deadline task, don't bother tracing +* another task until the first one wakes up. +*/ +if (dl_task (p)) + tracing_dl =1 ; +else + tracing_dl =0 ;
Do we need the else statement? I would think the only way to get here is if tracing_dl is already set to zero.
-- Steve
Quoted 31 lines
kernel/trace/trace_sched_wakeup.c +10
-2+ wakeup_task = p;get_task_struct (wakeup_task);@@ -700,10 +712,18 @@ static struct tracer wakeup_dl_tracer __read_mostly = .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, .wait_pipe = poll_wait_pipe, - .print_max = 1, + .print_max = true, + .print_header = wakeup_print_header, + .print_line = wakeup_print_line, + .flags = &tracer_flags, + .set_flag = wakeup_set_flag, + .flag_changed = wakeup_flag_changed, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_wakeup, #endif + .open = wakeup_trace_open, + .close = wakeup_trace_close, + .use_max_tr = true, };__init static int init_wakeup_tracer (void ) -----------------------------------Makes sense? :)
Thanks,
- Juri
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 13 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 10 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +1
-1@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data) {/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 ,
Quoted 9 lines
kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 5 lines
kernel/trace/trace_selftest.c +1
-1+ .sched_flags = 0, .sched_runtime = 100000ULL, .sched_deadline =10000000ULL , .sched_period = 10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Right.
Quoted 8 lines
Quoted 8 lines
kernel/trace/trace_selftest.c +0
-1};struct completion *x = data; -------------------------------- kernel/trace/trace_sched_wakeup.c |28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)
Quoted 13 lines
kernel/trace/trace_sched_wakeup.c static int wakeup_current_cpu; · line 31
Quoted 9 lines
kernel/trace/trace_sched_wakeup.c +1
-0@@ -28,6 +28,7 @@ static int wakeup_current_cpu;static unsigned wakeup_prio =-1 ;static int wakeup_rt;static int wakeup_dl; +static int tracing_dl =0 ;Get rid of the ' = 0', its implicit to all static and global variables that are not given any value.
And right.
Quoted 37 lines
Quoted 37 lines
kernel/trace/trace_sched_wakeup.c +5
-3static arch_spinlock_t wakeup_lock = (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;@@ -438,6 +439,7 @@ static void __wakeup_reset(struct trace_array *tr) { wakeup_cpu =-1 ; wakeup_prio =-1 ; + tracing_dl =0 ;if (wakeup_task)put_task_struct (wakeup_task);@@ -481,9 +483,9 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)* sched_rt class; * - wakeup_dl handles tasks belonging to sched_dl class only. */ -if ((wakeup_dl && !dl_task (p)) || +if (tracing_dl || (wakeup_dl && !dl_task (p)) || (wakeup_rt && !dl_task (p) && !rt_task (p)) || - (p->prio >= wakeup_prio || p->prio >= current->prio)) + (!dl_task (p) && (p->prio >= wakeup_prio || p->prio >= current->prio)))return ; pc =preempt_count ();@@ -495,7 +497,8 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || (!dl_task (p) && p->prio >= wakeup_prio)) +if (!tracer_enabled || tracing_dl || + (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;/* reset the trace */ @@ -505,6 +508,15 @@ probe_wakeup(void *ignore, struct task_struct *p, int success) wakeup_current_cpu = wakeup_cpu; wakeup_prio = p->prio;
Quoted 12 lines
kernel/trace/trace_sched_wakeup.c probe_wakeup(void *ignore, struct task_struct *p, int success) · line 518
Quoted 8 lines
kernel/trace/trace_sched_wakeup.c +8
-0+/* +* Once you start tracing a -deadline task, don't bother tracing +* another task until the first one wakes up. +*/ +if (dl_task (p)) + tracing_dl =1 ; +else + tracing_dl =0 ;Do we need the else statement? I would think the only way to get here is if
tracing_dlis already set to zero.
No, indeed.
Thanks,
- Juri
Quoted 32 lines
Quoted 31 lines
kernel/trace/trace_sched_wakeup.c +10
-2+ wakeup_task = p;get_task_struct (wakeup_task);@@ -700,10 +712,18 @@ static struct tracer wakeup_dl_tracer __read_mostly = .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, .wait_pipe = poll_wait_pipe, - .print_max = 1, + .print_max = true, + .print_header = wakeup_print_header, + .print_line = wakeup_print_line, + .flags = &tracer_flags, + .set_flag = wakeup_set_flag, + .flag_changed = wakeup_flag_changed, #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_wakeup, #endif + .open = wakeup_trace_open, + .close = wakeup_trace_close, + .use_max_tr = true, };__init static int init_wakeup_tracer (void ) -----------------------------------Makes sense? :)
Thanks,
- Juri
Peter Zijlstra
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Steven RostedtQuoted 13 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 10 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +1
-1@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data) {/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 ,
Quoted 8 lines
kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 5 lines
kernel/trace/trace_selftest.c +1
-1+ .sched_flags = 0, .sched_runtime = 100000ULL, .sched_deadline =10000000ULL , .sched_period = 10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Only because its static. Otherwise unnamed members have indeterminate value after initialization.
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 23 lines · Peter Zijlstra
Quoted 13 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 10 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +1
-1@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data) {/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 ,Quoted 8 lines
kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 5 lines
kernel/trace/trace_selftest.c +1
-1+ .sched_flags = 0, .sched_runtime = 100000ULL, .sched_deadline =10000000ULL , .sched_period = 10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Quoted 2 lines
Only because its static. Otherwise unnamed members have indeterminate value after initialization.
I think for 'struct' C will initialize them to zero, even if they are not mentioned and even if they are on the stack.
It will only be indeterminate when it's not initialized at all.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 26 lines · Ingo Molnar
Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 5 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +0
-0@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data)kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 10 lines
kernel/trace/trace_selftest.c +2
-2{/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 , + .sched_flags =0 , .sched_runtime =100000ULL , .sched_deadline =10000000ULL , .sched_period =10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Quoted 7 lines
Quoted 2 lines
Only because its static. Otherwise unnamed members have indeterminate value after initialization.
I think for 'struct' C will initialize them to zero, even if they are not mentioned and even if they are on the stack.
It will only be indeterminate when it's not initialized at all.
Language spec: ISO/IEC 9899:1999 (aka C99) section 6.7.8 point 9 says:
9. Except where explicitly stated otherwise, for the purpose of this subclause unnamed members of objects of structure and union type do no participate in initialization. Unnamed members of structure objects have indeterminate value even after initialization.
Later points (notably 21) make such an exception for aggregate objects of static storage.
Of course, its entirely possible I read the thing wrong; its 31 points detailing the initialization of objects.
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 43 lines · Peter Zijlstra
Quoted 26 lines · Ingo Molnar
Quoted 23 lines · Peter Zijlstra
Quoted 21 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 5 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +0
-0@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data)kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 10 lines
kernel/trace/trace_selftest.c +2
-2{/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 , + .sched_flags =0 , .sched_runtime =100000ULL , .sched_deadline =10000000ULL , .sched_period =10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Quoted 7 lines
Quoted 2 lines
Only because its static. Otherwise unnamed members have indeterminate value after initialization.
I think for 'struct' C will initialize them to zero, even if they are not mentioned and even if they are on the stack.
It will only be indeterminate when it's not initialized at all.
Language spec: ISO/IEC 9899:1999 (aka C99) section 6.7.8 point 9 says:
9. Except where explicitly stated otherwise, for the purpose of this subclause unnamed members of objects of structure and union type do no participate in initialization. Unnamed members of structure objects have indeterminate value even after initialization.
Quoted 5 lines
Later points (notably 21) make such an exception for aggregate objects of static storage.
Of course, its entirely possible I read the thing wrong; its 31 points detailing the initialization of objects.
So why does GCC then behave like this:
triton:~> cat test.c
struct foo {
int a;
int b;
};int main(void)
{
struct foo x = { .a = 1 };
return x.b;
}triton:~> gcc -Wall -Wextra -O2 -o test test.c; ./test; echo $? 0
I'd expect -Wall -Wextra to warn about as trivial as the uninitialized variable use that you argue happens.
I'd also expect it to not return 0 but some random value on the stack (which is most likely not 0).
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
So from C99 standard §6.7.8 (Initialization)/21:
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."
static initialization == zeroing in this case.
Thanks,
Ingo
Peter Zijlstra
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 14 lines · Ingo Molnar
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
So from C99 standard §6.7.8 (Initialization)/21:
Quoted 8 lines
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."static initialization == zeroing in this case.
Hurm for some reason I thought that was for static objects only.
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Ingo MolnarQuoted 14 lines · On Wed, 27 Nov 2013 16:46:00 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
So from C99 standard §6.7.8 (Initialization)/21:
Quoted 9 lines
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."static initialization == zeroing in this case.
The confusion here is that the above looks to be talking about arrays. But it really doesn't specify structures.
But searching the internet, it looks as though most people believe it applies to structures, and any compiler that does otherwise will most likely break applications.
That is, this looks to be one of the gray areas that the compiler writers just happen to do what's most sane. And they probably assume it's talking about structures as well, hence the lack of warnings.
It gets confusing, as the doc also shows:
struct { int a[3], b; } w[] = { { 1 }, 2 };
Then points out that w.a[0] is 1 and w.b[0] is 2, and all other elements are zero.
-- Steve
Peter Zijlstra
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 15 lines · Steven Rostedt
Quoted 11 lines
So from C99 standard §6.7.8 (Initialization)/21:
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."static initialization == zeroing in this case.
The confusion here is that the above looks to be talking about arrays. But it really doesn't specify structures.
Quoted 7 lines
But searching the internet, it looks as though most people believe it applies to structures, and any compiler that does otherwise will most likely break applications.
That is, this looks to be one of the gray areas that the compiler writers just happen to do what's most sane. And they probably assume it's talking about structures as well, hence the lack of warnings.
16 says initializers for aggregate or union types are brace-enclosed lists. A struct is an aggregate type.
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Steven RostedtQuoted 12 lines · On Wed, 27 Nov 2013 10:56:16 -0500 Steven Rostedt <[email protected]> wrote:
Quoted 9 lines · On Wed, 27 Nov 2013 16:46:00 +0100 Ingo Molnar <[email protected]> wrote:
So from C99 standard §6.7.8 (Initialization)/21:
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."
Quoted 5 lines
Quoted 2 lines
static initialization == zeroing in this case.
The confusion here is that the above looks to be talking about arrays. But it really doesn't specify structures.
Bah, re-reading it, it does seem to talk about structures.
I hate reading specs, they learn how to write documentation from lawyers.
-- Steve
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Steven RostedtQuoted 24 lines · Steven Rostedt
Quoted 21 lines · On Wed, 27 Nov 2013 16:46:00 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
So from C99 standard §6.7.8 (Initialization)/21:
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."
Quoted 5 lines
Quoted 2 lines
static initialization == zeroing in this case.
The confusion here is that the above looks to be talking about arrays. But it really doesn't specify structures.
It talks about neither 'arrays' nor 'structures', it talks about 'aggregates' - which is defined as _both_: 'structures and arrays'.
That's what compiler legalese brings you ;-)
Quoted 7 lines
But searching the internet, it looks as though most people believe it applies to structures, and any compiler that does otherwise will most likely break applications.
That is, this looks to be one of the gray areas that the compiler writers just happen to do what's most sane. And they probably assume it's talking about structures as well, hence the lack of warnings.
I don't think it's grey, I think it's pretty well specified.
Quoted 3 lines
It gets confusing, as the doc also shows:
struct { int a[3], b; } w[] = { { 1 }, 2 };
I don't think this is valid syntax, I think this needs one more set of braces:
struct { int a[3], b; } w[] = { { { 1 }, 2 } };
Quoted 2 lines
Then points out that w.a[0] is 1 and w.b[0] is 2, and all other elements are zero.
If by 'w.a[0]' you mean 'w[0].a[0]', and if by 'w.b[0]' you mean 'w[0].b' then yes, this comes from the definition and it's what I'd call 'obvious' initialization behavior.
What makes it confusing to you?
Thanks,
Ingo
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
Quoted 27 lines · On Wed, 27 Nov 2013 17:13:08 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 24 lines · Steven Rostedt
Quoted 21 lines · On Wed, 27 Nov 2013 16:46:00 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Peter Zijlstra
Quoted 1 line · Ingo Molnar
So why does GCC then behave like this:
I think because its a much saner behaviour; also it might still be the spec actually says this, its a somewhat opaque text.
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
So from C99 standard §6.7.8 (Initialization)/21:
"If there are fewer initializers in a brace-enclosed list than
there are elements or members of an aggregate, or fewer characters in a string literal used to initialize an array of known size than there are elements in the array, the remainder of the aggregate shall be initialized implicitly the same as objects that have static storage duration."
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
static initialization == zeroing in this case.
The confusion here is that the above looks to be talking about arrays. But it really doesn't specify structures.
It talks about neither 'arrays' nor 'structures', it talks about 'aggregates' - which is defined as _both_: 'structures and arrays'.
Yeah, I misread it. I was reading the array section for awhile, and got confused.
Quoted 2 lines
That's what compiler legalese brings you ;-)
Yep.
Quoted 20 lines
Quoted 8 lines
But searching the internet, it looks as though most people believe it applies to structures, and any compiler that does otherwise will most likely break applications.
That is, this looks to be one of the gray areas that the compiler writers just happen to do what's most sane. And they probably assume it's talking about structures as well, hence the lack of warnings.
I don't think it's grey, I think it's pretty well specified.
Quoted 3 lines
It gets confusing, as the doc also shows:
struct { int a[3], b; } w[] = { { 1 }, 2 };
I don't think this is valid syntax, I think this needs one more set of braces:
struct { int a[3], b; } w[] = { { { 1 }, 2 } };
Quoted 8 lines
Quoted 2 lines
Then points out that w.a[0] is 1 and w.b[0] is 2, and all other elements are zero.
If by 'w.a[0]' you mean 'w[0].a[0]', and if by 'w.b[0]' you mean 'w[0].b' then yes, this comes from the definition and it's what I'd call 'obvious' initialization behavior.
What makes it confusing to you?
Well, because it's mixing arrays and structures, and I was on the misconception that the paragraph was talking about just arrays.
Can I just have my turkey now?
-- Steve
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Peter ZijlstraQuoted 3 lines · Peter Zijlstra
Anyway, yes GCC seems to behave as we 'expect' it to; I just can't find the language spec actually guaranteeing this.
And the kernel already assumes that gcc should do this, for example
struct siginfo info = {};in do_tkill().
(this initialization should prevent the leak of uninitialized members
to userspace).
Oleg.
Re: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Peter Zijlstraand even this works:
triton:~> cat test.c
struct foo {
int a;
int b;
};int litter_our_stack(void)
{
volatile struct foo x = { .a = 1, .b = 2 };
return x.b;
}int test_code(void)
{
volatile struct foo x = { .a = 1, /* .b not initialized explicitly */ };
return x.b;
}int main(void)
{
return litter_our_stack() + test_code();
}triton:~> gcc -Wall -Wextra -O0 -o test test.c; ./test; echo $? 2
triton:~>
The result is 2, so x.b in test_code() got explicitly set to 0.
If it was uninitialized, not only would we expect a compiler warning, but we'd also get a result of '4'. (the two functions have the same stack depth, so 'litter_our_stack()' initializes .b to 2.)
-O0 guarantees that GCC just dumbly implements these functions without any optimizations.
Thanks,
ngo
Steven Rostedt
maintainerRe: [PATCH 08/14] sched: add latency tracing for -deadline tasks.
In reply to Peter ZijlstraQuoted 23 lines · On Wed, 27 Nov 2013 15:26:49 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 13 lines · Steven Rostedt
On Wed, 27 Nov 2013 14:43:45 +0100
Juri Lelli <[email protected]> wrote:kernel/trace/trace_selftest.c +0
-0Quoted 10 lines
index f76f8d6..ad94604 100644
kernel/trace/trace_selftest.c +1
-1@@ -1023,16 +1023,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)static int trace_wakeup_test_thread (void *data) {/* Make this a -deadline thread */ -struct sched_param2 paramx = { +static const struct sched_param2 param = { .sched_priority =0 ,Quoted 8 lines
kernel/trace/trace_selftest.c trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr) · line 1031
Quoted 5 lines
kernel/trace/trace_selftest.c +1
-1+ .sched_flags = 0, .sched_runtime = 100000ULL, .sched_deadline =10000000ULL , .sched_period = 10000000ULL - .sched_flags =0 Assigning structures like this, you don't need to set the zero fields. all fields not explicitly stated, are set to zero.
Quoted 2 lines
Only because its static. Otherwise unnamed members have indeterminate value after initialization.
Hmm, I wonder if there's a gcc extension that guarantees it. Because I could have sworn I've seen initialization of structures that expected zero'd behavior elsewhere in the kernel.
-- Steve
tip-bot for Dario Faggioli
merge bot[tip:sched/core] sched/deadline: Add latency tracing for SCHED_DEADLINE tasks
In reply to Juri Lellisched/deadline: Add latency tracing for SCHED_DEADLINE tasks
Commit-ID: af6ace764d03900524e9b1ac621a1c520ee49fc6 Gitweb: http://git.kernel.org/tip/af6ace764d03900524e9b1ac621a1c520ee49fc6 Author: Dario Faggioli <[email protected]> AuthorDate: Thu, 7 Nov 2013 14:43:42 +0100 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 13 Jan 2014 13:41:11 +0100
sched/deadline: Add latency tracing for SCHED_DEADLINE tasks
It is very likely that systems that wants/needs to use the new SCHED_DEADLINE policy also want to have the scheduling latency of the -deadline tasks under control.
For this reason a new version of the scheduling wakeup latency, called "wakeup_dl", is introduced.
As a consequence of applying this patch there will be three wakeup latency tracer:
* "wakeup", that deals with all tasks in the system;
* "wakeup_rt", that deals with -rt and -deadline tasks only;
* "wakeup_dl", that deals with -deadline tasks only.
--- kernel/trace/trace_sched_wakeup.c | 64 ++++++++++++++++++++++++++++++++++++--- kernel/trace/trace_selftest.c | 33 +++++++++++--------- 2 files changed, 79 insertions(+), 18 deletions(-)
kernel/trace/trace_sched_wakeup.c +60 -4
@@ -27,6 +27,8 @@ static int wakeup_cpu;static int wakeup_current_cpu;static unsigned wakeup_prio =-1 ;static int wakeup_rt; +static int wakeup_dl; +static int tracing_dl =0 ;static arch_spinlock_t wakeup_lock = (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;
@@ -437,6 +439,7 @@ static void __wakeup_reset(struct trace_array *tr) { wakeup_cpu =-1 ; wakeup_prio =-1 ; + tracing_dl =0 ;if (wakeup_task)put_task_struct (wakeup_task);
@@ -472,9 +475,17 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)tracing_record_cmdline (p);tracing_record_cmdline (current); -if ((wakeup_rt && !rt_task (p)) || - p->prio >= wakeup_prio || - p->prio >= current->prio) +/* +* Semantic is like this: +* - wakeup tracer handles all tasks in the system, independently +* from their scheduling class; +* - wakeup_rt tracer handles tasks belonging to sched_dl and +* sched_rt class; +* - wakeup_dl handles tasks belonging to sched_dl class only. +*/ +if (tracing_dl || (wakeup_dl && !dl_task (p)) || + (wakeup_rt && !dl_task (p) && !rt_task (p)) || + (!dl_task (p) && (p->prio >= wakeup_prio || p->prio >= current->prio)))return ; pc =preempt_count ();
@@ -486,7 +497,8 @@ probe_wakeup(void *ignore, struct task_struct *p, int success)arch_spin_lock (&wakeup_lock);/* check for races. */ -if (!tracer_enabled || p->prio >= wakeup_prio) +if (!tracer_enabled || tracing_dl || + (!dl_task (p) && p->prio >= wakeup_prio)) goto out_locked;/* reset the trace */
@@ -496,6 +508,15 @@ probe_wakeup(void *ignore, struct task_struct *p, int success) wakeup_current_cpu = wakeup_cpu; wakeup_prio = p->prio; +/* +* Once you start tracing a -deadline task, don't bother tracing +* another task until the first one wakes up. +*/ +if (dl_task (p)) + tracing_dl =1 ; +else + tracing_dl =0 ; + wakeup_task = p;get_task_struct (wakeup_task);
@@ -597,16 +618,25 @@ static int __wakeup_tracer_init(struct trace_array *tr)static int wakeup_tracer_init (struct trace_array *tr) { + wakeup_dl =0 ; wakeup_rt =0 ;return __wakeup_tracer_init (tr); }static int wakeup_rt_tracer_init (struct trace_array *tr) { + wakeup_dl =0 ; wakeup_rt =1 ;return __wakeup_tracer_init (tr); } +static int wakeup_dl_tracer_init (struct trace_array *tr) +{ + wakeup_dl =1 ; + wakeup_rt =0 ; +return __wakeup_tracer_init (tr); +} +static void wakeup_tracer_reset (struct trace_array *tr) {int lat_flag = save_flags &TRACE_ITER_LATENCY_FMT ;
@@ -674,6 +704,28 @@ static struct tracer wakeup_rt_tracer __read_mostly = .use_max_tr = true, }; +static struct tracer wakeup_dl_tracer__read_mostly = +{ + .name ="wakeup_dl" , + .init = wakeup_dl_tracer_init, + .reset = wakeup_tracer_reset, + .start = wakeup_tracer_start, + .stop = wakeup_tracer_stop, + .wait_pipe = poll_wait_pipe, + .print_max = true, + .print_header = wakeup_print_header, + .print_line = wakeup_print_line, + .flags = &tracer_flags, + .set_flag = wakeup_set_flag, + .flag_changed = wakeup_flag_changed, +#ifdef CONFIG_FTRACE_SELFTEST + .selftest = trace_selftest_startup_wakeup, +#endif + .open = wakeup_trace_open, + .close = wakeup_trace_close, + .use_max_tr = true, +}; +__init static int init_wakeup_tracer (void ) {int ret;
@@ -686,6 +738,10 @@ __init static int init_wakeup_tracer(void)if (ret)return ret; + ret =register_tracer (&wakeup_dl_tracer); +if (ret) +return ret; +return 0 ; }core_initcall (init_wakeup_tracer);
kernel/trace/trace_selftest.c +19 -14
@@ -1022,11 +1022,16 @@ trace_selftest_startup_nop(struct tracer *trace, struct trace_array *tr)#ifdef CONFIG_SCHED_TRACER static int trace_wakeup_test_thread (void *data) { -/* Make this a RT thread, doesn't need to be too high */ -static const struct sched_param param = { .sched_priority =5 }; +/* Make this a -deadline thread */ +static const struct sched_attr attr = { + .sched_policy =SCHED_DEADLINE , + .sched_runtime =100000ULL , + .sched_deadline =10000000ULL , + .sched_period =10000000ULL + };struct completion *x = data; -sched_setscheduler (current,SCHED_FIFO , ¶m); +sched_setattr (current, &attr);/* Make it know we have a new prio */ complete (x);
@@ -1040,8 +1045,8 @@ static int trace_wakeup_test_thread(void *data)/* we are awake, now wait to disappear */ while (!kthread_should_stop ()) {/* -* This is an RT task, do short sleeps to let -* others run. +* This will likely be the system top priority +* task, do short sleeps to let others run. */ msleep (100 ); }
@@ -1054,21 +1059,21 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr) {unsigned long save_max = tracing_max_latency;struct task_struct *p; -struct completion isrt; +struct completion is_ready;unsigned long count;int ret; -init_completion (&isrt); +init_completion (&is_ready); -/* create a high prio thread */ - p =kthread_run (trace_wakeup_test_thread, &isrt,"ftrace-test" ); +/* create a -deadline thread */ + p =kthread_run (trace_wakeup_test_thread, &is_ready,"ftrace-test" );if (IS_ERR (p)) {printk (KERN_CONT "Failed to create ftrace wakeup test thread " );return -1 ; } -/* make sure the thread is running at an RT prio */ -wait_for_completion (&isrt); +/* make sure the thread is running at -deadline policy */ +wait_for_completion (&is_ready);/* start the tracing */ ret =tracer_init (trace, tr);
@@ -1082,19 +1087,19 @@ trace_selftest_startup_wakeup(struct tracer *trace, struct trace_array *tr)while (p->on_rq) {/* -* Sleep to make sure the RT thread is asleep too. +* Sleep to make sure the -deadline thread is asleep too. * On virtual machines we can't rely on timings, * but we want to make sure this test still works. */ msleep (100 ); } -init_completion (&isrt); +init_completion (&is_ready);wake_up_process (p);/* Wait for the task to wake up */ -wait_for_completion (&isrt); +wait_for_completion (&is_ready);/* stop the tracing. */ tracing_stop ();
PATCH 9/14 rtmutex: turn the plist into an rb-tree.
In reply to Juri LelliFrom: Peter Zijlstra <[email protected]>
Turn the pi-chains from plist to rb-tree, in the rt_mutex code, and provide a proper comparison function for -deadline and -priority tasks.
This is done mainly because:
- classical prio field of the plist is just an int, which might not be enough for representing a deadline;
- manipulating such a list would become O(nr_deadline_tasks), which might be to much, as the number of -deadline task increases.
Therefore, an rb-tree is used, and tasks are queued in it according to the following logic:
- among two -priority (i.e., SCHED_BATCH/OTHER/RR/FIFO) tasks, the one with the higher (lower, actually!) prio wins;
- among a -priority and a -deadline task, the latter always wins;
- among two -deadline tasks, the one with the earliest deadline wins.
Queueing and dequeueing functions are changed accordingly, for both the list of a task's pi-waiters and the list of tasks blocked on a pi-lock.
--- include/linux/init_task.h | 10 +++ include/linux/rtmutex.h | 18 ++---- include/linux/sched.h | 4 +- kernel/fork.c | 3 +- kernel/futex.c | 2 + kernel/rtmutex-debug.c | 8 +-- kernel/rtmutex.c | 152 ++++++++++++++++++++++++++++++++++++--------- kernel/rtmutex_common.h | 22 +++---- kernel/sched/core.c | 4 -- 9 files changed, 157 insertions(+), 66 deletions(-)
include/linux/init_task.h +10 -0
@@ -11,6 +11,7 @@#include <linux/user_namespace.h> #include <linux/securebits.h> #include <linux/seqlock.h> +#include <linux/rbtree.h> #include <net/net_namespace.h> #include <linux/sched/rt.h>
@@ -154,6 +155,14 @@ extern struct task_group root_task_group;#define INIT_TASK_COMM "swapper" +#ifdef CONFIG_RT_MUTEXES +# define INIT_RT_MUTEXES (tsk) \ + .pi_waiters = RB_ROOT, \ + .pi_waiters_leftmost = NULL, +#else +# define INIT_RT_MUTEXES (tsk) +#endif +/* * INIT_TASK is used to set up the first task table, touch at * your own risk!. Base=0, limit=0x1fffff (=2MB)
@@ -221,6 +230,7 @@ extern struct task_group root_task_group; INIT_TRACE_RECURSION \ INIT_TASK_RCU_PREEMPT(tsk) \ INIT_CPUSET_SEQ \ + INIT_RT_MUTEXES(tsk) \ INIT_VTIME(tsk) \ }
include/linux/rtmutex.h +6 -12
@@ -13,7 +13,7 @@#define __LINUX_RT_MUTEX_H#include <linux/linkage.h> -#include <linux/plist.h> +#include <linux/rbtree.h> #include <linux/spinlock_types.h> extern int max_lock_depth;/* for sysctl */
@@ -22,12 +22,14 @@ extern int max_lock_depth; /* for sysctl */* The rt_mutex structure * * @wait_lock: spinlock to protect the structure -* @wait_list: pilist head to enqueue waiters in priority order +* @waiters: rbtree root to enqueue waiters in priority order +* @waiters_leftmost: top waiter * @owner: the mutex owner */ struct rt_mutex {raw_spinlock_t wait_lock; -struct plist_head wait_list; +struct rb_root waiters; +struct rb_node *waiters_leftmost;struct task_struct *owner;#ifdef CONFIG_DEBUG_RT_MUTEXES int save_state;
@@ -66,7 +68,7 @@ struct hrtimer_sleeper;#define __RT_MUTEX_INITIALIZER (mutexname) \ { .wait_lock = __RAW_SPIN_LOCK_UNLOCKED(mutexname.wait_lock) \ - , .wait_list = PLIST_HEAD_INIT(mutexname.wait_list) \ + , .waiters = RB_ROOT \ , .owner = NULL \ __DEBUG_RT_MUTEX_INITIALIZER(mutexname)}
@@ -98,12 +100,4 @@ extern int rt_mutex_trylock(struct rt_mutex *lock);extern void rt_mutex_unlock (struct rt_mutex *lock); -#ifdef CONFIG_RT_MUTEXES -# define INIT_RT_MUTEXES (tsk) \ - .pi_waiters = PLIST_HEAD_INIT(tsk.pi_waiters), \ - INIT_RT_MUTEX_DEBUG(tsk) -#else -# define INIT_RT_MUTEXES (tsk) -#endif -#endif
include/linux/sched.h +3 -1
@@ -16,6 +16,7 @@ struct sched_param {#include <linux/types.h> #include <linux/timex.h> #include <linux/jiffies.h> +#include <linux/plist.h> #include <linux/rbtree.h> #include <linux/thread_info.h> #include <linux/cpumask.h>
@@ -1340,7 +1341,8 @@ struct task_struct {#ifdef CONFIG_RT_MUTEXES /* PI waiters blocked on a rt_mutex held by this task */ -struct plist_head pi_waiters; +struct rb_root pi_waiters; +struct rb_node *pi_waiters_leftmost;/* Deadlock detection and priority inheritance handling */ struct rt_mutex_waiter *pi_blocked_on;#endif
kernel/fork.c +2 -1
@@ -1089,7 +1089,8 @@ static void rt_mutex_init_task(struct task_struct *p) {raw_spin_lock_init (&p->pi_lock);#ifdef CONFIG_RT_MUTEXES -plist_head_init (&p->pi_waiters); + p->pi_waiters =RB_ROOT ; + p->pi_waiters_leftmost =NULL ; p->pi_blocked_on =NULL ;#endif }
kernel/futex.c +2 -0
@@ -2315,6 +2315,8 @@ static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,* code while we sleep on uaddr. */ debug_rt_mutex_init_waiter (&rt_waiter); +RB_CLEAR_NODE (&rt_waiter.pi_tree_entry); +RB_CLEAR_NODE (&rt_waiter.tree_entry); rt_waiter.task =NULL ; ret =get_futex_key (uaddr2, flags &FLAGS_SHARED , &key2,VERIFY_WRITE );
kernel/rtmutex-debug.c +2 -6
@@ -24,7 +24,7 @@#include <linux/kallsyms.h> #include <linux/syscalls.h> #include <linux/interrupt.h> -#include <linux/plist.h> +#include <linux/rbtree.h> #include <linux/fs.h> #include <linux/debug_locks.h>
@@ -57,7 +57,7 @@ static void printk_lock(struct rt_mutex *lock, int print_owner)void rt_mutex_debug_task_free (struct task_struct *task) { -DEBUG_LOCKS_WARN_ON (!plist_head_empty (&task->pi_waiters)); +DEBUG_LOCKS_WARN_ON (!RB_EMPTY_ROOT (&task->pi_waiters));DEBUG_LOCKS_WARN_ON (task->pi_blocked_on); }
@@ -154,16 +154,12 @@ void debug_rt_mutex_proxy_unlock(struct rt_mutex *lock)void debug_rt_mutex_init_waiter (struct rt_mutex_waiter *waiter) {memset (waiter,0x11 ,sizeof (*waiter)); -plist_node_init (&waiter->list_entry,MAX_PRIO ); -plist_node_init (&waiter->pi_list_entry,MAX_PRIO ); waiter->deadlock_task_pid =NULL ; }void debug_rt_mutex_free_waiter (struct rt_mutex_waiter *waiter) {put_pid (waiter->deadlock_task_pid); -DEBUG_LOCKS_WARN_ON (!plist_node_empty (&waiter->list_entry)); -DEBUG_LOCKS_WARN_ON (!plist_node_empty (&waiter->pi_list_entry));memset (waiter,0x22 ,sizeof (*waiter)); }
kernel/rtmutex.c +121 -31
269 lines not on this page · raw message
kernel/rtmutex_common.h +11 -11
@@ -40,13 +40,13 @@ extern void schedule_rt_mutex_test(struct rt_mutex *lock);* This is the control structure for tasks blocked on a rt_mutex, * which is allocated on the kernel stack on of the blocked task. * -* @list_entry: pi node to enqueue into the mutex waiters list -* @pi_list_entry: pi node to enqueue into the mutex owner waiters list +* @tree_entry: pi node to enqueue into the mutex waiters tree +* @pi_tree_entry: pi node to enqueue into the mutex owner waiters tree * @task: task reference to the blocked task */ struct rt_mutex_waiter { -struct plist_node list_entry; -struct plist_node pi_list_entry; +struct rb_node tree_entry; +struct rb_node pi_tree_entry;struct task_struct *task;struct rt_mutex *lock;#ifdef CONFIG_DEBUG_RT_MUTEXES
@@ -57,11 +57,11 @@ struct rt_mutex_waiter { };/* -* Various helpers to access the waiters-plist: +* Various helpers to access the waiters-tree: */ static inline int rt_mutex_has_waiters (struct rt_mutex *lock) { -return !plist_head_empty (&lock->wait_list); +return !RB_EMPTY_ROOT (&lock->waiters); } static inline struct rt_mutex_waiter *
@@ -69,8 +69,8 @@ rt_mutex_top_waiter(struct rt_mutex *lock) {struct rt_mutex_waiter *w; - w = plist_first_entry(&lock->wait_list, struct rt_mutex_waiter, - list_entry); + w = rb_entry(lock->waiters_leftmost, struct rt_mutex_waiter, + tree_entry);BUG_ON (w->lock != lock);return w;
@@ -78,14 +78,14 @@ rt_mutex_top_waiter(struct rt_mutex *lock)static inline int task_has_pi_waiters (struct task_struct *p) { -return !plist_head_empty (&p->pi_waiters); +return !RB_EMPTY_ROOT (&p->pi_waiters); }static inline struct rt_mutex_waiter *task_top_pi_waiter (struct task_struct *p) { - return plist_first_entry(&p->pi_waiters, struct rt_mutex_waiter, - pi_list_entry); + return rb_entry(p->pi_waiters_leftmost, struct rt_mutex_waiter, + pi_tree_entry); }/*
kernel/sched/core.c +0 -4
@@ -6728,10 +6728,6 @@ void __init sched_init(void)INIT_HLIST_HEAD (&init_task.preempt_notifiers);#endif -#ifdef CONFIG_RT_MUTEXES -plist_head_init (&init_task.pi_waiters); -#endif -/* * The boot idle thread does lazy MMU switching as well: */
Signature · 1 line
1.7.9.5
Steven Rostedt
maintainerRe: [PATCH 09/14] rtmutex: turn the plist into an rb-tree.
Quoted 20 lines · On Thu, 7 Nov 2013 14:43:43 +0100 Juri Lelli <[email protected]> wrote:
From: Peter Zijlstra <[email protected]>
Turn the pi-chains from plist to rb-tree, in the
rt_mutexcode, and provide a proper comparison function for -deadline and -priority tasks.This is done mainly because:
- classical prio field of the plist is just an int, which might not be enough for representing a deadline;
- manipulating such a list would becomeO(nr_deadline_tasks), which might be to much, as the number of -deadline task increases.Therefore, an rb-tree is used, and tasks are queued in it according to the following logic:
- among two -priority (i.e., SCHED_BATCH/OTHER/RR/FIFO) tasks, the one with the higher (lower, actually!) prio wins;
- among a -priority and a -deadline task, the latter always wins;
- among two -deadline tasks, the one with the earliest deadline wins.
Quoted 3 lines
Queueing and dequeueing functions are changed accordingly, for both the list of a task's pi-waiters and the list of tasks blocked on a pi-lock.
It will be interesting to see if this affects performance of the -rt patch, as the pi lists are stressed much more.
Although this looks like it will remove that nasty hack in the -rt patch where the locks have to call "init_lists()" because plists are something not initialized easily on static variables.
Quoted 10 lines
kernel/rtmutex.c +2
-0@@ -91,10 +91,104 @@ static inline void mark_rt_mutex_waiters(struct rt_mutex *lock) }#endif +static inline int +rt_mutex_waiter_less(struct rt_mutex_waiter *left,
kernel/rtmutex.c static inline void mark_rt_mutex_waiters(struct rt_mutex *lock) · line 105
Quoted 10 lines
kernel/rtmutex.c +10
-0+ struct rt_mutex_waiter *right) +{ +if (left->task->prio < right->task->prio) +return 1 ; + +/* +* If both tasks are dl_task(), we check their deadlines. +*/ +if (dl_prio (left->task->prio) &&dl_prio (right->task->prio)) +return (left->task->dl.deadline < right->task->dl.deadline);
Hmm, actually you only need to check the left task if it has a dl_prio() or not. If it has a dl_prio, then the only way it could have not returned with a 1 from the first compare is if the right task also has a dl_prio().
Quoted 101 lines
kernel/rtmutex.c +85
-3+ +return 0 ; +} + +static void +rt_mutex_enqueue (struct rt_mutex *lock,struct rt_mutex_waiter *waiter) +{ +struct rb_node **link = &lock->waiters.rb_node; +struct rb_node *parent =NULL ; +struct rt_mutex_waiter *entry; +int leftmost =1 ; + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct rt_mutex_waiter, tree_entry); +if (rt_mutex_waiter_less (waiter, entry)) { + link = &parent->rb_left; + }else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + lock->waiters_leftmost = &waiter->tree_entry; + +rb_link_node (&waiter->tree_entry, parent, link); +rb_insert_color (&waiter->tree_entry, &lock->waiters); +} + +static void +rt_mutex_dequeue (struct rt_mutex *lock,struct rt_mutex_waiter *waiter) +{ +if (RB_EMPTY_NODE (&waiter->tree_entry)) +return ; + +if (lock->waiters_leftmost == &waiter->tree_entry) + lock->waiters_leftmost =rb_next (&waiter->tree_entry); + +rb_erase (&waiter->tree_entry, &lock->waiters); +RB_CLEAR_NODE (&waiter->tree_entry); +} + +static void +rt_mutex_enqueue_pi (struct task_struct *task,struct rt_mutex_waiter *waiter) +{ +struct rb_node **link = &task->pi_waiters.rb_node; +struct rb_node *parent =NULL ; +struct rt_mutex_waiter *entry; +int leftmost =1 ; + +while (*link) { + parent = *link; + entry = rb_entry(parent, struct rt_mutex_waiter, pi_tree_entry); +if (rt_mutex_waiter_less (waiter, entry)) { + link = &parent->rb_left; + }else { + link = &parent->rb_right; + leftmost =0 ; + } + } + +if (leftmost) + task->pi_waiters_leftmost = &waiter->pi_tree_entry; + +rb_link_node (&waiter->pi_tree_entry, parent, link); +rb_insert_color (&waiter->pi_tree_entry, &task->pi_waiters); +} + +static void +rt_mutex_dequeue_pi (struct task_struct *task,struct rt_mutex_waiter *waiter) +{ +if (RB_EMPTY_NODE (&waiter->pi_tree_entry)) +return ; + +if (task->pi_waiters_leftmost == &waiter->pi_tree_entry) + task->pi_waiters_leftmost =rb_next (&waiter->pi_tree_entry); + +rb_erase (&waiter->pi_tree_entry, &task->pi_waiters); +RB_CLEAR_NODE (&waiter->pi_tree_entry); +} +/* -* Calculate task priority from the waiter list priority +* Calculate task priority from the waiter tree priority * -* Return task->normal_prio when the waiter list is empty or when +* Return task->normal_prio when the waiter tree is empty or when * the waiter is not allowed to do priority boosting */ int rt_mutex_getprio (struct task_struct *task)@@ -102,7 +196,7 @@ int rt_mutex_getprio(struct task_struct *task)if (likely (!task_has_pi_waiters (task)))return task->normal_prio; -return min (task_top_pi_waiter (task)->pi_list_entry.prio, +return min (task_top_pi_waiter (task)->task->prio, task->normal_prio); }
kernel/rtmutex.c static int rt_mutex_adjust_prio_chain(struct task_struct *task, · line 330
Quoted 5 lines
kernel/rtmutex.c +1
-1@@ -233,7 +327,7 @@ static int rt_mutex_adjust_prio_chain(struct task_struct *task,* When deadlock detection is off then we check, if further * priority adjustment is necessary. */ -if (!detect_deadlock && waiter->list_entry.prio == task->prio) +if (!detect_deadlock && waiter->task->prio == task->prio)
This will always be true, as waiter->task == task.
kernel/rtmutex.c static int rt_mutex_adjust_prio_chain(struct task_struct *task, · line 352
Quoted 12 lines
kernel/rtmutex.c +2
-3goto out_unlock_pi; lock = waiter->lock;@@ -254,9 +348,9 @@ static int rt_mutex_adjust_prio_chain(struct task_struct *task, top_waiter =rt_mutex_top_waiter (lock);/* Requeue the waiter */ -plist_del (&waiter->list_entry, &lock->wait_list); - waiter->list_entry.prio = task->prio; -plist_add (&waiter->list_entry, &lock->wait_list); +rt_mutex_dequeue (lock, waiter); + waiter->task->prio = task->prio;
This is rather pointless, as waiter->task == task.
We need to add a prio to the rt_mutex_waiter structure, because we need a way to know if the prio changed or not. There's a reason we used the list_entry.prio and not the task prio.
Then you could substitute all the waiter->task->prio with just waiter->prio and that should also work.
-- Steve
Quoted 4 lines
kernel/rtmutex.c +1
-0+rt_mutex_enqueue (lock, waiter);/* Release the task */ raw_spin_unlock_irqrestore (&task->pi_lock, flags);
Steven Rostedt
maintainerPATCH 1/1 rtmutex: Fix compare of waiter prio and task prio
In reply to Juri LelliThe conversion of the rt_mutex from using plist to rbtree eliminated the use of the waiter->list_entry.prio, and instead used directly the waiter->task->prio.
The problem with this is that the priority inheritance code relies on the prio of the waiter being stored is different from the task's prio. The change didn't take into account waiter->task == task, which makes the compares of:
if (waiter->task->prio == task->prio)
rather pointless, since they will always be the same:
task->pi_blocked_on = waiter; waiter->task = task;
When deadlock detection is not being used (for internal users of rt_mutex_lock(); things other than futex), the code relies on the prio associated to the waiter being different than the prio associated to the task.
Another use case where this is critical, is when a task that is blocked on an rt_mutex has its priority increased by a separate task. Then the compare in rt_mutex_adjust_pi() (called from sched_setscheduler()), returns without doing anything. This is because it checks if the priority of the task is different than the priority of its waiter.
The simple solution is to add a prio member to the rt_mutex_waiter structure that associates the priority to the waiter that is separate from the task.
I created a test program that tests this case:
http://rostedt.homelinux.com/code/pi_mutex_test.c
(too big to include in a change log) I'll work on getting this test into other projects like LTP and the kernel (perf test?)
kernel/rtmutex.c +7 -6
@@ -197,7 +197,7 @@ int rt_mutex_getprio(struct task_structif (likely (!task_has_pi_waiters (task)))return task->normal_prio; -return min (task_top_pi_waiter (task)->task->prio, +return min (task_top_pi_waiter (task)->prio, task->normal_prio); }
@@ -336,7 +336,7 @@ static int rt_mutex_adjust_prio_chain(st* When deadlock detection is off then we check, if further * priority adjustment is necessary. */ -if (!detect_deadlock && waiter->task->prio == task->prio) +if (!detect_deadlock && waiter->prio == task->prio) goto out_unlock_pi; lock = waiter->lock;
@@ -358,7 +358,7 @@ static int rt_mutex_adjust_prio_chain(st/* Requeue the waiter */ rt_mutex_dequeue (lock, waiter); - waiter->task->prio = task->prio; + waiter->prio = task->prio;rt_mutex_enqueue (lock, waiter);/* Release the task */
@@ -456,7 +456,7 @@ static int try_to_take_rt_mutex(struct r* 3) it is top waiter */ if (rt_mutex_has_waiters (lock)) { -if (task->prio >=rt_mutex_top_waiter (lock)->task->prio) { +if (task->prio >=rt_mutex_top_waiter (lock)->prio) {if (!waiter || waiter !=rt_mutex_top_waiter (lock))return 0 ; }
@@ -516,7 +516,8 @@ static int task_blocks_on_rt_mutex(struc__rt_mutex_adjust_prio (task); waiter->task = task; waiter->lock = lock; - + waiter->prio = task->prio; +/* Get the top priority waiter on the lock */ if (rt_mutex_has_waiters (lock)) top_waiter =rt_mutex_top_waiter (lock);
@@ -661,7 +662,7 @@ void rt_mutex_adjust_pi(struct task_struraw_spin_lock_irqsave (&task->pi_lock, flags); waiter = task->pi_blocked_on; -if (!waiter || (waiter->task->prio == task->prio && +if (!waiter || (waiter->prio == task->prio && !dl_prio (task->prio))) {raw_spin_unlock_irqrestore (&task->pi_lock, flags);return ;
kernel/rtmutex_common.h +1 -0
@@ -54,6 +54,7 @@ struct rt_mutex_waiter {struct pid *deadlock_task_pid;struct rt_mutex *deadlock_lock;#endif +int prio; };/*
Re: [PATCH] rtmutex: Fix compare of waiter prio and task prio
Quoted 106 lines · Steven Rostedt
The conversion of the
rt_mutexfrom using plist to rbtree eliminated the use of thewaiter->list_entry.prio, and instead used directly thewaiter->task->prio.The problem with this is that the priority inheritance code relies on the prio of the waiter being stored is different from the task's prio. The change didn't take into account
waiter->task== task, which makes the compares of:if (waiter->task->prio == task->prio)rather pointless, since they will always be the same:
task->pi_blocked_on = waiter; waiter->task = task;When deadlock detection is not being used (for internal users of
rt_mutex_lock(); things other than futex), the code relies on the prio associated to the waiter being different than the prio associated to the task.Another use case where this is critical, is when a task that is blocked on an
rt_mutexhas its priority increased by a separate task. Then the compare inrt_mutex_adjust_pi()(called fromsched_setscheduler()), returns without doing anything. This is because it checks if the priority of the task is different than the priority of its waiter.The simple solution is to add a prio member to the
rt_mutex_waiterstructure that associates the priority to the waiter that is separate from the task.I created a test program that tests this case:
http://rostedt.homelinux.com/code/pi_mutex_test.c
(too big to include in a change log) I'll work on getting this test into other projects like LTP and the kernel (perf test?)
Signed-off-by Steven Rostedt <[email protected]>Index:
linux-rt.git/kernel/rtmutex.c
===================================================================kernel/rtmutex.c +7
-6@@ -197,7 +197,7 @@ int rt_mutex_getprio(struct task_structif (likely (!task_has_pi_waiters (task)))return task->normal_prio; -return min (task_top_pi_waiter (task)->task->prio, +return min (task_top_pi_waiter (task)->prio, task->normal_prio); }@@ -336,7 +336,7 @@ static int rt_mutex_adjust_prio_chain(st* When deadlock detection is off then we check, if further * priority adjustment is necessary. */ -if (!detect_deadlock && waiter->task->prio == task->prio) +if (!detect_deadlock && waiter->prio == task->prio) goto out_unlock_pi; lock = waiter->lock;@@ -358,7 +358,7 @@ static int rt_mutex_adjust_prio_chain(st/* Requeue the waiter */ rt_mutex_dequeue (lock, waiter); - waiter->task->prio = task->prio; + waiter->prio = task->prio;rt_mutex_enqueue (lock, waiter);/* Release the task */ @@ -456,7 +456,7 @@ static int try_to_take_rt_mutex(struct r* 3) it is top waiter */ if (rt_mutex_has_waiters (lock)) { -if (task->prio >=rt_mutex_top_waiter (lock)->task->prio) { +if (task->prio >=rt_mutex_top_waiter (lock)->prio) {if (!waiter || waiter !=rt_mutex_top_waiter (lock))return 0 ; }@@ -516,7 +516,8 @@ static int task_blocks_on_rt_mutex(struc__rt_mutex_adjust_prio (task); waiter->task = task; waiter->lock = lock; - + waiter->prio = task->prio; +/* Get the top priority waiter on the lock */ if (rt_mutex_has_waiters (lock)) top_waiter =rt_mutex_top_waiter (lock);@@ -661,7 +662,7 @@ void rt_mutex_adjust_pi(struct task_struraw_spin_lock_irqsave (&task->pi_lock, flags); waiter = task->pi_blocked_on; -if (!waiter || (waiter->task->prio == task->prio && +if (!waiter || (waiter->prio == task->prio && !dl_prio (task->prio))) {raw_spin_unlock_irqrestore (&task->pi_lock, flags);return ;Index:
linux-rt.git/kernel/rtmutex_common.h
===================================================================
kernel/rtmutex_common.h struct rt_mutex_waiter { · line 60
Quoted 7 lines
kernel/rtmutex_common.h +1
-0@@ -54,6 +54,7 @@ struct rt_mutex_waiter {struct pid *deadlock_task_pid;struct rt_mutex *deadlock_lock;#endif +int prio; };/*
Thanks! But, now that waiters have their own prio, don't we need to enqueue them using that?
Something like:
rtmutex: enqueue waiters by their prio
kernel/locking/rtmutex.c +6 -3
@@ -96,13 +96,16 @@ static inline int rt_mutex_waiter_less(struct rt_mutex_waiter *left, struct rt_mutex_waiter *right) { - if (left->task->prio < right->task->prio) + if (left->prio < right->prio) return 1;/* -* If both tasks are dl_task(), we check their deadlines. +* If both waiters have dl_prio(), we check the deadlines of the +* associated tasks. +* If left waiter has a dl_prio(), and we didn't return 1 above, +* then right waiter has a dl_prio() too. */ -if (dl_prio (left->task->prio) &&dl_prio (right->task->prio)) +if (dl_prio (left->prio))return (left->task->dl.deadline < right->task->dl.deadline);return 0 ;
Thanks,
- Juri