Skip to contentSkip to conversation

scheduler changes for v4.4

GIT PULL Ingo Molnar 1 participant 1 message Nov 3, 2015

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest sched-core-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched-core-for-linus

# HEAD: e73e85f0593832aa583b252f9a16cf90ed6d30fa sched: Don't scan all-offline ->cpus_allowed twice if !CONFIG_CPUSETS

The main changes in this cycle were:

- sched/fair load tracking fixes and cleanups. (Byungchul Park)

- Make load tracking frequency scale invariant. (Dietmar Eggemann)

- sched/deadline updates. (Juri Lelli)

- stop machine fixes, cleanups and enhancements for bugs triggered by CPU hotplug stress testing. (Oleg Nesterov)

- scheduler preemption code rework: remove PREEMPT_ACTIVE and related cleanups. (Peter Zijlstra)

- Rework the sched_info::run_delay code to fix races. (Peter Zijlstra)

- Optimize per entity utilization tracking. (Peter Zijlstra)

- ... misc other fixes, cleanups and smaller updates.

Thanks,

Ingo

------------------>
Byungchul Park (5):
sched/fair: Factor out the {at,de}taching of the per entity load {to,from} the runqueue sched/fair: Have task_move_group_fair() unconditionally add the entity load to the runqueue sched/fair: Have task_move_group_fair() also detach entity load from the old runqueue sched/fair: Fix switched_to_fair()'s per entity load tracking
sched/fair: Unify switched_{from,to}_fair() and task_move_group_fair()

Chuck Ebbert (1):
sched/x86: Fix typo in __switch_to() comments

Dietmar Eggemann (5):
sched/fair: Make load tracking frequency scale-invariant sched/fair: Make utilization tracking CPU scale-invariant sched/fair: Name utilization related data and functions consistently sched/fair: Get rid of scaling utilization by capacity_orig sched/fair: Defer calling scaling functions

Geliang Tang (1):
sched/core: Drop unlikely behind BUG_ON()

Henrik Austad (1):
sched/core: Make policy-testing consistent

Juergen Gross (1):
sched/core: Make 'sched_domain_topology' declaration static

Juri Lelli (3):
sched/deadline: Unify dl_time_before() usage
sched/deadline, locking/rtmutex: Fix open coded check in rt_mutex_waiter_less() sched/rt: Make (do_)balance_runtime() return void

Kirill Tkhai (1):
sched/core: Delete PF_EXITING checks from cpu_cgroup_exit() callback

Leo Yan (2):
sched/fair: Polish comments for LOAD_AVG_MAX sched/fair: Remove unnecessary parameter for group_classify()

Morten Rasmussen (2):
sched/fair: Convert arch_scale_cpu_capacity() from weak function to #define sched/fair: Initialize task load and utilization before placing task on rq

Oleg Nesterov (7):

      cpu/hotplug: Read_lock(tasklist_lock) doesn't need to disable irqs
      stop_machine: Ensure that a queued callback will be called before cpu_stop_park()
      stop_machine: Introduce __cpu_stop_queue_work() and cpu_stop_queue_two_works()
      stop_machine: Change cpu_stop_queue_two_works() to rely on stopper->enabled
      stop_machine: Kill smp_hotplug_thread->pre_unpark, introduce stop_machine_unpark()
      stop_machine: Kill cpu_stop_threads->setup() and cpu_stop_unpark()
      sched: Don't scan all-offline ->cpus_allowed twice if !CONFIG_CPUSETS

Peter Zijlstra (20):
sched/fair: Make the entity load aging on attaching tunable sched/core: Remove unused argument from sched_class::task_move_group sched/fair: Rename scale() to cap_scale()
sched/fair: Optimize __update_load_avg()
sched/fair: Optimize per entity utilization tracking sched/core: Simplify INIT_PREEMPT_COUNT
sched/core: Rework TASK_DEAD preemption exception
sched/core: Create preempt_count invariant
sched/core: Add preempt argument to __schedule()
sched/core: Fix trace_sched_switch()
sched/core: Stop setting PREEMPT_ACTIVE
sched/core: Robustify preemption leak checks
sched/core: Simplify preempt_count tests
sched/core, sched/x86: Kill thread_info::saved_preempt_count sched/core: Kill PREEMPT_ACTIVE
sched/core: More notrace annotations
sched/core: Add preempt_count invariant check
sched/core: Fix task and run queue sched_info::run_delay inconsistencies
sched: Start stopper early
sched: Move cpu_active() tests from stop_two_cpus() into migrate_swap_stop()

Rik van Riel (1):
sched/numa: Limit the amount of virtual memory scanned in task_numa_work()

Srikar Dronamraju (5):
sched/numa: Rename numabalancing_enabled to sched_numa_balancing sched/numa: Disable sched_numa_balancing on UMA systems sched/numa: Remove the NUMA sched_feature
sched/numa: Convert sched_numa_balancing to a static_branch sched/numa: Fix task_tick_fair() from disabling numa_balancing

Vincent Guittot (1):
sched/fair: Fix nohz.next_balance update

xiaofeng.yan (1):
sched/core: Remove a parameter in the migrate_task_rq() function

 arch/x86/include/asm/preempt.h     |   5 +-
 arch/x86/include/asm/thread_info.h |   2 -
 arch/x86/kernel/process_32.c       |   8 -
 arch/x86/kernel/process_64.c       |  10 +-
 include/asm-generic/preempt.h      |   2 +-
 include/linux/preempt.h            |  20 +-
 include/linux/sched.h              |  38 ++--
 include/linux/sched/deadline.h     |   5 +
 include/linux/smpboot.h            |   4 -
 include/linux/stop_machine.h       |   2 +
 include/trace/events/sched.h       |  22 +-
 kernel/cpu.c                       |  10 +-
 kernel/exit.c                      |   4 +-
 kernel/locking/rtmutex.c           |   3 +-
 kernel/sched/core.c                | 203 ++++++++++--------
 kernel/sched/cpudeadline.c         |   5 -
 kernel/sched/cpudeadline.h         |   1 +
 kernel/sched/fair.c                | 419 ++++++++++++++++++++-----------------
 kernel/sched/features.h            |  21 +-
 kernel/sched/rt.c                  |  22 +-
 kernel/sched/sched.h               |  55 ++---
 kernel/smpboot.c                   |   5 +-
 kernel/stop_machine.c              |  90 ++++----
 kernel/trace/ftrace.c              |   2 +-
 kernel/trace/trace_sched_switch.c  |   3 +-
 kernel/trace/trace_sched_wakeup.c  |   2 +-
 26 files changed, 493 insertions(+), 470 deletions(-)
arch/x86/include/asm/preempt.h +1 -4
index b12f81022a6b..01bcde84d3e4 100644
@@ -30,12 +30,9 @@ static __always_inline void preempt_count_set(int pc)
 /*
  * must be macros to avoid header recursion hell
  */
-#define init_task_preempt_count(p) do { \
-	task_thread_info(p)->saved_preempt_count = PREEMPT_DISABLED; \
-} while (0)
+#define init_task_preempt_count(p) do { } while (0)
 
 #define init_idle_preempt_count(p, cpu) do { \
-	task_thread_info(p)->saved_preempt_count = PREEMPT_ENABLED; \
 	per_cpu(__preempt_count, (cpu)) = PREEMPT_ENABLED; \
 } while (0)
 
arch/x86/include/asm/thread_info.h +0 -2
index 8afdc3e44247..809877e9030b 100644
@@ -57,7 +57,6 @@ struct thread_info {
 	__u32			flags;		/* low level flags */
 	__u32			status;		/* thread synchronous flags */
 	__u32			cpu;		/* current CPU */
-	int			saved_preempt_count;
 	mm_segment_t		addr_limit;
 	void __user		*sysenter_return;
 	unsigned int		sig_on_uaccess_error:1;
@@ -69,7 +68,6 @@ struct thread_info {
 	.task		= &tsk,			\
 	.flags		= 0,			\
 	.cpu		= 0,			\
-	.saved_preempt_count = INIT_PREEMPT_COUNT,	\
 	.addr_limit	= KERNEL_DS,		\
 }
 
arch/x86/kernel/process_32.c +0 -8
index 737527b40e5b..9f950917528b 100644
@@ -280,14 +280,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 		set_iopl_mask(next->iopl);
 
 	/*
-	 * If it were not for PREEMPT_ACTIVE we could guarantee that the
-	 * preempt_count of all tasks was equal here and this would not be
-	 * needed.
-	 */
-	task_thread_info(prev_p)->saved_preempt_count = this_cpu_read(__preempt_count);
-	this_cpu_write(__preempt_count, task_thread_info(next_p)->saved_preempt_count);
-
-	/*
 	 * Now maybe handle debug registers and/or IO bitmaps
 	 */
 	if (unlikely(task_thread_info(prev_p)->flags & _TIF_WORK_CTXSW_PREV ||
arch/x86/kernel/process_64.c +1 -9
index b35921a670b2..e835d263a33b 100644
@@ -332,7 +332,7 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	/*
 	 * Switch FS and GS.
 	 *
-	 * These are even more complicated than FS and GS: they have
+	 * These are even more complicated than DS and ES: they have
 	 * 64-bit bases are that controlled by arch_prctl.  Those bases
 	 * only differ from the values in the GDT or LDT if the selector
 	 * is 0.
@@ -401,14 +401,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p)
 	 */
 	this_cpu_write(current_task, next_p);
 
-	/*
-	 * If it were not for PREEMPT_ACTIVE we could guarantee that the
-	 * preempt_count of all tasks was equal here and this would not be
-	 * needed.
-	 */
-	task_thread_info(prev_p)->saved_preempt_count = this_cpu_read(__preempt_count);
-	this_cpu_write(__preempt_count, task_thread_info(next_p)->saved_preempt_count);
-
 	/* Reload esp0 and ss1.  This changes current_thread_info(). */
 	load_sp0(tss, next);
 
include/asm-generic/preempt.h +1 -1
index 0bec580a4885..5d8ffa3e6f8c 100644
@@ -24,7 +24,7 @@ static __always_inline void preempt_count_set(int pc)
  * must be macros to avoid header recursion hell
  */
 #define init_task_preempt_count(p) do { \
-	task_thread_info(p)->preempt_count = PREEMPT_DISABLED; \
+	task_thread_info(p)->preempt_count = FORK_PREEMPT_COUNT; \
 } while (0)
 
 #define init_idle_preempt_count(p, cpu) do { \
include/linux/preempt.h +1 -19
index bea8dd8ff5e0..75e4e30677f1 100644
@@ -26,7 +26,6 @@
  *         SOFTIRQ_MASK:	0x0000ff00
  *         HARDIRQ_MASK:	0x000f0000
  *             NMI_MASK:	0x00100000
- *       PREEMPT_ACTIVE:	0x00200000
  * PREEMPT_NEED_RESCHED:	0x80000000
  */
 #define PREEMPT_BITS	8
@@ -53,10 +52,6 @@
 
 #define SOFTIRQ_DISABLE_OFFSET	(2 * SOFTIRQ_OFFSET)
 
-#define PREEMPT_ACTIVE_BITS	1
-#define PREEMPT_ACTIVE_SHIFT	(NMI_SHIFT + NMI_BITS)
-#define PREEMPT_ACTIVE	(__IRQ_MASK(PREEMPT_ACTIVE_BITS) << PREEMPT_ACTIVE_SHIFT)
-
 /* We use the MSB mostly because its available */
 #define PREEMPT_NEED_RESCHED	0x80000000
 
@@ -126,8 +121,7 @@
  * Check whether we were atomic before we did preempt_disable():
  * (used by the scheduler)
  */
-#define in_atomic_preempt_off() \
-		((preempt_count() & ~PREEMPT_ACTIVE) != PREEMPT_DISABLE_OFFSET)
+#define in_atomic_preempt_off() (preempt_count() != PREEMPT_DISABLE_OFFSET)
 
 #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
 extern void preempt_count_add(int val);
@@ -146,18 +140,6 @@ extern void preempt_count_sub(int val);
 #define preempt_count_inc() preempt_count_add(1)
 #define preempt_count_dec() preempt_count_sub(1)
 
-#define preempt_active_enter() \
-do { \
-	preempt_count_add(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
-	barrier(); \
-} while (0)
-
-#define preempt_active_exit() \
-do { \
-	barrier(); \
-	preempt_count_sub(PREEMPT_ACTIVE + PREEMPT_DISABLE_OFFSET); \
-} while (0)
-
 #ifdef CONFIG_PREEMPT_COUNT
 
 #define preempt_disable() \
include/linux/sched.h +21 -17
index b7b9501b41af..23ca455d9582 100644
@@ -599,26 +599,32 @@ struct task_cputime_atomic {
 		.sum_exec_runtime = ATOMIC64_INIT(0),		\
 	}
 
-#ifdef CONFIG_PREEMPT_COUNT
-#define PREEMPT_DISABLED	(1 + PREEMPT_ENABLED)
-#else
-#define PREEMPT_DISABLED	PREEMPT_ENABLED
-#endif
+#define PREEMPT_DISABLED	(PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
+
+/*
+ * Disable preemption until the scheduler is running -- use an unconditional
+ * value so that it also works on !PREEMPT_COUNT kernels.
+ *
+ * Reset by start_kernel()->sched_init()->init_idle()->init_idle_preempt_count().
+ */
+#define INIT_PREEMPT_COUNT	PREEMPT_OFFSET
 
 /*
- * Disable preemption until the scheduler is running.
- * Reset by start_kernel()->sched_init()->init_idle().
+ * Initial preempt_count value; reflects the preempt_count schedule invariant
+ * which states that during context switches:
  *
- * We include PREEMPT_ACTIVE to avoid cond_resched() from working
- * before the scheduler is active -- see should_resched().
+ *    preempt_count() == 2*PREEMPT_DISABLE_OFFSET
+ *
+ * Note: PREEMPT_DISABLE_OFFSET is 0 for !PREEMPT_COUNT kernels.
+ * Note: See finish_task_switch().
  */
-#define INIT_PREEMPT_COUNT	(PREEMPT_DISABLED + PREEMPT_ACTIVE)
+#define FORK_PREEMPT_COUNT	(2*PREEMPT_DISABLE_OFFSET + PREEMPT_ENABLED)
 
 /**
  * struct thread_group_cputimer - thread group interval timer counts
  * @cputime_atomic:	atomic thread group interval timers.
  * @running:		non-zero when there are timers running and
- * 			@cputime receives updates.
+ *			@cputime receives updates.
  *
  * This structure contains the version of task_cputime, above, that is
  * used for thread group CPU timer calculations.
@@ -1139,8 +1145,6 @@ struct sched_domain_topology_level {
 #endif
 };
 
-extern struct sched_domain_topology_level *sched_domain_topology;
-
 extern void set_sched_topology(struct sched_domain_topology_level *tl);
 extern void wake_up_if_idle(int cpu);
 
@@ -1189,10 +1193,10 @@ struct load_weight {
 
 /*
  * The load_avg/util_avg accumulates an infinite geometric series.
- * 1) load_avg factors the amount of time that a sched_entity is
- * runnable on a rq into its weight. For cfs_rq, it is the aggregated
- * such weights of all runnable and blocked sched_entities.
- * 2) util_avg factors frequency scaling into the amount of time
+ * 1) load_avg factors frequency scaling into the amount of time that a
+ * sched_entity is runnable on a rq into its weight. For cfs_rq, it is the
+ * aggregated such weights of all runnable and blocked sched_entities.
+ * 2) util_avg factors frequency and cpu scaling into the amount of time
  * that a sched_entity is running on a CPU, in the range [0..SCHED_LOAD_SCALE].
  * For cfs_rq, it is the aggregated such times of all runnable and
  * blocked sched_entities.
include/linux/sched/deadline.h +5 -0
index 9d303b8847df..9089a2ae913d 100644
@@ -21,4 +21,9 @@ static inline int dl_task(struct task_struct *p)
 	return dl_prio(p->prio);
 }
 
+static inline bool dl_time_before(u64 a, u64 b)
+{
+	return (s64)(a - b) < 0;
+}
+
 #endif /* _SCHED_DEADLINE_H */
include/linux/smpboot.h +0 -4
index e6109a6cd8f6..12910cf19869 100644
@@ -24,9 +24,6 @@ struct smpboot_thread_data;
  *			parked (cpu offline)
  * @unpark:		Optional unpark function, called when the thread is
  *			unparked (cpu online)
- * @pre_unpark:		Optional unpark function, called before the thread is
- *			unparked (cpu online). This is not guaranteed to be
- *			called on the target cpu of the thread. Careful!
  * @cpumask:		Internal state.  To update which threads are unparked,
  *			call smpboot_update_cpumask_percpu_thread().
  * @selfparking:	Thread is not parked by the park function.
@@ -42,7 +39,6 @@ struct smp_hotplug_thread {
 	void				(*cleanup)(unsigned int cpu, bool online);
 	void				(*park)(unsigned int cpu);
 	void				(*unpark)(unsigned int cpu);
-	void				(*pre_unpark)(unsigned int cpu);
 	cpumask_var_t			cpumask;
 	bool				selfparking;
 	const char			*thread_comm;
include/linux/stop_machine.h +2 -0
index 414d924318ce..0adedca24c5b 100644
@@ -33,6 +33,8 @@ void stop_one_cpu_nowait(unsigned int cpu, cpu_stop_fn_t fn, void *arg,
 			 struct cpu_stop_work *work_buf);
 int stop_cpus(const struct cpumask *cpumask, cpu_stop_fn_t fn, void *arg);
 int try_stop_cpus(const struct cpumask *cpumask, cpu_stop_fn_t fn, void *arg);
+void stop_machine_park(int cpu);
+void stop_machine_unpark(int cpu);
 
 #else	/* CONFIG_SMP */
 
include/trace/events/sched.h +9 -13
index 539d6bc3216a..9b90c57517a9 100644
@@ -104,22 +104,17 @@ DEFINE_EVENT(sched_wakeup_template, sched_wakeup_new,
 	     TP_ARGS(p));
 
 #ifdef CREATE_TRACE_POINTS
-static inline long __trace_sched_switch_state(struct task_struct *p)
+static inline long __trace_sched_switch_state(bool preempt, struct task_struct *p)
 {
-	long state = p->state;
-
-#ifdef CONFIG_PREEMPT
 #ifdef CONFIG_SCHED_DEBUG
 	BUG_ON(p != current);
 #endif /* CONFIG_SCHED_DEBUG */
+
 	/*
-	 * For all intents and purposes a preempted task is a running task.
+	 * Preemption ignores task state, therefore preempted tasks are always
+	 * RUNNING (we will not have dequeued if state != RUNNING).
 	 */
-	if (preempt_count() & PREEMPT_ACTIVE)
-		state = TASK_RUNNING | TASK_STATE_MAX;
-#endif /* CONFIG_PREEMPT */
-
-	return state;
+	return preempt ? TASK_RUNNING | TASK_STATE_MAX : p->state;
 }
 #endif /* CREATE_TRACE_POINTS */
 
@@ -128,10 +123,11 @@ static inline long __trace_sched_switch_state(struct task_struct *p)
  */
 TRACE_EVENT(sched_switch,
 
-	TP_PROTO(struct task_struct *prev,
+	TP_PROTO(bool preempt,
+		 struct task_struct *prev,
 		 struct task_struct *next),
 
-	TP_ARGS(prev, next),
+	TP_ARGS(preempt, prev, next),
 
 	TP_STRUCT__entry(
 		__array(	char,	prev_comm,	TASK_COMM_LEN	)
@@ -147,7 +143,7 @@ TRACE_EVENT(sched_switch,
 		memcpy(__entry->next_comm, next->comm, TASK_COMM_LEN);
 		__entry->prev_pid	= prev->pid;
 		__entry->prev_prio	= prev->prio;
-		__entry->prev_state	= __trace_sched_switch_state(prev);
+		__entry->prev_state	= __trace_sched_switch_state(preempt, prev);
 		memcpy(__entry->prev_comm, prev->comm, TASK_COMM_LEN);
 		__entry->next_pid	= next->pid;
 		__entry->next_prio	= next->prio;
kernel/cpu.c +5 -5
index 82cf9dff4295..c85df2775b73 100644
@@ -304,8 +304,8 @@ static inline void check_for_tasks(int dead_cpu)
 {
 	struct task_struct *g, *p;
 
-	read_lock_irq(&tasklist_lock);
-	do_each_thread(g, p) {
+	read_lock(&tasklist_lock);
+	for_each_process_thread(g, p) {
 		if (!p->on_rq)
 			continue;
 		/*
@@ -320,8 +320,8 @@ static inline void check_for_tasks(int dead_cpu)
 
 		pr_warn("Task %s (pid=%d) is on cpu %d (state=%ld, flags=%x)\n",
 			p->comm, task_pid_nr(p), dead_cpu, p->state, p->flags);
-	} while_each_thread(g, p);
-	read_unlock_irq(&tasklist_lock);
+	}
+	read_unlock(&tasklist_lock);
 }
 
 struct take_cpu_down_param {
@@ -344,7 +344,7 @@ static int take_cpu_down(void *_param)
 	/* Give up timekeeping duties */
 	tick_handover_do_timer();
 	/* Park the stopper thread */
-	kthread_park(current);
+	stop_machine_park((long)param->hcpu);
 	return 0;
 }
 
kernel/exit.c +3 -1
index ea95ee1b5ef7..443677c8efe6 100644
@@ -706,10 +706,12 @@ void do_exit(long code)
 	smp_mb();
 	raw_spin_unlock_wait(&tsk->pi_lock);
 
-	if (unlikely(in_atomic()))
+	if (unlikely(in_atomic())) {
 		pr_info("note: %s[%d] exited with preempt_count %d\n",
 			current->comm, task_pid_nr(current),
 			preempt_count());
+		preempt_count_set(PREEMPT_ENABLED);
+	}
 
 	/* sync mm's RSS info before statistics gathering */
 	if (tsk->mm)
kernel/locking/rtmutex.c +2 -1
index 7781d801212f..35e9bfcc6ad9 100644
@@ -158,7 +158,8 @@ rt_mutex_waiter_less(struct rt_mutex_waiter *left,
 	 * then right waiter has a dl_prio() too.
 	 */
 	if (dl_prio(left->prio))
-		return (left->task->dl.deadline < right->task->dl.deadline);
+		return dl_time_before(left->task->dl.deadline,
+				      right->task->dl.deadline);
 
 	return 0;
 }
kernel/sched/core.c +115 -88

510 lines not on this page · raw message

kernel/sched/cpudeadline.c +0 -5
index c6acb07466bb..5a75b08cfd85 100644
@@ -31,11 +31,6 @@ static inline int right_child(int i)
 	return (i << 1) + 2;
 }
 
-static inline int dl_time_before(u64 a, u64 b)
-{
-	return (s64)(a - b) < 0;
-}
-
 static void cpudl_exchange(struct cpudl *cp, int a, int b)
 {
 	int cpu_a = cp->elements[a].cpu, cpu_b = cp->elements[b].cpu;
kernel/sched/cpudeadline.h +1 -0
index 1a0a6ef2fbe1..fcbdf83fed7e 100644
@@ -2,6 +2,7 @@
 #define _LINUX_CPUDL_H
 
 #include <linux/sched.h>
+#include <linux/sched/deadline.h>
 
 #define IDX_INVALID     -1
 
kernel/sched/fair.c +228 -191

727 lines not on this page · raw message

kernel/sched/features.h +1 -20
index 83a50e7ca533..69631fa46c2f 100644
@@ -36,11 +36,6 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true)
  */
 SCHED_FEAT(WAKEUP_PREEMPTION, true)
 
-/*
- * Use arch dependent cpu capacity functions
- */
-SCHED_FEAT(ARCH_CAPACITY, true)
-
 SCHED_FEAT(HRTICK, false)
 SCHED_FEAT(DOUBLE_TICK, false)
 SCHED_FEAT(LB_BIAS, true)
@@ -72,19 +67,5 @@ SCHED_FEAT(RT_PUSH_IPI, true)
 SCHED_FEAT(FORCE_SD_OVERLAP, false)
 SCHED_FEAT(RT_RUNTIME_SHARE, true)
 SCHED_FEAT(LB_MIN, false)
+SCHED_FEAT(ATTACH_AGE_LOAD, true)
 
-/*
- * Apply the automatic NUMA scheduling policy. Enabled automatically
- * at runtime if running on a NUMA machine. Can be controlled via
- * numa_balancing=
- */
-#ifdef CONFIG_NUMA_BALANCING
-
-/*
- * NUMA will favor moving tasks towards nodes where a higher number of
- * hinting faults are recorded during active load balancing. It will
- * resist moving tasks towards nodes where a lower number of hinting
- * faults have been recorded.
- */
-SCHED_FEAT(NUMA,	true)
-#endif
kernel/sched/rt.c +6 -16
index d2ea59364a1c..e3cc16312046 100644
@@ -635,11 +635,11 @@ bool sched_rt_bandwidth_account(struct rt_rq *rt_rq)
 /*
  * We ran out of runtime, see if we can borrow some from our neighbours.
  */
-static int do_balance_runtime(struct rt_rq *rt_rq)
+static void do_balance_runtime(struct rt_rq *rt_rq)
 {
 	struct rt_bandwidth *rt_b = sched_rt_bandwidth(rt_rq);
 	struct root_domain *rd = rq_of_rt_rq(rt_rq)->rd;
-	int i, weight, more = 0;
+	int i, weight;
 	u64 rt_period;
 
 	weight = cpumask_weight(rd->span);
@@ -673,7 +673,6 @@ static int do_balance_runtime(struct rt_rq *rt_rq)
 				diff = rt_period - rt_rq->rt_runtime;
 			iter->rt_runtime -= diff;
 			rt_rq->rt_runtime += diff;
-			more = 1;
 			if (rt_rq->rt_runtime == rt_period) {
 				raw_spin_unlock(&iter->rt_runtime_lock);
 				break;
@@ -683,8 +682,6 @@ static int do_balance_runtime(struct rt_rq *rt_rq)
 		raw_spin_unlock(&iter->rt_runtime_lock);
 	}
 	raw_spin_unlock(&rt_b->rt_runtime_lock);
-
-	return more;
 }
 
 /*
@@ -796,26 +793,19 @@ static void __enable_runtime(struct rq *rq)
 	}
 }
 
-static int balance_runtime(struct rt_rq *rt_rq)
+static void balance_runtime(struct rt_rq *rt_rq)
 {
-	int more = 0;
-
 	if (!sched_feat(RT_RUNTIME_SHARE))
-		return more;
+		return;
 
 	if (rt_rq->rt_time > rt_rq->rt_runtime) {
 		raw_spin_unlock(&rt_rq->rt_runtime_lock);
-		more = do_balance_runtime(rt_rq);
+		do_balance_runtime(rt_rq);
 		raw_spin_lock(&rt_rq->rt_runtime_lock);
 	}
-
-	return more;
 }
 #else /* !CONFIG_SMP */
-static inline int balance_runtime(struct rt_rq *rt_rq)
-{
-	return 0;
-}
+static inline void balance_runtime(struct rt_rq *rt_rq) {}
 #endif /* CONFIG_SMP */
 
 static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)
kernel/sched/sched.h +31 -24

107 lines not on this page · raw message

kernel/smpboot.c +2 -3
index a818cbc73e14..d264f59bff56 100644
@@ -222,9 +222,8 @@ static void smpboot_unpark_thread(struct smp_hotplug_thread *ht, unsigned int cp
 {
 	struct task_struct *tsk = *per_cpu_ptr(ht->store, cpu);
 
-	if (ht->pre_unpark)
-		ht->pre_unpark(cpu);
-	kthread_unpark(tsk);
+	if (!ht->selfparking)
+		kthread_unpark(tsk);
 }
 
 void smpboot_unpark_threads(unsigned int cpu)
kernel/stop_machine.c +54 -36

152 lines not on this page · raw message

kernel/trace/ftrace.c +1 -1
index b0623ac785a2..00611e95a8ee 100644
@@ -5697,7 +5697,7 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list)
 }
 
 static void
-ftrace_graph_probe_sched_switch(void *ignore,
+ftrace_graph_probe_sched_switch(void *ignore, bool preempt,
 			struct task_struct *prev, struct task_struct *next)
 {
 	unsigned long long timestamp;
kernel/trace/trace_sched_switch.c +2 -1
index f270088e9929..4c896a0101bd 100644
@@ -16,7 +16,8 @@ static int			sched_ref;
 static DEFINE_MUTEX(sched_register_mutex);
 
 static void
-probe_sched_switch(void *ignore, struct task_struct *prev, struct task_struct *next)
+probe_sched_switch(void *ignore, bool preempt,
+		   struct task_struct *prev, struct task_struct *next)
 {
 	if (unlikely(!sched_ref))
 		return;
kernel/trace/trace_sched_wakeup.c +1 -1
index 12cbe77b4136..4bcfbac289ff 100644
@@ -420,7 +420,7 @@ tracing_sched_wakeup_trace(struct trace_array *tr,
 }
 
 static void notrace
-probe_wakeup_sched_switch(void *ignore,
+probe_wakeup_sched_switch(void *ignore, bool preempt,
 			  struct task_struct *prev, struct task_struct *next)
 {
 	struct trace_array_cpu *data;