Skip to contentSkip to conversation

Core scheduling (v9)

PATCH In review Joel Fernandes (Google) 12 participants 150 messages Nov 17, 2020 – Jan 15, 2021 Page 1 of 2

Subsystems (19)
lore
Share & tools
lore
#0 lore raw

cover Core scheduling (v9)

Core-Scheduling
===============
Enclosed is series v9 of core scheduling.
v9 is rebased on tip/master (fe4adf6f92c4 ("Merge branch 'irq/core'")).. I hope that this version is acceptable to be merged (pending any new review comments that arise) as the main issues in the past are all resolved:
1. Vruntime comparison.
2. Documentation updates.
3. CGroup and per-task interface developed by Google and Oracle.
4. Hotplug fixes.
Almost all patches also have Reviewed-by or Acked-by tag. See below for full list of changes in v9.

Introduction of feature
=======================
Core scheduling is a feature that allows only trusted tasks to run concurrently on cpus sharing compute resources (eg: hyperthreads on a core). The goal is to mitigate the core-level side-channel attacks without requiring to disable SMT (which has a significant impact on performance in some situations). Core scheduling (as of v7) mitigates user-space to user-space attacks and user to kernel attack when one of the siblings enters the kernel via interrupts or system call.

By default, the feature doesn't change any of the current scheduler behavior. The user decides which tasks can run simultaneously on the same core (for now by having them in the same tagged cgroup). When a tag is enabled in a cgroup and a task from that cgroup is running on a hardware thread, the scheduler ensures that only idle or trusted tasks run on the other sibling(s). Besides security concerns, this feature can also be beneficial for RT and performance applications where we want to control how tasks make use of SMT dynamically.

Both a CGroup and Per-task interface via prctl(2) are provided for configuring core sharing. More details are provided in documentation patch. Kselftests are provided to verify the correctness/rules of the interface.

Testing
=======
ChromeOS testing shows 300% improvement in keypress latency on a Google docs key press with Google hangout test (the maximum latency drops from 150ms to 50ms for keypresses).

Julien: TPCC tests showed improvements with core-scheduling as below. With kernel protection enabled, it does not show any regression. Possibly ASI will improve the performance for those who choose kernel protection (can be toggled through sched_core_protect_kernel sysctl).
average stdev diff
baseline (SMT on) 1197.272 44.78312824
core sched ( kernel protect) 412.9895 45.42734343 -65.51%
core sched (no kernel protect) 686.6515 71.77756931 -42.65%
nosmt 408.667 39.39042872 -65.87%
(Note these results are from v8).

Vineeth tested sysbench and does not see any regressions.
Hong and Aubrey tested v9 and see results similar to v8. There is a known issue with uperf that does regress. This appears to be because of ksoftirq heavily contending with other tasks on the core. The consensus is this can be improved in the future.

Changes in v9
=============
- Note that the vruntime snapshot change is written in 2 patches to show the progression of the idea and prevent merge conflicts:
sched/fair: Snapshot the min_vruntime of CPUs on force idle
sched: Improve snapshotting of min_vruntime for CGroups
Same with the RT priority inversion change:
sched: Fix priority inversion of cookied task with sibling
sched: Improve snapshotting of min_vruntime for CGroups
- Disable coresched on certain AMD HW.

Changes in v8
=============
- New interface/API implementation
- Joel
- Revised kernel protection patch
- Joel
- Revised Hotplug fixes
- Joel
- Minor bug fixes and address review comments
- Vineeth

Changes in v7
=============
- Kernel protection from untrusted usermode tasks
- Joel, Vineeth
- Fix for hotplug crashes and hangs
- Joel, Vineeth

Changes in v6
=============
- Documentation
- Joel
- Pause siblings on entering nmi/irq/softirq
- Joel, Vineeth
- Fix for RCU crash
- Joel
- Fix for a crash in pick_next_task
- Yu Chen, Vineeth
- Minor re-write of core-wide vruntime comparison
- Aaron Lu
- Cleanup: Address Review comments
- Cleanup: Remove hotplug support (for now)
- Build fixes: 32 bit, SMT=n, AUTOGROUP=n etc
- Joel, Vineeth

Changes in v5
=============
- Fixes for cgroup/process tagging during corner cases like cgroup destroy, task moving across cgroups etc
- Tim Chen
- Coresched aware task migrations
- Aubrey Li
- Other minor stability fixes.

Changes in v4
=============
- Implement a core wide min_vruntime for vruntime comparison of tasks across cpus in a core.
- Aaron Lu
- Fixes a typo bug in setting the forced_idle cpu.
- Aaron Lu

Changes in v3
=============
- Fixes the issue of sibling picking up an incompatible task
- Aaron Lu
- Vineeth Pillai
- Julien Desfossez
- Fixes the issue of starving threads due to forced idle
- Peter Zijlstra
- Fixes the refcounting issue when deleting a cgroup with tag
- Julien Desfossez
- Fixes a crash during cpu offline/online with coresched enabled
- Vineeth Pillai
- Fixes a comparison logic issue in sched_core_find
- Aaron Lu

Changes in v2
=============
- Fixes for couple of NULL pointer dereference crashes
- Subhra Mazumdar
- Tim Chen
- Improves priority comparison logic for process in different cpus
- Peter Zijlstra
- Aaron Lu
- Fixes a hard lockup in rq locking
- Vineeth Pillai
- Julien Desfossez
- Fixes a performance issue seen on IO heavy workloads
- Vineeth Pillai
- Julien Desfossez
- Fix for 32bit build
- Aubrey Li

Future work
===========
- Load balancing/Migration fixes for core scheduling. With v6, Load balancing is partially coresched aware, but has some issues w.r.t process/taskgroup weights:
https://lwn.net/ml/linux-kernel/20200225034438.GA617271@z...

Aubrey Li (1):
sched: migration changes for core scheduling

Joel Fernandes (Google) (16):
sched/fair: Snapshot the min_vruntime of CPUs on force idle
sched: Enqueue task into core queue only after vruntime is updated
sched: Simplify the core pick loop for optimized case
sched: Improve snapshotting of min_vruntime for CGroups arch/x86: Add a new TIF flag for untrusted tasks kernel/entry: Add support for core-wide protection of kernel-mode entry/idle: Enter and exit kernel protection during idle entry and exit
sched: Split the cookie and setup per-task cookie on fork
sched: Add a per-thread core scheduling interface
sched: Release references to the per-task cookie on exit sched/debug: Add CGroup node for printing group cookie if SCHED_DEBUG
kselftest: Add tests for core-sched interface
sched: Move core-scheduler interfacing code to a new file
Documentation: Add core scheduling documentation
sched: Add a coresched command line option
sched: Debug bits...

Josh Don (2):
sched: Refactor core cookie into struct
sched: Add a second-level tag for nested CGroup usecase

Peter Zijlstra (11):
sched: Wrap rq::lock access
sched: Introduce sched_class::pick_task() sched/fair: Fix pick_task_fair crashes due to empty rbtree
sched: Core-wide rq->lock
sched/fair: Add a few assertions
sched: Basic tracking of matching tasks
sched: Add core wide task selection and scheduling.
sched: Fix priority inversion of cookied task with sibling
sched: Trivial forced-newidle balancer
irq_work: Cleanup
sched: CGroup tagging interface for core scheduling

Vineeth Pillai (2):
sched/fair: Fix forced idle sibling starvation corner case entry/kvm: Protect the kernel when entering from guest

.../admin-guide/hw-vuln/core-scheduling.rst   |  330 +++++
Documentation/admin-guide/hw-vuln/index.rst   |    1 +
.../admin-guide/kernel-parameters.txt         |   25 +
arch/x86/include/asm/thread_info.h            |    2 +
arch/x86/kernel/cpu/bugs.c                    |   19 +
arch/x86/kvm/x86.c                            |    2 +
drivers/gpu/drm/i915/i915_request.c           |    4 +-
include/linux/cpu.h                           |    1 +
include/linux/entry-common.h                  |   30 +-
include/linux/entry-kvm.h                     |   12 +
include/linux/irq_work.h                      |   33 +-
include/linux/irqflags.h                      |    4 +-
include/linux/sched.h                         |   28 +-
include/linux/sched/smt.h                     |    4 +
include/uapi/linux/prctl.h                    |    3 +
kernel/Kconfig.preempt                        |    5 +
kernel/bpf/stackmap.c                         |    2 +-
kernel/cpu.c                                  |   43 +
kernel/entry/common.c                         |   28 +-
kernel/entry/kvm.c                            |   33 +
kernel/fork.c                                 |    1 +
kernel/irq_work.c                             |   18 +-
kernel/printk/printk.c                        |    6 +-
kernel/rcu/tree.c                             |    3 +-
kernel/sched/Makefile                         |    1 +
kernel/sched/core.c                           | 1278 +++++++++++++++--
kernel/sched/coretag.c                        |  819 +++++++++++
kernel/sched/cpuacct.c                        |   12 +-
kernel/sched/deadline.c                       |   38 +-
kernel/sched/debug.c                          |   12 +-
kernel/sched/fair.c                           |  313 +++-
kernel/sched/idle.c                           |   24 +-
kernel/sched/pelt.h                           |    2 +-
kernel/sched/rt.c                             |   31 +-
kernel/sched/sched.h                          |  315 +++-
kernel/sched/stop_task.c                      |   14 +-
kernel/sched/topology.c                       |    4 +-
kernel/sys.c                                  |    3 +
kernel/time/tick-sched.c                      |    6 +-
kernel/trace/bpf_trace.c                      |    2 +-
tools/include/uapi/linux/prctl.h              |    3 +
tools/testing/selftests/sched/.gitignore      |    1 +
tools/testing/selftests/sched/Makefile        |   14 +
tools/testing/selftests/sched/config          |    1 +
.../testing/selftests/sched/test_coresched.c  |  818 +++++++++++
45 files changed, 4033 insertions(+), 315 deletions(-)

create mode 100644 Documentation/admin-guide/hw-vuln/core-scheduling.rst create mode 100644 kernel/sched/coretag.c
create mode 100644 tools/testing/selftests/sched/.gitignore create mode 100644 tools/testing/selftests/sched/Makefile create mode 100644 tools/testing/selftests/sched/config create mode 100644 tools/testing/selftests/sched/test_coresched.c

--
2.29.2.299.gdc1121823c-goog

#1 lore raw

PATCH 1/32 sched: Wrap rq::lock access

From: Peter Zijlstra <[email protected]>

In preparation of playing games with rq->lock, abstract the thing using an accessor.

---
 kernel/sched/core.c     |  68 ++++++++++++-------------
 kernel/sched/cpuacct.c  |  12 ++---
 kernel/sched/deadline.c |  22 ++++----
 kernel/sched/debug.c    |   4 +-
 kernel/sched/fair.c     |  38 +++++++-------
 kernel/sched/idle.c     |   4 +-
 kernel/sched/pelt.h     |   2 +-
 kernel/sched/rt.c       |  16 +++---
 kernel/sched/sched.h    | 108 +++++++++++++++++++++-------------------
 kernel/sched/topology.c |   4 +-
 10 files changed, 141 insertions(+), 137 deletions(-)
kernel/sched/core.c +34 -34
index a6aaf9fb3400..db5cc05a68bc 100644
@@ -186,12 +186,12 @@ struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
 
 	for (;;) {
 		rq = task_rq(p);
-		raw_spin_lock(&rq->lock);
+		raw_spin_lock(rq_lockp(rq));
 		if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) {
 			rq_pin_lock(rq, rf);
 			return rq;
 		}
-		raw_spin_unlock(&rq->lock);
+		raw_spin_unlock(rq_lockp(rq));
 
 		while (unlikely(task_on_rq_migrating(p)))
 			cpu_relax();
@@ -210,7 +210,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
 	for (;;) {
 		raw_spin_lock_irqsave(&p->pi_lock, rf->flags);
 		rq = task_rq(p);
-		raw_spin_lock(&rq->lock);
+		raw_spin_lock(rq_lockp(rq));
 		/*
 		 *	move_queued_task()		task_rq_lock()
 		 *
@@ -232,7 +232,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
 			rq_pin_lock(rq, rf);
 			return rq;
 		}
-		raw_spin_unlock(&rq->lock);
+		raw_spin_unlock(rq_lockp(rq));
 		raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
 
 		while (unlikely(task_on_rq_migrating(p)))
@@ -302,7 +302,7 @@ void update_rq_clock(struct rq *rq)
 {
 	s64 delta;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	if (rq->clock_update_flags & RQCF_ACT_SKIP)
 		return;
@@ -611,7 +611,7 @@ void resched_curr(struct rq *rq)
 	struct task_struct *curr = rq->curr;
 	int cpu;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	if (test_tsk_need_resched(curr))
 		return;
@@ -635,10 +635,10 @@ void resched_cpu(int cpu)
 	struct rq *rq = cpu_rq(cpu);
 	unsigned long flags;
 
-	raw_spin_lock_irqsave(&rq->lock, flags);
+	raw_spin_lock_irqsave(rq_lockp(rq), flags);
 	if (cpu_online(cpu) || cpu == smp_processor_id())
 		resched_curr(rq);
-	raw_spin_unlock_irqrestore(&rq->lock, flags);
+	raw_spin_unlock_irqrestore(rq_lockp(rq), flags);
 }
 
 #ifdef CONFIG_SMP
@@ -1137,7 +1137,7 @@ static inline void uclamp_rq_inc_id(struct rq *rq, struct task_struct *p,
 	struct uclamp_se *uc_se = &p->uclamp[clamp_id];
 	struct uclamp_bucket *bucket;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	/* Update task effective clamp */
 	p->uclamp[clamp_id] = uclamp_eff_get(p, clamp_id);
@@ -1177,7 +1177,7 @@ static inline void uclamp_rq_dec_id(struct rq *rq, struct task_struct *p,
 	unsigned int bkt_clamp;
 	unsigned int rq_clamp;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	/*
 	 * If sched_uclamp_used was enabled after task @p was enqueued,
@@ -1807,7 +1807,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)
 static struct rq *move_queued_task(struct rq *rq, struct rq_flags *rf,
 				   struct task_struct *p, int new_cpu)
 {
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	deactivate_task(rq, p, DEQUEUE_NOCLOCK);
 	set_task_cpu(p, new_cpu);
@@ -1973,7 +1973,7 @@ int push_cpu_stop(void *arg)
 	struct task_struct *p = arg;
 
 	raw_spin_lock_irq(&p->pi_lock);
-	raw_spin_lock(&rq->lock);
+	raw_spin_lock(rq_lockp(rq));
 
 	if (task_rq(p) != rq)
 		goto out_unlock;
@@ -2003,7 +2003,7 @@ int push_cpu_stop(void *arg)
 
 out_unlock:
 	rq->push_busy = false;
-	raw_spin_unlock(&rq->lock);
+	raw_spin_unlock(rq_lockp(rq));
 	raw_spin_unlock_irq(&p->pi_lock);
 
 	put_task_struct(p);
@@ -2056,7 +2056,7 @@ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32
 		 * Because __kthread_bind() calls this on blocked tasks without
 		 * holding rq->lock.
 		 */
-		lockdep_assert_held(&rq->lock);
+		lockdep_assert_held(rq_lockp(rq));
 		dequeue_task(rq, p, DEQUEUE_SAVE | DEQUEUE_NOCLOCK);
 	}
 	if (running)
@@ -2395,7 +2395,7 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)
 	 * task_rq_lock().
 	 */
 	WARN_ON_ONCE(debug_locks && !(lockdep_is_held(&p->pi_lock) ||
-				      lockdep_is_held(&task_rq(p)->lock)));
+				      lockdep_is_held(rq_lockp(task_rq(p)))));
 #endif
 	/*
 	 * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
@@ -2941,7 +2941,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
 {
 	int en_flags = ENQUEUE_WAKEUP | ENQUEUE_NOCLOCK;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	if (p->sched_contributes_to_load)
 		rq->nr_uninterruptible--;
@@ -3941,7 +3941,7 @@ static void do_balance_callbacks(struct rq *rq, struct callback_head *head)
 	void (*func)(struct rq *rq);
 	struct callback_head *next;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	while (head) {
 		func = (void (*)(struct rq *))head->func;
@@ -3957,7 +3957,7 @@ static inline struct callback_head *splice_balance_callbacks(struct rq *rq)
 {
 	struct callback_head *head = rq->balance_callback;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 	if (head) {
 		rq->balance_callback = NULL;
 		rq->balance_flags &= ~BALANCE_WORK;
@@ -3976,9 +3976,9 @@ static inline void balance_callbacks(struct rq *rq, struct callback_head *head)
 	unsigned long flags;
 
 	if (unlikely(head)) {
-		raw_spin_lock_irqsave(&rq->lock, flags);
+		raw_spin_lock_irqsave(rq_lockp(rq), flags);
 		do_balance_callbacks(rq, head);
-		raw_spin_unlock_irqrestore(&rq->lock, flags);
+		raw_spin_unlock_irqrestore(rq_lockp(rq), flags);
 	}
 }
 
@@ -4028,10 +4028,10 @@ prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf
 	 * do an early lockdep release here:
 	 */
 	rq_unpin_lock(rq, rf);
-	spin_release(&rq->lock.dep_map, _THIS_IP_);
+	spin_release(&rq_lockp(rq)->dep_map, _THIS_IP_);
 #ifdef CONFIG_DEBUG_SPINLOCK
 	/* this is a valid case when another task releases the spinlock */
-	rq->lock.owner = next;
+	rq_lockp(rq)->owner = next;
 #endif
 }
 
@@ -4042,9 +4042,9 @@ static inline void finish_lock_switch(struct rq *rq)
 	 * fix up the runqueue lock - which gets 'carried over' from
 	 * prev into current:
 	 */
-	spin_acquire(&rq->lock.dep_map, 0, 0, _THIS_IP_);
+	spin_acquire(&rq_lockp(rq)->dep_map, 0, 0, _THIS_IP_);
 	balance_switch(rq);
-	raw_spin_unlock_irq(&rq->lock);
+	raw_spin_unlock_irq(rq_lockp(rq));
 }
 
 /*
@@ -5023,7 +5023,7 @@ static void __sched notrace __schedule(bool preempt)
 
 		rq_unpin_lock(rq, &rf);
 		__balance_callbacks(rq);
-		raw_spin_unlock_irq(&rq->lock);
+		raw_spin_unlock_irq(rq_lockp(rq));
 	}
 }
 
@@ -5438,7 +5438,7 @@ void rt_mutex_setprio(struct task_struct *p, struct task_struct *pi_task)
 
 	rq_unpin_lock(rq, &rf);
 	__balance_callbacks(rq);
-	raw_spin_unlock(&rq->lock);
+	raw_spin_unlock(rq_lockp(rq));
 
 	preempt_enable();
 }
@@ -7020,7 +7020,7 @@ void init_idle(struct task_struct *idle, int cpu)
 	__sched_fork(0, idle);
 
 	raw_spin_lock_irqsave(&idle->pi_lock, flags);
-	raw_spin_lock(&rq->lock);
+	raw_spin_lock(rq_lockp(rq));
 
 	idle->state = TASK_RUNNING;
 	idle->se.exec_start = sched_clock();
@@ -7058,7 +7058,7 @@ void init_idle(struct task_struct *idle, int cpu)
 #ifdef CONFIG_SMP
 	idle->on_cpu = 1;
 #endif
-	raw_spin_unlock(&rq->lock);
+	raw_spin_unlock(rq_lockp(rq));
 	raw_spin_unlock_irqrestore(&idle->pi_lock, flags);
 
 	/* Set the preempt count _outside_ the spinlocks! */
@@ -7221,7 +7221,7 @@ static void balance_push(struct rq *rq)
 {
 	struct task_struct *push_task = rq->curr;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 	SCHED_WARN_ON(rq->cpu != smp_processor_id());
 
 	/*
@@ -7242,9 +7242,9 @@ static void balance_push(struct rq *rq)
 		 */
 		if (!rq->nr_running && !rq_has_pinned_tasks(rq) &&
 		    rcuwait_active(&rq->hotplug_wait)) {
-			raw_spin_unlock(&rq->lock);
+			raw_spin_unlock(rq_lockp(rq));
 			rcuwait_wake_up(&rq->hotplug_wait);
-			raw_spin_lock(&rq->lock);
+			raw_spin_lock(rq_lockp(rq));
 		}
 		return;
 	}
@@ -7254,7 +7254,7 @@ static void balance_push(struct rq *rq)
 	 * Temporarily drop rq->lock such that we can wake-up the stop task.
 	 * Both preemption and IRQs are still disabled.
 	 */
-	raw_spin_unlock(&rq->lock);
+	raw_spin_unlock(rq_lockp(rq));
 	stop_one_cpu_nowait(rq->cpu, __balance_push_cpu_stop, push_task,
 			    this_cpu_ptr(&push_work));
 	/*
@@ -7262,7 +7262,7 @@ static void balance_push(struct rq *rq)
 	 * schedule(). The next pick is obviously going to be the stop task
 	 * which is_per_cpu_kthread() and will push this task away.
 	 */
-	raw_spin_lock(&rq->lock);
+	raw_spin_lock(rq_lockp(rq));
 }
 
 static void balance_push_set(int cpu, bool on)
@@ -7682,7 +7682,7 @@ void __init sched_init(void)
 		struct rq *rq;
 
 		rq = cpu_rq(i);
-		raw_spin_lock_init(&rq->lock);
+		raw_spin_lock_init(&rq->__lock);
 		rq->nr_running = 0;
 		rq->calc_load_active = 0;
 		rq->calc_load_update = jiffies + LOAD_FREQ;
kernel/sched/cpuacct.c +6 -6
index 941c28cf9738..38c1a68e91f0 100644
@@ -112,7 +112,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu,
 	/*
 	 * Take rq->lock to make 64-bit read safe on 32-bit platforms.
 	 */
-	raw_spin_lock_irq(&cpu_rq(cpu)->lock);
+	raw_spin_lock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 
 	if (index == CPUACCT_STAT_NSTATS) {
@@ -126,7 +126,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu,
 	}
 
 #ifndef CONFIG_64BIT
-	raw_spin_unlock_irq(&cpu_rq(cpu)->lock);
+	raw_spin_unlock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 
 	return data;
@@ -141,14 +141,14 @@ static void cpuacct_cpuusage_write(struct cpuacct *ca, int cpu, u64 val)
 	/*
 	 * Take rq->lock to make 64-bit write safe on 32-bit platforms.
 	 */
-	raw_spin_lock_irq(&cpu_rq(cpu)->lock);
+	raw_spin_lock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 
 	for (i = 0; i < CPUACCT_STAT_NSTATS; i++)
 		cpuusage->usages[i] = val;
 
 #ifndef CONFIG_64BIT
-	raw_spin_unlock_irq(&cpu_rq(cpu)->lock);
+	raw_spin_unlock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 }
 
@@ -253,13 +253,13 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)
 			 * Take rq->lock to make 64-bit read safe on 32-bit
 			 * platforms.
 			 */
-			raw_spin_lock_irq(&cpu_rq(cpu)->lock);
+			raw_spin_lock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 
 			seq_printf(m, " %llu", cpuusage->usages[index]);
 
 #ifndef CONFIG_64BIT
-			raw_spin_unlock_irq(&cpu_rq(cpu)->lock);
+			raw_spin_unlock_irq(rq_lockp(cpu_rq(cpu)));
 #endif
 		}
 		seq_puts(m, "\n");
kernel/sched/deadline.c +11 -11
index 2a5836f440e0..0f2ea0a3664c 100644
@@ -135,7 +135,7 @@ void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
 	u64 old = dl_rq->running_bw;
 
-	lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+	lockdep_assert_held(rq_lockp(rq_of_dl_rq(dl_rq)));
 	dl_rq->running_bw += dl_bw;
 	SCHED_WARN_ON(dl_rq->running_bw < old); /* overflow */
 	SCHED_WARN_ON(dl_rq->running_bw > dl_rq->this_bw);
@@ -148,7 +148,7 @@ void __sub_running_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
 	u64 old = dl_rq->running_bw;
 
-	lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+	lockdep_assert_held(rq_lockp(rq_of_dl_rq(dl_rq)));
 	dl_rq->running_bw -= dl_bw;
 	SCHED_WARN_ON(dl_rq->running_bw > old); /* underflow */
 	if (dl_rq->running_bw > old)
@@ -162,7 +162,7 @@ void __add_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
 	u64 old = dl_rq->this_bw;
 
-	lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+	lockdep_assert_held(rq_lockp(rq_of_dl_rq(dl_rq)));
 	dl_rq->this_bw += dl_bw;
 	SCHED_WARN_ON(dl_rq->this_bw < old); /* overflow */
 }
@@ -172,7 +172,7 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq)
 {
 	u64 old = dl_rq->this_bw;
 
-	lockdep_assert_held(&(rq_of_dl_rq(dl_rq))->lock);
+	lockdep_assert_held(rq_lockp(rq_of_dl_rq(dl_rq)));
 	dl_rq->this_bw -= dl_bw;
 	SCHED_WARN_ON(dl_rq->this_bw > old); /* underflow */
 	if (dl_rq->this_bw > old)
@@ -982,7 +982,7 @@ static int start_dl_timer(struct task_struct *p)
 	ktime_t now, act;
 	s64 delta;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	/*
 	 * We want the timer to fire at the deadline, but considering
@@ -1092,9 +1092,9 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
 		 * If the runqueue is no longer available, migrate the
 		 * task elsewhere. This necessarily changes rq.
 		 */
-		lockdep_unpin_lock(&rq->lock, rf.cookie);
+		lockdep_unpin_lock(rq_lockp(rq), rf.cookie);
 		rq = dl_task_offline_migration(rq, p);
-		rf.cookie = lockdep_pin_lock(&rq->lock);
+		rf.cookie = lockdep_pin_lock(rq_lockp(rq));
 		update_rq_clock(rq);
 
 		/*
@@ -1746,7 +1746,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
 	 * from try_to_wake_up(). Hence, p->pi_lock is locked, but
 	 * rq->lock is not... So, lock it
 	 */
-	raw_spin_lock(&rq->lock);
+	raw_spin_lock(rq_lockp(rq));
 	if (p->dl.dl_non_contending) {
 		sub_running_bw(&p->dl, &rq->dl);
 		p->dl.dl_non_contending = 0;
@@ -1761,7 +1761,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused
 			put_task_struct(p);
 	}
 	sub_rq_bw(&p->dl, &rq->dl);
-	raw_spin_unlock(&rq->lock);
+	raw_spin_unlock(rq_lockp(rq));
 }
 
 static void check_preempt_equal_dl(struct rq *rq, struct task_struct *p)
@@ -2306,10 +2306,10 @@ static void pull_dl_task(struct rq *this_rq)
 		double_unlock_balance(this_rq, src_rq);
 
 		if (push_task) {
-			raw_spin_unlock(&this_rq->lock);
+			raw_spin_unlock(rq_lockp(this_rq));
 			stop_one_cpu_nowait(src_rq->cpu, push_cpu_stop,
 					    push_task, &src_rq->push_work);
-			raw_spin_lock(&this_rq->lock);
+			raw_spin_lock(rq_lockp(this_rq));
 		}
 	}
 
kernel/sched/debug.c +2 -2
index 2357921580f9..60a922d3f46f 100644
@@ -551,7 +551,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "exec_clock",
 			SPLIT_NS(cfs_rq->exec_clock));
 
-	raw_spin_lock_irqsave(&rq->lock, flags);
+	raw_spin_lock_irqsave(rq_lockp(rq), flags);
 	if (rb_first_cached(&cfs_rq->tasks_timeline))
 		MIN_vruntime = (__pick_first_entity(cfs_rq))->vruntime;
 	last = __pick_last_entity(cfs_rq);
@@ -559,7 +559,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
 		max_vruntime = last->vruntime;
 	min_vruntime = cfs_rq->min_vruntime;
 	rq0_min_vruntime = cpu_rq(0)->cfs.min_vruntime;
-	raw_spin_unlock_irqrestore(&rq->lock, flags);
+	raw_spin_unlock_irqrestore(rq_lockp(rq), flags);
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "MIN_vruntime",
 			SPLIT_NS(MIN_vruntime));
 	SEQ_printf(m, "  .%-30s: %Ld.%06ld\n", "min_vruntime",
kernel/sched/fair.c +19 -19
index 27536f37ba1a..52ddfec7cea6 100644
@@ -1110,7 +1110,7 @@ struct numa_group {
 static struct numa_group *deref_task_numa_group(struct task_struct *p)
 {
 	return rcu_dereference_check(p->numa_group, p == current ||
-		(lockdep_is_held(&task_rq(p)->lock) && !READ_ONCE(p->on_cpu)));
+		(lockdep_is_held(rq_lockp(task_rq(p))) && !READ_ONCE(p->on_cpu)));
 }
 
 static struct numa_group *deref_curr_numa_group(struct task_struct *p)
@@ -5309,7 +5309,7 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq)
 {
 	struct task_group *tg;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(tg, &task_groups, list) {
@@ -5328,7 +5328,7 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
 {
 	struct task_group *tg;
 
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(tg, &task_groups, list) {
@@ -6813,7 +6813,7 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)
 		 * In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old'
 		 * rq->lock and can modify state directly.
 		 */
-		lockdep_assert_held(&task_rq(p)->lock);
+		lockdep_assert_held(rq_lockp(task_rq(p)));
 		detach_entity_cfs_rq(&p->se);
 
 	} else {
@@ -7441,7 +7441,7 @@ static int task_hot(struct task_struct *p, struct lb_env *env)
 {
 	s64 delta;
 
-	lockdep_assert_held(&env->src_rq->lock);
+	lockdep_assert_held(rq_lockp(env->src_rq));
 
 	if (p->sched_class != &fair_sched_class)
 		return 0;
@@ -7539,7 +7539,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 {
 	int tsk_cache_hot;
 
-	lockdep_assert_held(&env->src_rq->lock);
+	lockdep_assert_held(rq_lockp(env->src_rq));
 
 	/*
 	 * We do not migrate tasks that are:
@@ -7617,7 +7617,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
  */
 static void detach_task(struct task_struct *p, struct lb_env *env)
 {
-	lockdep_assert_held(&env->src_rq->lock);
+	lockdep_assert_held(rq_lockp(env->src_rq));
 
 	deactivate_task(env->src_rq, p, DEQUEUE_NOCLOCK);
 	set_task_cpu(p, env->dst_cpu);
@@ -7633,7 +7633,7 @@ static struct task_struct *detach_one_task(struct lb_env *env)
 {
 	struct task_struct *p;
 
-	lockdep_assert_held(&env->src_rq->lock);
+	lockdep_assert_held(rq_lockp(env->src_rq));
 
 	list_for_each_entry_reverse(p,
 			&env->src_rq->cfs_tasks, se.group_node) {
@@ -7669,7 +7669,7 @@ static int detach_tasks(struct lb_env *env)
 	struct task_struct *p;
 	int detached = 0;
 
-	lockdep_assert_held(&env->src_rq->lock);
+	lockdep_assert_held(rq_lockp(env->src_rq));
 
 	if (env->imbalance <= 0)
 		return 0;
@@ -7791,7 +7791,7 @@ static int detach_tasks(struct lb_env *env)
  */
 static void attach_task(struct rq *rq, struct task_struct *p)
 {
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 
 	BUG_ON(task_rq(p) != rq);
 	activate_task(rq, p, ENQUEUE_NOCLOCK);
@@ -9726,7 +9726,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 		if (need_active_balance(&env)) {
 			unsigned long flags;
 
-			raw_spin_lock_irqsave(&busiest->lock, flags);
+			raw_spin_lock_irqsave(rq_lockp(busiest), flags);
 
 			/*
 			 * Don't kick the active_load_balance_cpu_stop,
@@ -9734,7 +9734,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 			 * moved to this_cpu:
 			 */
 			if (!cpumask_test_cpu(this_cpu, busiest->curr->cpus_ptr)) {
-				raw_spin_unlock_irqrestore(&busiest->lock,
+				raw_spin_unlock_irqrestore(rq_lockp(busiest),
 							    flags);
 				env.flags |= LBF_ALL_PINNED;
 				goto out_one_pinned;
@@ -9750,7 +9750,7 @@ static int load_balance(int this_cpu, struct rq *this_rq,
 				busiest->push_cpu = this_cpu;
 				active_balance = 1;
 			}
-			raw_spin_unlock_irqrestore(&busiest->lock, flags);
+			raw_spin_unlock_irqrestore(rq_lockp(busiest), flags);
 
 			if (active_balance) {
 				stop_one_cpu_nowait(cpu_of(busiest),
@@ -10506,7 +10506,7 @@ static void nohz_newidle_balance(struct rq *this_rq)
 	    time_before(jiffies, READ_ONCE(nohz.next_blocked)))
 		return;
 
-	raw_spin_unlock(&this_rq->lock);
+	raw_spin_unlock(rq_lockp(this_rq));
 	/*
 	 * This CPU is going to be idle and blocked load of idle CPUs
 	 * need to be updated. Run the ilb locally as it is a good
@@ -10515,7 +10515,7 @@ static void nohz_newidle_balance(struct rq *this_rq)
 	 */
 	if (!_nohz_idle_balance(this_rq, NOHZ_STATS_KICK, CPU_NEWLY_IDLE))
 		kick_ilb(NOHZ_STATS_KICK);
-	raw_spin_lock(&this_rq->lock);
+	raw_spin_lock(rq_lockp(this_rq));
 }
 
 #else /* !CONFIG_NO_HZ_COMMON */
@@ -10581,7 +10581,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf)
 		goto out;
 	}
 
-	raw_spin_unlock(&this_rq->lock);
+	raw_spin_unlock(rq_lockp(this_rq));
 
 	update_blocked_averages(this_cpu);
 	rcu_read_lock();
@@ -10619,7 +10619,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf)
 	}
 	rcu_read_unlock();
 
-	raw_spin_lock(&this_rq->lock);
+	raw_spin_lock(rq_lockp(this_rq));
 
 	if (curr_cost > this_rq->max_idle_balance_cost)
 		this_rq->max_idle_balance_cost = curr_cost;
@@ -11095,9 +11095,9 @@ void unregister_fair_sched_group(struct task_group *tg)
 
 		rq = cpu_rq(cpu);
 
-		raw_spin_lock_irqsave(&rq->lock, flags);
+		raw_spin_lock_irqsave(rq_lockp(rq), flags);
 		list_del_leaf_cfs_rq(tg->cfs_rq[cpu]);
-		raw_spin_unlock_irqrestore(&rq->lock, flags);
+		raw_spin_unlock_irqrestore(rq_lockp(rq), flags);
 	}
 }
 
kernel/sched/idle.c +2 -2
index df91b198a74c..50e128b899c4 100644
@@ -422,10 +422,10 @@ struct task_struct *pick_next_task_idle(struct rq *rq)
 static void
 dequeue_task_idle(struct rq *rq, struct task_struct *p, int flags)
 {
-	raw_spin_unlock_irq(&rq->lock);
+	raw_spin_unlock_irq(rq_lockp(rq));
 	printk(KERN_ERR "bad: scheduling from the idle thread!\n");
 	dump_stack();
-	raw_spin_lock_irq(&rq->lock);
+	raw_spin_lock_irq(rq_lockp(rq));
 }
 
 /*
kernel/sched/pelt.h +1 -1
index 795e43e02afc..e850bd71a8ce 100644
@@ -141,7 +141,7 @@ static inline void update_idle_rq_clock_pelt(struct rq *rq)
 
 static inline u64 rq_clock_pelt(struct rq *rq)
 {
-	lockdep_assert_held(&rq->lock);
+	lockdep_assert_held(rq_lockp(rq));
 	assert_clock_updated(rq);
 
 	return rq->clock_pelt - rq->lost_idle_time;
kernel/sched/rt.c +8 -8
index dbe4629cf7ba..a6f9d132c24f 100644
@@ -888,7 +888,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)
 		if (skip)
 			continue;
 
-		raw_spin_lock(&rq->lock);
+		raw_spin_lock(rq_lockp(rq));
 		update_rq_clock(rq);
 
 		if (rt_rq->rt_time) {
@@ -926,7 +926,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)
 
 		if (enqueue)
 			sched_rt_rq_enqueue(rt_rq);
-		raw_spin_unlock(&rq->lock);
+		raw_spin_unlock(rq_lockp(rq));
 	}
 
 	if (!throttled && (!rt_bandwidth_enabled() || rt_b->rt_runtime == RUNTIME_INF))
@@ -1894,10 +1894,10 @@ static int push_rt_task(struct rq *rq, bool pull)
 		 */
 		push_task = get_push_task(rq);
 		if (push_task) {
-			raw_spin_unlock(&rq->lock);
+			raw_spin_unlock(rq_lockp(rq));
 			stop_one_cpu_nowait(rq->cpu, push_cpu_stop,
 					    push_task, &rq->push_work);
-			raw_spin_lock(&rq->lock);
+			raw_spin_lock(rq_lockp(rq));
 		}
 
 		return 0;
@@ -2122,10 +2122,10 @@ void rto_push_irq_work_func(struct irq_work *work)
 	 * When it gets updated, a check is made if a push is possible.
 	 */
 	if (has_pushable_tasks(rq)) {
-		raw_spin_lock(&rq->lock);
+		raw_spin_lock(rq_lockp(rq));
 		while (push_rt_task(rq, true))
 			;
-		raw_spin_unlock(&rq->lock);
+		raw_spin_unlock(rq_lockp(rq));
 	}
 
 	raw_spin_lock(&rd->rto_lock);
@@ -2243,10 +2243,10 @@ static void pull_rt_task(struct rq *this_rq)
 		double_unlock_balance(this_rq, src_rq);
 
 		if (push_task) {
-			raw_spin_unlock(&this_rq->lock);
+			raw_spin_unlock(rq_lockp(this_rq));
 			stop_one_cpu_nowait(src_rq->cpu, push_cpu_stop,
 					    push_task, &src_rq->push_work);
-			raw_spin_lock(&this_rq->lock);
+			raw_spin_lock(rq_lockp(this_rq));
 		}
 	}
 
kernel/sched/sched.h +56 -52

278 lines not on this page · raw message

kernel/sched/topology.c +2 -2
index 90f3e5558fa2..82924db74ccb 100644
@@ -454,7 +454,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	struct root_domain *old_rd = NULL;
 	unsigned long flags;
 
-	raw_spin_lock_irqsave(&rq->lock, flags);
+	raw_spin_lock_irqsave(rq_lockp(rq), flags);
 
 	if (rq->rd) {
 		old_rd = rq->rd;
@@ -480,7 +480,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)
 	if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
 		set_rq_online(rq);
 
-	raw_spin_unlock_irqrestore(&rq->lock, flags);
+	raw_spin_unlock_irqrestore(rq_lockp(rq), flags);
 
 	if (old_rd)
 		call_rcu(&old_rd->rcu, free_rootdomain);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#2 lore raw

Re: [PATCH -tip 01/32] sched: Wrap rq::lock access

Quoted 5 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

In preparation of playing games with rq->lock, abstract the thing using an accessor.

Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.

Balbir Singh

#3 lore raw

Re: [PATCH -tip 01/32] sched: Wrap rq::lock access

Quoted 10 lines · Balbir
Quoted 5 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

In preparation of playing games with rq->lock, abstract the thing using an accessor.

Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.

Your line wrapping broke, I fixed it.

That is in fact the game. By wrapping it, the nature of the locking is dynamic based on whether core sched is enabled or not (both statically and dynamically).

thanks,

- Joel

Quoted 1 line

Balbir Singh

#4 lore raw

Re: [PATCH -tip 01/32] sched: Wrap rq::lock access

Quoted 8 lines · Joel Fernandes
Quoted 7 lines · Balbir
Quoted 5 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

In preparation of playing games with rq->lock, abstract the thing using an accessor.

Quoted 6 lines
Quoted 3 lines

Could you clarify games? I presume the intention is to redefine the scope of the lock based on whether core sched is enabled or not? I presume patch 4/32 has the details.

Your line wrapping broke, I fixed it.

Sorry, I've been using thunderbird from time to time and even though I set the options specified in the Documentation (email-clients), it's not working as expected.

Quoted 4 lines

That is in fact the game. By wrapping it, the nature of the locking is dynamic based on whether core sched is enabled or not (both statically and dynamically).

My point was that the word game does not do justice to the change, some details around how this abstractions helps based on the (re)definition of rq with coresched might help.

Balbir Singh.

#5 lore raw

PATCH 2/32 sched: Introduce sched_class::pick_task()

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Because sched_class::pick_next_task() also implies sched_class::set_next_task() (and possibly put_prev_task() and newidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.

---
 kernel/sched/deadline.c  | 16 ++++++++++++++--
 kernel/sched/fair.c      | 32 +++++++++++++++++++++++++++++++-
 kernel/sched/idle.c      |  8 ++++++++
 kernel/sched/rt.c        | 15 +++++++++++++--
 kernel/sched/sched.h     |  3 +++
 kernel/sched/stop_task.c | 14 ++++++++++++--
 6 files changed, 81 insertions(+), 7 deletions(-)
kernel/sched/deadline.c +14 -2
index 0f2ea0a3664c..abfc8b505d0d 100644
@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
 	return rb_entry(left, struct sched_dl_entity, rb_node);
 }
 
-static struct task_struct *pick_next_task_dl(struct rq *rq)
+static struct task_struct *pick_task_dl(struct rq *rq)
 {
 	struct sched_dl_entity *dl_se;
 	struct dl_rq *dl_rq = &rq->dl;
@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq)
 	dl_se = pick_next_dl_entity(rq, dl_rq);
 	BUG_ON(!dl_se);
 	p = dl_task_of(dl_se);
-	set_next_task_dl(rq, p, true);
+
+	return p;
+}
+
+static struct task_struct *pick_next_task_dl(struct rq *rq)
+{
+	struct task_struct *p;
+
+	p = pick_task_dl(rq);
+	if (p)
+		set_next_task_dl(rq, p, true);
+
 	return p;
 }
 
@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_dl,
+	.pick_task		= pick_task_dl,
 	.select_task_rq		= select_task_rq_dl,
 	.migrate_task_rq	= migrate_task_rq_dl,
 	.set_cpus_allowed       = set_cpus_allowed_dl,
kernel/sched/fair.c +31 -1
index 52ddfec7cea6..12cf068eeec8 100644
@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 	 * Avoid running the skip buddy, if running something else can
 	 * be done without getting too unfair.
 	 */
-	if (cfs_rq->skip == se) {
+	if (cfs_rq->skip && cfs_rq->skip == se) {
 		struct sched_entity *second;
 
 		if (se == curr) {
@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
 		set_last_buddy(se);
 }
 
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	struct sched_entity *se;
+
+	if (!cfs_rq->nr_running)
+		return NULL;
+
+	do {
+		struct sched_entity *curr = cfs_rq->curr;
+
+		se = pick_next_entity(cfs_rq, NULL);
+
+		if (curr) {
+			if (se && curr->on_rq)
+				update_curr(cfs_rq);
+
+			if (!se || entity_before(curr, se))
+				se = curr;
+		}
+
+		cfs_rq = group_cfs_rq(se);
+	} while (cfs_rq);
+
+	return task_of(se);
+}
+#endif
+
 struct task_struct *
 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
@@ -11219,6 +11248,7 @@ DEFINE_SCHED_CLASS(fair) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_fair,
+	.pick_task		= pick_task_fair,
 	.select_task_rq		= select_task_rq_fair,
 	.migrate_task_rq	= migrate_task_rq_fair,
 
kernel/sched/idle.c +8 -0
index 50e128b899c4..33864193a2f9 100644
@@ -406,6 +406,13 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 	schedstat_inc(rq->sched_goidle);
 }
 
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_idle(struct rq *rq)
+{
+	return rq->idle;
+}
+#endif
+
 struct task_struct *pick_next_task_idle(struct rq *rq)
 {
 	struct task_struct *next = rq->idle;
@@ -473,6 +480,7 @@ DEFINE_SCHED_CLASS(idle) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_idle,
+	.pick_task		= pick_task_idle,
 	.select_task_rq		= select_task_rq_idle,
 	.set_cpus_allowed	= set_cpus_allowed_common,
 #endif
kernel/sched/rt.c +13 -2
index a6f9d132c24f..a0e245b0c4bd 100644
@@ -1626,7 +1626,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)
 	return rt_task_of(rt_se);
 }
 
-static struct task_struct *pick_next_task_rt(struct rq *rq)
+static struct task_struct *pick_task_rt(struct rq *rq)
 {
 	struct task_struct *p;
 
@@ -1634,7 +1634,17 @@ static struct task_struct *pick_next_task_rt(struct rq *rq)
 		return NULL;
 
 	p = _pick_next_task_rt(rq);
-	set_next_task_rt(rq, p, true);
+
+	return p;
+}
+
+static struct task_struct *pick_next_task_rt(struct rq *rq)
+{
+	struct task_struct *p = pick_task_rt(rq);
+
+	if (p)
+		set_next_task_rt(rq, p, true);
+
 	return p;
 }
 
@@ -2483,6 +2493,7 @@ DEFINE_SCHED_CLASS(rt) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_rt,
+	.pick_task		= pick_task_rt,
 	.select_task_rq		= select_task_rq_rt,
 	.set_cpus_allowed       = set_cpus_allowed_common,
 	.rq_online              = rq_online_rt,
kernel/sched/sched.h +3 -0
index f794c9337047..5a0dd2b312aa 100644
@@ -1839,6 +1839,9 @@ struct sched_class {
 #ifdef CONFIG_SMP
 	int (*balance)(struct rq *rq, struct task_struct *prev, struct rq_flags *rf);
 	int  (*select_task_rq)(struct task_struct *p, int task_cpu, int flags);
+
+	struct task_struct * (*pick_task)(struct rq *rq);
+
 	void (*migrate_task_rq)(struct task_struct *p, int new_cpu);
 
 	void (*task_woken)(struct rq *this_rq, struct task_struct *task);
kernel/sched/stop_task.c +12 -2
index 55f39125c0e1..f988ebe3febb 100644
@@ -34,15 +34,24 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool fir
 	stop->se.exec_start = rq_clock_task(rq);
 }
 
-static struct task_struct *pick_next_task_stop(struct rq *rq)
+static struct task_struct *pick_task_stop(struct rq *rq)
 {
 	if (!sched_stop_runnable(rq))
 		return NULL;
 
-	set_next_task_stop(rq, rq->stop, true);
 	return rq->stop;
 }
 
+static struct task_struct *pick_next_task_stop(struct rq *rq)
+{
+	struct task_struct *p = pick_task_stop(rq);
+
+	if (p)
+		set_next_task_stop(rq, p, true);
+
+	return p;
+}
+
 static void
 enqueue_task_stop(struct rq *rq, struct task_struct *p, int flags)
 {
@@ -123,6 +132,7 @@ DEFINE_SCHED_CLASS(stop) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_stop,
+	.pick_task		= pick_task_stop,
 	.select_task_rq		= select_task_rq_stop,
 	.set_cpus_allowed	= set_cpus_allowed_common,
 #endif
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#6 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

Quoted 7 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Because sched_class::pick_next_task() also implies sched_class::set_next_task() (and possibly put_prev_task() and newidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.

The change makes sense, a small suggestion below

Quoted 83 lines
---
 kernel/sched/deadline.c  | 16 ++++++++++++++--
 kernel/sched/fair.c      | 32 +++++++++++++++++++++++++++++++-
 kernel/sched/idle.c      |  8 ++++++++
 kernel/sched/rt.c        | 15 +++++++++++++--
 kernel/sched/sched.h     |  3 +++
 kernel/sched/stop_task.c | 14 ++++++++++++--
 6 files changed, 81 insertions(+), 7 deletions(-)
kernel/sched/deadline.c +14 -2
index 0f2ea0a3664c..abfc8b505d0d 100644
@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
 	return rb_entry(left, struct sched_dl_entity, rb_node);
 }
 
-static struct task_struct *pick_next_task_dl(struct rq *rq)
+static struct task_struct *pick_task_dl(struct rq *rq)
 {
 	struct sched_dl_entity *dl_se;
 	struct dl_rq *dl_rq = &rq->dl;
@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq)
 	dl_se = pick_next_dl_entity(rq, dl_rq);
 	BUG_ON(!dl_se);
 	p = dl_task_of(dl_se);
-	set_next_task_dl(rq, p, true);
+
+	return p;
+}
+
+static struct task_struct *pick_next_task_dl(struct rq *rq)
+{
+	struct task_struct *p;
+
+	p = pick_task_dl(rq);
+	if (p)
+		set_next_task_dl(rq, p, true);
+
 	return p;
 }
 
@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {
 
 #ifdef CONFIG_SMP
 	.balance		= balance_dl,
+	.pick_task		= pick_task_dl,
 	.select_task_rq		= select_task_rq_dl,
 	.migrate_task_rq	= migrate_task_rq_dl,
 	.set_cpus_allowed       = set_cpus_allowed_dl,
kernel/sched/fair.c +12 -1
index 52ddfec7cea6..12cf068eeec8 100644
@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
 	 * Avoid running the skip buddy, if running something else can
 	 * be done without getting too unfair.
 	 */
-	if (cfs_rq->skip == se) {
+	if (cfs_rq->skip && cfs_rq->skip == se) {
 		struct sched_entity *second;
 
 		if (se == curr) {
@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
 		set_last_buddy(se);
 }
 
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	struct sched_entity *se;
+
+	if (!cfs_rq->nr_running)
+		return NULL;
+
+	do {
+		struct sched_entity *curr = cfs_rq->curr;

kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040

Quoted 10 lines
kernel/sched/fair.c +10 -0
+
+		se = pick_next_entity(cfs_rq, NULL);
+
+		if (curr) {
+			if (se && curr->on_rq)
+				update_curr(cfs_rq);
+
+			if (!se || entity_before(curr, se))
+				se = curr;
+		}

Do we want to optimize this a bit

if (curr) {
	if (!se || entity_before(curr, se))
		se = curr;

	if ((se != curr) && curr->on_rq)
		update_curr(cfs_rq);

}

Balbir

#7 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

On Fri, Nov 20, 2020 at 10:56:09AM +1100, Singh, Balbir wrote:
[..]

Quoted 24 lines
Quoted 11 lines
kernel/sched/fair.c +11 -0
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	struct sched_entity *se;
+
+	if (!cfs_rq->nr_running)
+		return NULL;
+
+	do {
+		struct sched_entity *curr = cfs_rq->curr;

kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040

Quoted 10 lines
kernel/sched/fair.c +10 -0
+
+		se = pick_next_entity(cfs_rq, NULL);
+
+		if (curr) {
+			if (se && curr->on_rq)
+				update_curr(cfs_rq);
+
+			if (!se || entity_before(curr, se))
+				se = curr;
+		}

Do we want to optimize this a bit

Quoted 8 lines
if (curr) {
	if (!se || entity_before(curr, se))
		se = curr;

	if ((se != curr) && curr->on_rq)
		update_curr(cfs_rq);

}

Do you see a difference in codegen? What's the optimization?

Also in later patches this code changes, so it should not matter:
See: 3e0838fa3c51 ("sched/fair: Fix pick_task_fair crashes due to empty rbtree")

thanks,

- Joel

#8 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

Quoted 33 lines · Joel Fernandes

On Fri, Nov 20, 2020 at 10:56:09AM +1100, Singh, Balbir wrote:
[..]

Quoted 31 lines
Quoted 11 lines
kernel/sched/fair.c +11 -0
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+	struct cfs_rq *cfs_rq = &rq->cfs;
+	struct sched_entity *se;
+
+	if (!cfs_rq->nr_running)
+		return NULL;
+
+	do {
+		struct sched_entity *curr = cfs_rq->curr;

kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7040

Quoted 10 lines
kernel/sched/fair.c +10 -0
+
+		se = pick_next_entity(cfs_rq, NULL);
+
+		if (curr) {
+			if (se && curr->on_rq)
+				update_curr(cfs_rq);
+
+			if (!se || entity_before(curr, se))
+				se = curr;
+		}

Do we want to optimize this a bit

if (curr) {
	if (!se || entity_before(curr, se))
		se = curr;

	if ((se != curr) && curr->on_rq)
		update_curr(cfs_rq);
Quoted 7 lines
Quoted 1 line

}

Do you see a difference in codegen? What's the optimization?

Also in later patches this code changes, so it should not matter:
See: 3e0838fa3c51 ("sched/fair: Fix pick_task_fair crashes due to empty rbtree")

I did see the next patch, but the idea is that we don't need to update_curr() if the picked sched entity is the same as current (se == curr). What are the side-effects of not updating curr when se == curr?

Balbir

maintainer
#9 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

In reply to Joel Fernandes (Google)
Quoted 83 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Because sched_class::pick_next_task() also implies sched_class::set_next_task() (and possibly put_prev_task() and newidle_balance) it is not state invariant. This makes it unsuitable for remote task selection.

---
 kernel/sched/deadline.c  | 16 ++++++++++++++--
 kernel/sched/fair.c      | 32 +++++++++++++++++++++++++++++++-
 kernel/sched/idle.c      |  8 ++++++++
 kernel/sched/rt.c        | 15 +++++++++++++--
 kernel/sched/sched.h     |  3 +++
 kernel/sched/stop_task.c | 14 ++++++++++++--
 6 files changed, 81 insertions(+), 7 deletions(-)
kernel/sched/deadline.c +14 -2
index 0f2ea0a3664c..abfc8b505d0d 100644
@@ -1867,7 +1867,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
        return rb_entry(left, struct sched_dl_entity, rb_node);
 }
 
-static struct task_struct *pick_next_task_dl(struct rq *rq)
+static struct task_struct *pick_task_dl(struct rq *rq)
 {
        struct sched_dl_entity *dl_se;
        struct dl_rq *dl_rq = &rq->dl;
@@ -1879,7 +1879,18 @@ static struct task_struct *pick_next_task_dl(struct rq *rq)
        dl_se = pick_next_dl_entity(rq, dl_rq);
        BUG_ON(!dl_se);
        p = dl_task_of(dl_se);
-       set_next_task_dl(rq, p, true);
+
+       return p;
+}
+
+static struct task_struct *pick_next_task_dl(struct rq *rq)
+{
+       struct task_struct *p;
+
+       p = pick_task_dl(rq);
+       if (p)
+               set_next_task_dl(rq, p, true);
+
        return p;
 }
 
@@ -2551,6 +2562,7 @@ DEFINE_SCHED_CLASS(dl) = {
 
 #ifdef CONFIG_SMP
        .balance                = balance_dl,
+       .pick_task              = pick_task_dl,
        .select_task_rq         = select_task_rq_dl,
        .migrate_task_rq        = migrate_task_rq_dl,
        .set_cpus_allowed       = set_cpus_allowed_dl,
kernel/sched/fair.c +4 -1
index 52ddfec7cea6..12cf068eeec8 100644
@@ -4459,7 +4459,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr)
         * Avoid running the skip buddy, if running something else can
         * be done without getting too unfair.
         */
-       if (cfs_rq->skip == se) {
+       if (cfs_rq->skip && cfs_rq->skip == se) {
                struct sched_entity *second;
 
                if (se == curr) {
@@ -7017,6 +7017,35 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
                set_last_buddy(se);
 }
 
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{

kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7032

Quoted 10 lines
kernel/sched/fair.c +10 -0
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       struct sched_entity *se;
+
+       if (!cfs_rq->nr_running)
+               return NULL;
+
+       do {
+               struct sched_entity *curr = cfs_rq->curr;
+
+               se = pick_next_entity(cfs_rq, NULL);

Calling pick_next_entity clears buddies. This is fine without coresched because the se will be the next one. But calling pick_task_fair doesn't mean that the se will be used

kernel/sched/fair.c static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ · line 7037

Quoted 5 lines
kernel/sched/fair.c +5 -0
+
+               if (curr) {
+                       if (se && curr->on_rq)
+                               update_curr(cfs_rq);
+

Shouldn't you check if cfs_rq is throttled ?

Quoted 144 lines
kernel/sched/fair.c +12 -0
+                       if (!se || entity_before(curr, se))
+                               se = curr;
+               }
+
+               cfs_rq = group_cfs_rq(se);
+       } while (cfs_rq);
+
+       return task_of(se);
+}
+#endif
+
 struct task_struct *
 pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
@@ -11219,6 +11248,7 @@ DEFINE_SCHED_CLASS(fair) = {
 
 #ifdef CONFIG_SMP
        .balance                = balance_fair,
+       .pick_task              = pick_task_fair,
        .select_task_rq         = select_task_rq_fair,
        .migrate_task_rq        = migrate_task_rq_fair,
 
kernel/sched/idle.c +8 -0
index 50e128b899c4..33864193a2f9 100644
@@ -406,6 +406,13 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
        schedstat_inc(rq->sched_goidle);
 }
 
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_idle(struct rq *rq)
+{
+       return rq->idle;
+}
+#endif
+
 struct task_struct *pick_next_task_idle(struct rq *rq)
 {
        struct task_struct *next = rq->idle;
@@ -473,6 +480,7 @@ DEFINE_SCHED_CLASS(idle) = {
 
 #ifdef CONFIG_SMP
        .balance                = balance_idle,
+       .pick_task              = pick_task_idle,
        .select_task_rq         = select_task_rq_idle,
        .set_cpus_allowed       = set_cpus_allowed_common,
 #endif
kernel/sched/rt.c +13 -2
index a6f9d132c24f..a0e245b0c4bd 100644
@@ -1626,7 +1626,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)
        return rt_task_of(rt_se);
 }
 
-static struct task_struct *pick_next_task_rt(struct rq *rq)
+static struct task_struct *pick_task_rt(struct rq *rq)
 {
        struct task_struct *p;
 
@@ -1634,7 +1634,17 @@ static struct task_struct *pick_next_task_rt(struct rq *rq)
                return NULL;
 
        p = _pick_next_task_rt(rq);
-       set_next_task_rt(rq, p, true);
+
+       return p;
+}
+
+static struct task_struct *pick_next_task_rt(struct rq *rq)
+{
+       struct task_struct *p = pick_task_rt(rq);
+
+       if (p)
+               set_next_task_rt(rq, p, true);
+
        return p;
 }
 
@@ -2483,6 +2493,7 @@ DEFINE_SCHED_CLASS(rt) = {
 
 #ifdef CONFIG_SMP
        .balance                = balance_rt,
+       .pick_task              = pick_task_rt,
        .select_task_rq         = select_task_rq_rt,
        .set_cpus_allowed       = set_cpus_allowed_common,
        .rq_online              = rq_online_rt,
kernel/sched/sched.h +3 -0
index f794c9337047..5a0dd2b312aa 100644
@@ -1839,6 +1839,9 @@ struct sched_class {
 #ifdef CONFIG_SMP
        int (*balance)(struct rq *rq, struct task_struct *prev, struct rq_flags *rf);
        int  (*select_task_rq)(struct task_struct *p, int task_cpu, int flags);
+
+       struct task_struct * (*pick_task)(struct rq *rq);
+
        void (*migrate_task_rq)(struct task_struct *p, int new_cpu);
 
        void (*task_woken)(struct rq *this_rq, struct task_struct *task);
kernel/sched/stop_task.c +12 -3
index 55f39125c0e1..f988ebe3febb 100644
@@ -34,15 +34,24 @@ static void set_next_task_stop(struct rq *rq, struct task_struct *stop, bool fir
        stop->se.exec_start = rq_clock_task(rq);
 }
 
-static struct task_struct *pick_next_task_stop(struct rq *rq)
+static struct task_struct *pick_task_stop(struct rq *rq)
 {
        if (!sched_stop_runnable(rq))
                return NULL;
 
-       set_next_task_stop(rq, rq->stop, true);
        return rq->stop;
 }
 
+static struct task_struct *pick_next_task_stop(struct rq *rq)
+{
+       struct task_struct *p = pick_task_stop(rq);
+
+       if (p)
+               set_next_task_stop(rq, p, true);
+
+       return p;
+}
+
 static void
 enqueue_task_stop(struct rq *rq, struct task_struct *p, int flags)
 {
@@ -123,6 +132,7 @@ DEFINE_SCHED_CLASS(stop) = {
 
 #ifdef CONFIG_SMP
        .balance                = balance_stop,
+       .pick_task              = pick_task_stop,
        .select_task_rq         = select_task_rq_stop,
        .set_cpus_allowed       = set_cpus_allowed_common,
 #endif
--

2.29.2.299.gdc1121823c-goog

maintainer
#10 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

Quoted 11 lines · Vincent Guittot

On Wed, 18 Nov 2020 at 00:20, Joel Fernandes (Google)

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       struct sched_entity *se;
+
+       if (!cfs_rq->nr_running)
+               return NULL;
+
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+       do {
+               struct sched_entity *curr = cfs_rq->curr;
+
+               se = pick_next_entity(cfs_rq, NULL);

Calling pick_next_entity clears buddies. This is fine without coresched because the se will be the next one. But calling pick_task_fair doesn't mean that the se will be used

Urgh, nice one :/

Quoted 7 lines

Review

Quoted 5 lines
(quoted diff) +5 -0
+
+               if (curr) {
+                       if (se && curr->on_rq)
+                               update_curr(cfs_rq);
+

Shouldn't you check if cfs_rq is throttled ?

Hmm,... I suppose we do.

Quoted 10 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+                       if (!se || entity_before(curr, se))
+                               se = curr;
+               }
+
+               cfs_rq = group_cfs_rq(se);
+       } while (cfs_rq);
+
+       return task_of(se);
+}
+#endif

Something like so then?

kernel/sched/fair.c +16 -7
@@ -4354,6 +4354,8 @@ check_preempt_tick(struct cfs_rq *cfs_rq
 static void
 set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+	clear_buddies(cfs_rq, se);
+
 	/* 'current' is not kept within the tree. */
 	if (se->on_rq) {
 		/*
@@ -4440,8 +4442,6 @@ pick_next_entity(struct cfs_rq *cfs_rq,
 		se = cfs_rq->last;
 	}
 
-	clear_buddies(cfs_rq, se);
-
 	return se;
 }
 
@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct
 #ifdef CONFIG_SMP
 static struct task_struct *pick_task_fair(struct rq *rq)
 {
-	struct cfs_rq *cfs_rq = &rq->cfs;
 	struct sched_entity *se;
-
+	struct cfs_rq *cfs_rq;
+       
+again:
+	cfs_rq = &rq->cfs;
 	if (!cfs_rq->nr_running)
 		return NULL;
 
 	do {
 		struct sched_entity *curr = cfs_rq->curr;
 
-		if (curr && curr->on_rq)
-			update_curr(cfs_rq);
+		/* When we pick for a remote RQ, we'll not have done put_prev_entity() */
+		if (curr) {
+			if (curr->on_rq)
+				update_curr(cfs_rq);
+			else
+				curr = NULL;
 
-		se = pick_next_entity(cfs_rq, curr);
+			if (unlikely(check_cfs_rq_runtime(cfs_rq)))
+				goto again;
+		}
 
+		se = pick_next_entity(cfs_rq, curr);
 		cfs_rq = group_cfs_rq(se);
 	} while (cfs_rq);
 
maintainer
#11 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

Quoted 34 lines · Peter Zijlstra
Quoted 11 lines · Vincent Guittot

On Wed, 18 Nov 2020 at 00:20, Joel Fernandes (Google)

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_SMP
+static struct task_struct *pick_task_fair(struct rq *rq)
+{
+       struct cfs_rq *cfs_rq = &rq->cfs;
+       struct sched_entity *se;
+
+       if (!cfs_rq->nr_running)
+               return NULL;
+
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+       do {
+               struct sched_entity *curr = cfs_rq->curr;
+
+               se = pick_next_entity(cfs_rq, NULL);

Calling pick_next_entity clears buddies. This is fine without coresched because the se will be the next one. But calling pick_task_fair doesn't mean that the se will be used

Urgh, nice one :/

Quoted 7 lines

kernel/sched/fair.c static void check_preempt_wakeup(struct

Quoted 5 lines
kernel/sched/fair.c +5 -0
+
+               if (curr) {
+                       if (se && curr->on_rq)
+                               update_curr(cfs_rq);
+

Shouldn't you check if cfs_rq is throttled ?

Hmm,... I suppose we do.

Quoted 12 lines
Quoted 10 lines

Review

Quoted 10 lines
(quoted diff) +10 -0
+                       if (!se || entity_before(curr, se))
+                               se = curr;
+               }
+
+               cfs_rq = group_cfs_rq(se);
+       } while (cfs_rq);
+
+       return task_of(se);
+}
+#endif

Something like so then?

yes. it seems ok

Quoted 56 lines
kernel/sched/fair.c +16 -7
@@ -4354,6 +4354,8 @@ check_preempt_tick(struct cfs_rq *cfs_rq
 static void
 set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
 {
+       clear_buddies(cfs_rq, se);
+
        /* 'current' is not kept within the tree. */
        if (se->on_rq) {
                /*
@@ -4440,8 +4442,6 @@ pick_next_entity(struct cfs_rq *cfs_rq,
                se = cfs_rq->last;
        }
 
-       clear_buddies(cfs_rq, se);
-
        return se;
 }
 
@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct
 #ifdef CONFIG_SMP
 static struct task_struct *pick_task_fair(struct rq *rq)
 {
-       struct cfs_rq *cfs_rq = &rq->cfs;
        struct sched_entity *se;
-
+       struct cfs_rq *cfs_rq;
+
+again:
+       cfs_rq = &rq->cfs;
        if (!cfs_rq->nr_running)
                return NULL;
 
        do {
                struct sched_entity *curr = cfs_rq->curr;
 
-               if (curr && curr->on_rq)
-                       update_curr(cfs_rq);
+               /* When we pick for a remote RQ, we'll not have done put_prev_entity() */
+               if (curr) {
+                       if (curr->on_rq)
+                               update_curr(cfs_rq);
+                       else
+                               curr = NULL;
 
-               se = pick_next_entity(cfs_rq, curr);
+                       if (unlikely(check_cfs_rq_runtime(cfs_rq)))
+                               goto again;
+               }
 
+               se = pick_next_entity(cfs_rq, curr);
                cfs_rq = group_cfs_rq(se);
        } while (cfs_rq);
 
maintainer
#12 lore raw

Re: [PATCH -tip 02/32] sched: Introduce sched_class::pick_task()

Quoted 30 lines · Vincent Guittot
Quoted 1 line

Something like so then?

yes. it seems ok

Quoted 26 lines
kernel/sched/fair.c +10 -0
@@ -6982,20 +6982,29 @@ static void check_preempt_wakeup(struct
 #ifdef CONFIG_SMP
 static struct task_struct *pick_task_fair(struct rq *rq)
 {
        struct sched_entity *se;
+       struct cfs_rq *cfs_rq;
+
+again:
+       cfs_rq = &rq->cfs;
        if (!cfs_rq->nr_running)
                return NULL;
 
        do {
                struct sched_entity *curr = cfs_rq->curr;
 
+               /* When we pick for a remote RQ, we'll not have done put_prev_entity() */
+               if (curr) {
+                       if (curr->on_rq)
+                               update_curr(cfs_rq);
+                       else
+                               curr = NULL;
 
Quoted 2 lines

kernel/sched/fair.c static void check_preempt_wakeup(struct · line 7004

Quoted 2 lines
kernel/sched/fair.c +2 -0
+                       if (unlikely(check_cfs_rq_runtime(cfs_rq)))
+                               goto again;

Head-ache though; pick_task() was supposed to be stateless, but now we're modifying a remote runqueue... I suppose it still works, because irrespective of which task we end up picking (even idle), we'll schedule the remote CPU, which would've resulted in the same (and possibly triggered a reschedule if we'd not done it here).

There's a wrinkle through, other than in schedule(), where we dequeue() and keep running with the current task while we release rq->lock, this has preemption enabled as well.

This means that if we do this, the remote CPU could preempt, but the task is then no longer on the runqueue.

I _think_ it all still works, but yuck!

Quoted 6 lines
Quoted 6 lines
kernel/sched/fair.c +2 -0
+               }
 
+               se = pick_next_entity(cfs_rq, curr);
                cfs_rq = group_cfs_rq(se);
        } while (cfs_rq);
 
#13 lore raw

PATCH 3/32 sched/fair: Fix pick_task_fair crashes due to empty rbtree

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.

This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().

Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.

Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.

---
 kernel/sched/fair.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)
kernel/sched/fair.c +3 -8
index 12cf068eeec8..51483a00a755 100644
@@ -7029,15 +7029,10 @@ static struct task_struct *pick_task_fair(struct rq *rq)
 	do {
 		struct sched_entity *curr = cfs_rq->curr;
 
-		se = pick_next_entity(cfs_rq, NULL);
-
-		if (curr) {
-			if (se && curr->on_rq)
-				update_curr(cfs_rq);
+		if (curr && curr->on_rq)
+			update_curr(cfs_rq);
 
-			if (!se || entity_before(curr, se))
-				se = curr;
-		}
+		se = pick_next_entity(cfs_rq, curr);
 
 		cfs_rq = group_cfs_rq(se);
 	} while (cfs_rq);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#14 lore raw

Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree

Quoted 12 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.

This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().

Quoted 8 lines

Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.

Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.

It is not clear from the changelog as to what does put_prev_task() do to prevent the crash from occuring? Why did we pass NULL as curr in the first place to pick_next_entity?

The patch looks functionally correct as in, it passes curr as the reference to pick_next_entity() for left and entity_before comparisons.

Balbir Singh

#15 lore raw

Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree

Hi Balbir,

Quoted 17 lines · Balbir
Quoted 16 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.

This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().

Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.

Quoted 7 lines
Quoted 4 lines

Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.

It is not clear from the changelog as to what does put_prev_task() do to prevent the crash from occuring? Why did we pass NULL as curr in the first place to pick_next_entity?

A little more context on this crash in v8 is here:
https://lwn.net/ml/linux-kernel/[email protected]/

The issue here arises from the fact that, we try to pick task for a sibling while sibling is running a task. Running tasks are not in the cfs_rq and pick_next_entity can return NULL if there is only one cfs task in the cfs_rq. This would not happen normally because put_prev_task is called before pick_task and put_prev_task adds the task back to cfs_rq. But for coresched, pick_task is called on a remote sibling's cfs_rq without calling put_prev_task and this can lead to pick_next_entity returning NULL.

The initial logic of passing NULL would work fine as long as we call put_prev_task before calling pick_task_fair. But for coresched, we call pick_task_fair on siblings while the task is running and would not be able to call put_prev_task. So this refactor of the code fixes the crash by explicitly passing curr.

Hope this clarifies..

Thanks,
Vineeth

#16 lore raw

Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree

Quoted 39 lines · Vineeth Pillai

Hi Balbir,

Quoted 17 lines · Balbir
Quoted 16 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.

This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().

Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.

Quoted 7 lines
Quoted 4 lines

Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.

It is not clear from the changelog as to what does put_prev_task() do to prevent the crash from occuring? Why did we pass NULL as curr in the first place to pick_next_entity?

A little more context on this crash in v8 is here:
https://lwn.net/ml/linux-kernel/[email protected]/

The issue here arises from the fact that, we try to pick task for a sibling while sibling is running a task. Running tasks are not in the cfs_rq and pick_next_entity can return NULL if there is only one cfs task in the cfs_rq. This would not happen normally because put_prev_task is called before pick_task and put_prev_task adds the task back to cfs_rq. But for coresched, pick_task is called on a remote sibling's cfs_rq without calling put_prev_task and this can lead to pick_next_entity returning NULL.

Quoted 8 lines

The initial logic of passing NULL would work fine as long as we call put_prev_task before calling pick_task_fair. But for coresched, we call pick_task_fair on siblings while the task is running and would not be able to call put_prev_task. So this refactor of the code fixes the crash by explicitly passing curr.

Hope this clarifies..

Yes, it does!

Thanks,
Balbir Singh.

maintainer
#17 lore raw

Re: [PATCH -tip 03/32] sched/fair: Fix pick_task_fair crashes due to empty rbtree

In reply to Singh, Balbir
Quoted 22 lines · Balbir
Quoted 12 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

pick_next_entity() is passed curr == NULL during core-scheduling. Due to this, if the rbtree is empty, the 'left' variable is set to NULL within the function. This can cause crashes within the function.

This is not an issue if put_prev_task() is invoked on the currently running task before calling pick_next_entity(). However, in core scheduling, it is possible that a sibling CPU picks for another RQ in the core, via pick_task_fair(). This remote sibling would not get any opportunities to do a put_prev_task().

Quoted 8 lines

Fix it by refactoring pick_task_fair() such that pick_next_entity() is called with the cfs_rq->curr. This will prevent pick_next_entity() from crashing if its rbtree is empty.

Also this fixes another possible bug where update_curr() would not be called on the cfs_rq hierarchy if the rbtree is empty. This could effect cross-cpu comparison of vruntime.

Quoted 6 lines

It is not clear from the changelog as to what does put_prev_task() do to prevent the crash from occuring? Why did we pass NULL as curr in the first place to pick_next_entity?

The patch looks functionally correct as in, it passes curr as the reference to pick_next_entity() for left and entity_before comparisons.

This patch should not exist; it should be smashed into the previous patch. There is no point in preserving a crash.

#18 lore raw

PATCH 4/32 sched: Core-wide rq->lock

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Introduce the basic infrastructure to have a core wide rq->lock.

---
 kernel/Kconfig.preempt |   5 ++
 kernel/sched/core.c    | 108 +++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h   |  31 ++++++++++++
 3 files changed, 144 insertions(+)
kernel/Kconfig.preempt +5 -0
index bf82259cff96..6d8be4630bd6 100644
@@ -80,3 +80,8 @@ config PREEMPT_COUNT
 config PREEMPTION
        bool
        select PREEMPT_COUNT
+
+config SCHED_CORE
+	bool "Core Scheduling for SMT"
+	default y
+	depends on SCHED_SMT
kernel/sched/core.c +108 -0

132 lines not on this page · raw message

kernel/sched/sched.h +31 -0
index 5a0dd2b312aa..0dfccf988998 100644
@@ -1061,6 +1061,12 @@ struct rq {
 #endif
 	unsigned int		push_busy;
 	struct cpu_stop_work	push_work;
+
+#ifdef CONFIG_SCHED_CORE
+	/* per rq */
+	struct rq		*core;
+	unsigned int		core_enabled;
+#endif
 };
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -1099,11 +1105,36 @@ static inline bool is_migration_disabled(struct task_struct *p)
 #endif
 }
 
+#ifdef CONFIG_SCHED_CORE
+DECLARE_STATIC_KEY_FALSE(__sched_core_enabled);
+
+static inline bool sched_core_enabled(struct rq *rq)
+{
+	return static_branch_unlikely(&__sched_core_enabled) && rq->core_enabled;
+}
+
+static inline raw_spinlock_t *rq_lockp(struct rq *rq)
+{
+	if (sched_core_enabled(rq))
+		return &rq->core->__lock;
+
+	return &rq->__lock;
+}
+
+#else /* !CONFIG_SCHED_CORE */
+
+static inline bool sched_core_enabled(struct rq *rq)
+{
+	return false;
+}
+
 static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 {
 	return &rq->__lock;
 }
 
+#endif /* CONFIG_SCHED_CORE */
+
 #ifdef CONFIG_SCHED_SMT
 extern void __update_idle_core(struct rq *rq);
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#19 lore raw

Re: [PATCH -tip 04/32] sched: Core-wide rq->lock

Quoted 4 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Introduce the basic infrastructure to have a core wide rq->lock.

Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled? I don't see a strong use case for it, but I am wondering if the design will fall apart if that assumption is broken?

Balbir Singh

maintainer
#20 lore raw

Re: [PATCH -tip 04/32] sched: Core-wide rq->lock

Quoted 9 lines · Balbir Singh
Quoted 4 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Introduce the basic infrastructure to have a core wide rq->lock.

Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled?

Yep, patch even says so:

+ * XXX entirely possible to selectively enable cores, don't bother for now.

Quoted 3 lines

I don't see a strong use case for it,
but I am wondering if the design will fall apart if that assumption is broken?

The use-case I have is not using stop-machine. That is, stopping a whole core at a time, instead of the whole sodding machine. It's on the todo list *somewhere*....

#21 lore raw

Re: [PATCH -tip 04/32] sched: Core-wide rq->lock

Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Balbir Singh
Quoted 4 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Introduce the basic infrastructure to have a core wide rq->lock.

Quoted 7 lines
Quoted 3 lines

Reading through the patch, it seems like all the CPUs have to be running with sched core enabled/disabled? Is it possible to have some cores with core sched disabled?

Yep, patch even says so:

+ * XXX entirely possible to selectively enable cores, don't bother for now.

Yes, it does in the comments, I looked at just the changelog :)

Quoted 10 lines
Quoted 4 lines

I don't see a strong use case for it,
but I am wondering if the design will fall apart if that assumption is broken?

The use-case I have is not using stop-machine. That is, stopping a whole core at a time, instead of the whole sodding machine. It's on the todo list *somewhere*....

Good to know, I guess that would need a transition of the entire core to idle and maintenance of a mask that prevents tasks that need core scheduling from getting off of CPUs that enable core scheduling.

Balbir Singh.

#22 lore raw

PATCH 5/32 sched/fair: Add a few assertions

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

---
 kernel/sched/fair.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/fair.c +9 -2
index 51483a00a755..ca35bfc0a368 100644
@@ -6245,6 +6245,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)
 		task_util = uclamp_task_util(p);
 	}
 
+	/*
+	 * per-cpu select_idle_mask usage
+	 */
+	lockdep_assert_irqs_disabled();
+
 	if ((available_idle_cpu(target) || sched_idle_cpu(target)) &&
 	    asym_fits_capacity(task_util, target))
 		return target;
@@ -6710,8 +6715,6 @@ static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu)
  * certain conditions an idle sibling CPU if the domain has SD_WAKE_AFFINE set.
  *
  * Returns the target CPU number.
- *
- * preempt must be disabled.
  */
 static int
 select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
@@ -6724,6 +6727,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 	/* SD_flags and WF_flags share the first nibble */
 	int sd_flag = wake_flags & 0xF;
 
+	/*
+	 * required for stable ->cpus_allowed
+	 */
+	lockdep_assert_held(&p->pi_lock);
 	if (wake_flags & WF_TTWU) {
 		record_wakee(p);
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#23 lore raw

PATCH 6/32 sched: Basic tracking of matching tasks

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Introduce task_struct::core_cookie as an opaque identifier for core scheduling. When enabled; core scheduling will only allow matching task to be on the core; where idle matches everything.

When task_struct::core_cookie is set (and core scheduling is enabled) these tasks are indexed in a second RB-tree, first on cookie value then on scheduling function, such that matching task selection always finds the most elegible match.

NOTE: *shudder* at the overhead...

NOTE: *sigh*, a 3rd copy of the scheduling function; the alternative is per class tracking of cookies and that just duplicates a lot of stuff for no raisin (the 2nd copy lives in the rt-mutex PI code).

---
 include/linux/sched.h |   8 ++-
 kernel/sched/core.c   | 146 ++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/fair.c   |  46 -------------
 kernel/sched/sched.h  |  55 ++++++++++++++++
 4 files changed, 208 insertions(+), 47 deletions(-)
include/linux/sched.h +7 -1
index 7abbdd7f3884..344499ab29f2 100644
@@ -683,10 +683,16 @@ struct task_struct {
 	const struct sched_class	*sched_class;
 	struct sched_entity		se;
 	struct sched_rt_entity		rt;
+	struct sched_dl_entity		dl;
+
+#ifdef CONFIG_SCHED_CORE
+	struct rb_node			core_node;
+	unsigned long			core_cookie;
+#endif
+
 #ifdef CONFIG_CGROUP_SCHED
 	struct task_group		*sched_task_group;
 #endif
-	struct sched_dl_entity		dl;
 
 #ifdef CONFIG_UCLAMP_TASK
 	/*
kernel/sched/core.c +146 -0

170 lines not on this page · raw message

kernel/sched/fair.c +0 -46

66 lines not on this page · raw message

kernel/sched/sched.h +55 -0

67 lines not on this page · raw message

Signature · 1 line
2.29.2.299.gdc1121823c-goog
#24 lore raw

PATCH 7/32 sched: Add core wide task selection and scheduling.

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Instead of only selecting a local task, select a task for all SMT siblings for every reschedule on the core (irrespective which logical CPU does the reschedule).

---
 kernel/sched/core.c  | 301 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/sched.h |   6 +-
 2 files changed, 305 insertions(+), 2 deletions(-)
kernel/sched/core.c +300 -1
index 9d521033777f..1bd0b0bbb040 100644
@@ -5029,7 +5029,7 @@ static void put_prev_task_balance(struct rq *rq, struct task_struct *prev,
  * Pick up the highest-prio task:
  */
 static inline struct task_struct *
-pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+__pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
 	const struct sched_class *class;
 	struct task_struct *p;
@@ -5070,6 +5070,294 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 }
 
 #ifdef CONFIG_SCHED_CORE
+static inline bool is_task_rq_idle(struct task_struct *t)
+{
+	return (task_rq(t)->idle == t);
+}
+
+static inline bool cookie_equals(struct task_struct *a, unsigned long cookie)
+{
+	return is_task_rq_idle(a) || (a->core_cookie == cookie);
+}
+
+static inline bool cookie_match(struct task_struct *a, struct task_struct *b)
+{
+	if (is_task_rq_idle(a) || is_task_rq_idle(b))
+		return true;
+
+	return a->core_cookie == b->core_cookie;
+}
+
+// XXX fairness/fwd progress conditions
+/*
+ * Returns
+ * - NULL if there is no runnable task for this class.
+ * - the highest priority task for this runqueue if it matches
+ *   rq->core->core_cookie or its priority is greater than max.
+ * - Else returns idle_task.
+ */
+static struct task_struct *
+pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *max)
+{
+	struct task_struct *class_pick, *cookie_pick;
+	unsigned long cookie = rq->core->core_cookie;
+
+	class_pick = class->pick_task(rq);
+	if (!class_pick)
+		return NULL;
+
+	if (!cookie) {
+		/*
+		 * If class_pick is tagged, return it only if it has
+		 * higher priority than max.
+		 */
+		if (max && class_pick->core_cookie &&
+		    prio_less(class_pick, max))
+			return idle_sched_class.pick_task(rq);
+
+		return class_pick;
+	}
+
+	/*
+	 * If class_pick is idle or matches cookie, return early.
+	 */
+	if (cookie_equals(class_pick, cookie))
+		return class_pick;
+
+	cookie_pick = sched_core_find(rq, cookie);
+
+	/*
+	 * If class > max && class > cookie, it is the highest priority task on
+	 * the core (so far) and it must be selected, otherwise we must go with
+	 * the cookie pick in order to satisfy the constraint.
+	 */
+	if (prio_less(cookie_pick, class_pick) &&
+	    (!max || prio_less(max, class_pick)))
+		return class_pick;
+
+	return cookie_pick;
+}
+
+static struct task_struct *
+pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+{
+	struct task_struct *next, *max = NULL;
+	const struct sched_class *class;
+	const struct cpumask *smt_mask;
+	bool need_sync;
+	int i, j, cpu;
+
+	if (!sched_core_enabled(rq))
+		return __pick_next_task(rq, prev, rf);
+
+	cpu = cpu_of(rq);
+
+	/* Stopper task is switching into idle, no need core-wide selection. */
+	if (cpu_is_offline(cpu)) {
+		/*
+		 * Reset core_pick so that we don't enter the fastpath when
+		 * coming online. core_pick would already be migrated to
+		 * another cpu during offline.
+		 */
+		rq->core_pick = NULL;
+		return __pick_next_task(rq, prev, rf);
+	}
+
+	/*
+	 * If there were no {en,de}queues since we picked (IOW, the task
+	 * pointers are all still valid), and we haven't scheduled the last
+	 * pick yet, do so now.
+	 *
+	 * rq->core_pick can be NULL if no selection was made for a CPU because
+	 * it was either offline or went offline during a sibling's core-wide
+	 * selection. In this case, do a core-wide selection.
+	 */
+	if (rq->core->core_pick_seq == rq->core->core_task_seq &&
+	    rq->core->core_pick_seq != rq->core_sched_seq &&
+	    rq->core_pick) {
+		WRITE_ONCE(rq->core_sched_seq, rq->core->core_pick_seq);
+
+		next = rq->core_pick;
+		if (next != prev) {
+			put_prev_task(rq, prev);
+			set_next_task(rq, next);
+		}
+
+		rq->core_pick = NULL;
+		return next;
+	}
+
+	put_prev_task_balance(rq, prev, rf);
+
+	smt_mask = cpu_smt_mask(cpu);
+
+	/*
+	 * core->core_task_seq, core->core_pick_seq, rq->core_sched_seq
+	 *
+	 * @task_seq guards the task state ({en,de}queues)
+	 * @pick_seq is the @task_seq we did a selection on
+	 * @sched_seq is the @pick_seq we scheduled
+	 *
+	 * However, preemptions can cause multiple picks on the same task set.
+	 * 'Fix' this by also increasing @task_seq for every pick.
+	 */
+	rq->core->core_task_seq++;
+	need_sync = !!rq->core->core_cookie;
+
+	/* reset state */
+	rq->core->core_cookie = 0UL;
+	for_each_cpu(i, smt_mask) {
+		struct rq *rq_i = cpu_rq(i);
+
+		rq_i->core_pick = NULL;
+
+		if (rq_i->core_forceidle) {
+			need_sync = true;
+			rq_i->core_forceidle = false;
+		}
+
+		if (i != cpu)
+			update_rq_clock(rq_i);
+	}
+
+	/*
+	 * Try and select tasks for each sibling in decending sched_class
+	 * order.
+	 */
+	for_each_class(class) {
+again:
+		for_each_cpu_wrap(i, smt_mask, cpu) {
+			struct rq *rq_i = cpu_rq(i);
+			struct task_struct *p;
+
+			if (rq_i->core_pick)
+				continue;
+
+			/*
+			 * If this sibling doesn't yet have a suitable task to
+			 * run; ask for the most elegible task, given the
+			 * highest priority task already selected for this
+			 * core.
+			 */
+			p = pick_task(rq_i, class, max);
+			if (!p) {
+				/*
+				 * If there weren't no cookies; we don't need to
+				 * bother with the other siblings.
+				 * If the rest of the core is not running a tagged
+				 * task, i.e.  need_sync == 0, and the current CPU
+				 * which called into the schedule() loop does not
+				 * have any tasks for this class, skip selecting for
+				 * other siblings since there's no point. We don't skip
+				 * for RT/DL because that could make CFS force-idle RT.
+				 */
+				if (i == cpu && !need_sync && class == &fair_sched_class)
+					goto next_class;
+
+				continue;
+			}
+
+			/*
+			 * Optimize the 'normal' case where there aren't any
+			 * cookies and we don't need to sync up.
+			 */
+			if (i == cpu && !need_sync && !p->core_cookie) {
+				next = p;
+				goto done;
+			}
+
+			rq_i->core_pick = p;
+
+			/*
+			 * If this new candidate is of higher priority than the
+			 * previous; and they're incompatible; we need to wipe
+			 * the slate and start over. pick_task makes sure that
+			 * p's priority is more than max if it doesn't match
+			 * max's cookie.
+			 *
+			 * NOTE: this is a linear max-filter and is thus bounded
+			 * in execution time.
+			 */
+			if (!max || !cookie_match(max, p)) {
+				struct task_struct *old_max = max;
+
+				rq->core->core_cookie = p->core_cookie;
+				max = p;
+
+				if (old_max) {
+					for_each_cpu(j, smt_mask) {
+						if (j == i)
+							continue;
+
+						cpu_rq(j)->core_pick = NULL;
+					}
+					goto again;
+				} else {
+					/*
+					 * Once we select a task for a cpu, we
+					 * should not be doing an unconstrained
+					 * pick because it might starve a task
+					 * on a forced idle cpu.
+					 */
+					need_sync = true;
+				}
+
+			}
+		}
+next_class:;
+	}
+
+	rq->core->core_pick_seq = rq->core->core_task_seq;
+	next = rq->core_pick;
+	rq->core_sched_seq = rq->core->core_pick_seq;
+
+	/* Something should have been selected for current CPU */
+	WARN_ON_ONCE(!next);
+
+	/*
+	 * Reschedule siblings
+	 *
+	 * NOTE: L1TF -- at this point we're no longer running the old task and
+	 * sending an IPI (below) ensures the sibling will no longer be running
+	 * their task. This ensures there is no inter-sibling overlap between
+	 * non-matching user state.
+	 */
+	for_each_cpu(i, smt_mask) {
+		struct rq *rq_i = cpu_rq(i);
+
+		/*
+		 * An online sibling might have gone offline before a task
+		 * could be picked for it, or it might be offline but later
+		 * happen to come online, but its too late and nothing was
+		 * picked for it.  That's Ok - it will pick tasks for itself,
+		 * so ignore it.
+		 */
+		if (!rq_i->core_pick)
+			continue;
+
+		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running)
+			rq_i->core_forceidle = true;
+
+		if (i == cpu) {
+			rq_i->core_pick = NULL;
+			continue;
+		}
+
+		/* Did we break L1TF mitigation requirements? */
+		WARN_ON_ONCE(!cookie_match(next, rq_i->core_pick));
+
+		if (rq_i->curr == rq_i->core_pick) {
+			rq_i->core_pick = NULL;
+			continue;
+		}
+
+		resched_curr(rq_i);
+	}
+
+done:
+	set_next_task(rq, next);
+	return next;
+}
 
 static inline void sched_core_cpu_starting(unsigned int cpu)
 {
@@ -5103,6 +5391,12 @@ static inline void sched_core_cpu_starting(unsigned int cpu)
 
 static inline void sched_core_cpu_starting(unsigned int cpu) {}
 
+static struct task_struct *
+pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
+{
+	return __pick_next_task(rq, prev, rf);
+}
+
 #endif /* CONFIG_SCHED_CORE */
 
 /*
@@ -7999,7 +8293,12 @@ void __init sched_init(void)
 
 #ifdef CONFIG_SCHED_CORE
 		rq->core = NULL;
+		rq->core_pick = NULL;
 		rq->core_enabled = 0;
+		rq->core_tree = RB_ROOT;
+		rq->core_forceidle = false;
+
+		rq->core_cookie = 0UL;
 #endif
 	}
 
kernel/sched/sched.h +5 -1
index 8ee0ca8ee5c3..63b28e1843ee 100644
@@ -1065,11 +1065,16 @@ struct rq {
 #ifdef CONFIG_SCHED_CORE
 	/* per rq */
 	struct rq		*core;
+	struct task_struct	*core_pick;
 	unsigned int		core_enabled;
+	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
+	unsigned char		core_forceidle;
 
 	/* shared state */
 	unsigned int		core_task_seq;
+	unsigned int		core_pick_seq;
+	unsigned long		core_cookie;
 #endif
 };
 
@@ -1977,7 +1982,6 @@ static inline void put_prev_task(struct rq *rq, struct task_struct *prev)
 
 static inline void set_next_task(struct rq *rq, struct task_struct *next)
 {
-	WARN_ON_ONCE(rq->curr != next);
 	next->sched_class->set_next_task(rq, next, false);
 }
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#25 lore raw

PATCH 8/32 sched/fair: Fix forced idle sibling starvation corner case

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

If there is only one long running local task and the sibling is forced idle, it might not get a chance to run until a schedule event happens on any cpu in the core.

So we check for this condition during a tick to see if a sibling is starved and then give it a chance to schedule.

---
 kernel/sched/core.c  | 15 ++++++++-------
 kernel/sched/fair.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  2 +-
 3 files changed, 49 insertions(+), 8 deletions(-)
kernel/sched/core.c +8 -7
index 1bd0b0bbb040..52d0e83072a4 100644
@@ -5206,16 +5206,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 
 	/* reset state */
 	rq->core->core_cookie = 0UL;
+	if (rq->core->core_forceidle) {
+		need_sync = true;
+		rq->core->core_forceidle = false;
+	}
 	for_each_cpu(i, smt_mask) {
 		struct rq *rq_i = cpu_rq(i);
 
 		rq_i->core_pick = NULL;
 
-		if (rq_i->core_forceidle) {
-			need_sync = true;
-			rq_i->core_forceidle = false;
-		}
-
 		if (i != cpu)
 			update_rq_clock(rq_i);
 	}
@@ -5335,8 +5334,10 @@ next_class:;
 		if (!rq_i->core_pick)
 			continue;
 
-		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running)
-			rq_i->core_forceidle = true;
+		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running &&
+		    !rq_i->core->core_forceidle) {
+			rq_i->core->core_forceidle = true;
+		}
 
 		if (i == cpu) {
 			rq_i->core_pick = NULL;
kernel/sched/fair.c +40 -0
index f53681cd263e..42965c4fd71f 100644
@@ -10692,6 +10692,44 @@ static void rq_offline_fair(struct rq *rq)
 
 #endif /* CONFIG_SMP */
 
+#ifdef CONFIG_SCHED_CORE
+static inline bool
+__entity_slice_used(struct sched_entity *se, int min_nr_tasks)
+{
+	u64 slice = sched_slice(cfs_rq_of(se), se);
+	u64 rtime = se->sum_exec_runtime - se->prev_sum_exec_runtime;
+
+	return (rtime * min_nr_tasks > slice);
+}
+
+#define MIN_NR_TASKS_DURING_FORCEIDLE	2
+static inline void task_tick_core(struct rq *rq, struct task_struct *curr)
+{
+	if (!sched_core_enabled(rq))
+		return;
+
+	/*
+	 * If runqueue has only one task which used up its slice and
+	 * if the sibling is forced idle, then trigger schedule to
+	 * give forced idle task a chance.
+	 *
+	 * sched_slice() considers only this active rq and it gets the
+	 * whole slice. But during force idle, we have siblings acting
+	 * like a single runqueue and hence we need to consider runnable
+	 * tasks on this cpu and the forced idle cpu. Ideally, we should
+	 * go through the forced idle rq, but that would be a perf hit.
+	 * We can assume that the forced idle cpu has atleast
+	 * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
+	 * if we need to give up the cpu.
+	 */
+	if (rq->core->core_forceidle && rq->cfs.nr_running == 1 &&
+	    __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
+		resched_curr(rq);
+}
+#else
+static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
+#endif
+
 /*
  * scheduler tick hitting a task of our scheduling class.
  *
@@ -10715,6 +10753,8 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
 
 	update_misfit_status(curr, rq);
 	update_overutilized_status(task_rq(curr));
+
+	task_tick_core(rq, curr);
 }
 
 /*
kernel/sched/sched.h +1 -1
index 63b28e1843ee..be656ca8693d 100644
@@ -1069,12 +1069,12 @@ struct rq {
 	unsigned int		core_enabled;
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
-	unsigned char		core_forceidle;
 
 	/* shared state */
 	unsigned int		core_task_seq;
 	unsigned int		core_pick_seq;
 	unsigned long		core_cookie;
+	unsigned char		core_forceidle;
 #endif
 };
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#26 lore raw

Re: [PATCH -tip 08/32] sched/fair: Fix forced idle sibling starvation corner case

Quoted 63 lines · Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

If there is only one long running local task and the sibling is forced idle, it might not get a chance to run until a schedule event happens on any cpu in the core.

So we check for this condition during a tick to see if a sibling is starved and then give it a chance to schedule.

---
 kernel/sched/core.c  | 15 ++++++++-------
 kernel/sched/fair.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  2 +-
 3 files changed, 49 insertions(+), 8 deletions(-)
kernel/sched/core.c +8 -7
index 1bd0b0bbb040..52d0e83072a4 100644
@@ -5206,16 +5206,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 
 	/* reset state */
 	rq->core->core_cookie = 0UL;
+	if (rq->core->core_forceidle) {
+		need_sync = true;
+		rq->core->core_forceidle = false;
+	}
 	for_each_cpu(i, smt_mask) {
 		struct rq *rq_i = cpu_rq(i);
 
 		rq_i->core_pick = NULL;
 
-		if (rq_i->core_forceidle) {
-			need_sync = true;
-			rq_i->core_forceidle = false;
-		}
-
 		if (i != cpu)
 			update_rq_clock(rq_i);
 	}
@@ -5335,8 +5334,10 @@ next_class:;
 		if (!rq_i->core_pick)
 			continue;
 
-		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running)
-			rq_i->core_forceidle = true;
+		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running &&
+		    !rq_i->core->core_forceidle) {
+			rq_i->core->core_forceidle = true;
+		}
 
 		if (i == cpu) {
 			rq_i->core_pick = NULL;
kernel/sched/fair.c +0 -0
index f53681cd263e..42965c4fd71f 100644

kernel/sched/fair.c static void rq_offline_fair(struct rq *rq) · line 10699

Quoted 8 lines
kernel/sched/fair.c +5 -0
@@ -10692,6 +10692,44 @@ static void rq_offline_fair(struct rq *rq)
 
 #endif /* CONFIG_SMP */
 
+#ifdef CONFIG_SCHED_CORE
+static inline bool
+__entity_slice_used(struct sched_entity *se, int min_nr_tasks)
+{
+	u64 slice = sched_slice(cfs_rq_of(se), se);

I wonder if the definition of sched_slice() should be revisited for core scheduling?

Should we use sched_slice = sched_slice / cpumask_weight(smt_mask)? Would that resolve the issue your seeing? Effectively we need to answer if two sched core siblings should be treated as executing one large slice?

Balbir Singh.

#27 lore raw

PATCH 9/32 sched/fair: Snapshot the min_vruntime of CPUs on force idle

In reply to Joel Fernandes (Google)

During force-idle, we end up doing cross-cpu comparison of vruntimes during pick_next_task. If we simply compare (vruntime-min_vruntime) across CPUs, and if the CPUs only have 1 task each, we will always end up comparing 0 with 0 and pick just one of the tasks all the time. This starves the task that was not picked. To fix this, take a snapshot of the min_vruntime when entering force idle and use it for comparison. This min_vruntime snapshot will only be used for cross-CPU vruntime comparison, and nothing else.

This resolves several performance issues that were seen in ChromeOS audio usecase.

NOTE: Note, this patch will be improved in a later patch. It is just
kept here as the basis for the later patch and to make rebasing easier. Further, it may make reverting the improvement easier in case the improvement causes any regression.

---
 kernel/sched/core.c  | 33 ++++++++++++++++++++-------------
 kernel/sched/fair.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 kernel/sched/sched.h |  5 +++++
 3 files changed, 65 insertions(+), 13 deletions(-)
kernel/sched/core.c +20 -13
index 52d0e83072a4..4ee4902c2cf5 100644
@@ -115,19 +115,8 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)
 	if (pa == -1) /* dl_prio() doesn't work because of stop_class above */
 		return !dl_time_before(a->dl.deadline, b->dl.deadline);
 
-	if (pa == MAX_RT_PRIO + MAX_NICE)  { /* fair */
-		u64 vruntime = b->se.vruntime;
-
-		/*
-		 * Normalize the vruntime if tasks are in different cpus.
-		 */
-		if (task_cpu(a) != task_cpu(b)) {
-			vruntime -= task_cfs_rq(b)->min_vruntime;
-			vruntime += task_cfs_rq(a)->min_vruntime;
-		}
-
-		return !((s64)(a->se.vruntime - vruntime) <= 0);
-	}
+	if (pa == MAX_RT_PRIO + MAX_NICE)	/* fair */
+		return cfs_prio_less(a, b);
 
 	return false;
 }
@@ -5144,6 +5133,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	struct task_struct *next, *max = NULL;
 	const struct sched_class *class;
 	const struct cpumask *smt_mask;
+	bool fi_before = false;
 	bool need_sync;
 	int i, j, cpu;
 
@@ -5208,6 +5198,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	rq->core->core_cookie = 0UL;
 	if (rq->core->core_forceidle) {
 		need_sync = true;
+		fi_before = true;
 		rq->core->core_forceidle = false;
 	}
 	for_each_cpu(i, smt_mask) {
@@ -5219,6 +5210,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			update_rq_clock(rq_i);
 	}
 
+	/* Reset the snapshot if core is no longer in force-idle. */
+	if (!fi_before) {
+		for_each_cpu(i, smt_mask) {
+			struct rq *rq_i = cpu_rq(i);
+			rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime;
+		}
+	}
+
 	/*
 	 * Try and select tasks for each sibling in decending sched_class
 	 * order.
@@ -5355,6 +5354,14 @@ next_class:;
 		resched_curr(rq_i);
 	}
 
+	/* Snapshot if core is in force-idle. */
+	if (!fi_before && rq->core->core_forceidle) {
+		for_each_cpu(i, smt_mask) {
+			struct rq *rq_i = cpu_rq(i);
+			rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime;
+		}
+	}
+
 done:
 	set_next_task(rq, next);
 	return next;
kernel/sched/fair.c +40 -0
index 42965c4fd71f..de82f88ba98c 100644
@@ -10726,6 +10726,46 @@ static inline void task_tick_core(struct rq *rq, struct task_struct *curr)
 	    __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
 		resched_curr(rq);
 }
+
+bool cfs_prio_less(struct task_struct *a, struct task_struct *b)
+{
+	bool samecpu = task_cpu(a) == task_cpu(b);
+	struct sched_entity *sea = &a->se;
+	struct sched_entity *seb = &b->se;
+	struct cfs_rq *cfs_rqa;
+	struct cfs_rq *cfs_rqb;
+	s64 delta;
+
+	if (samecpu) {
+		/* vruntime is per cfs_rq */
+		while (!is_same_group(sea, seb)) {
+			int sea_depth = sea->depth;
+			int seb_depth = seb->depth;
+			if (sea_depth >= seb_depth)
+				sea = parent_entity(sea);
+			if (sea_depth <= seb_depth)
+				seb = parent_entity(seb);
+		}
+
+		delta = (s64)(sea->vruntime - seb->vruntime);
+		goto out;
+	}
+
+	/* crosscpu: compare root level se's vruntime to decide priority */
+	while (sea->parent)
+		sea = sea->parent;
+	while (seb->parent)
+		seb = seb->parent;
+
+	cfs_rqa = sea->cfs_rq;
+	cfs_rqb = seb->cfs_rq;
+
+	/* normalize vruntime WRT their rq's base */
+	delta = (s64)(sea->vruntime - seb->vruntime) +
+		(s64)(cfs_rqb->min_vruntime_fi - cfs_rqa->min_vruntime_fi);
+out:
+	return delta > 0;
+}
 #else
 static inline void task_tick_core(struct rq *rq, struct task_struct *curr) {}
 #endif
kernel/sched/sched.h +5 -0
index be656ca8693d..d934cc51acf1 100644
@@ -517,6 +517,9 @@ struct cfs_rq {
 
 	u64			exec_clock;
 	u64			min_vruntime;
+#ifdef CONFIG_SCHED_CORE
+	u64			min_vruntime_fi;
+#endif
 #ifndef CONFIG_64BIT
 	u64			min_vruntime_copy;
 #endif
@@ -1130,6 +1133,8 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 	return &rq->__lock;
 }
 
+bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
+
 #else /* !CONFIG_SCHED_CORE */
 
 static inline bool sched_core_enabled(struct rq *rq)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#28 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Quoted 9 lines · Joel Fernandes (Google)

During force-idle, we end up doing cross-cpu comparison of vruntimes during pick_next_task. If we simply compare (vruntime-min_vruntime) across CPUs, and if the CPUs only have 1 task each, we will always end up comparing 0 with 0 and pick just one of the tasks all the time. This starves the task that was not picked. To fix this, take a snapshot of the min_vruntime when entering force idle and use it for comparison. This min_vruntime snapshot will only be used for cross-CPU vruntime comparison, and nothing else.

Quoted 8 lines

This resolves several performance issues that were seen in ChromeOS audio usecase.

NOTE: Note, this patch will be improved in a later patch. It is just
kept here as the basis for the later patch and to make rebasing easier. Further, it may make reverting the improvement easier in case the improvement causes any regression.

This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?

Balbir Singh.

#29 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Hi Balbir,

Quoted 3 lines · Balbir Singh

This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?

Do you mean to have a core wide min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:

https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/

Thanks,
Vineeth

#30 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Quoted 6 lines · Vineeth Pillai

Hi Balbir,

Quoted 3 lines · Balbir Singh

This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?

Quoted 8 lines

Do you mean to have a core wide min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:

https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/

One of the equations in the link is

">From which immediately follows that:

          T_k + T_l
  S_k+l = ---------                                       (13)
          W_k + W_l

On which we can define a combined lag:

lag_k+l(i) := S_k+l - s_i (14)

And that gives us the tools to compare tasks across a combined runqueue. "

S_k+l reads like rq->core->vruntime, but it sounds like the equivalent of rq->core->vruntime is updated when we enter forced idle as opposed to all the time.

Balbir

maintainer
#31 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Quoted 26 lines · Balbir Singh
Quoted 6 lines · Vineeth Pillai

Hi Balbir,

Quoted 3 lines · Balbir Singh

This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?

Quoted 8 lines

Do you mean to have a core wide min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:

https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/

One of the equations in the link is

">From which immediately follows that:

          T_k + T_l
  S_k+l = ---------                                       (13)
          W_k + W_l

On which we can define a combined lag:

Quoted 8 lines

lag_k+l(i) := S_k+l - s_i (14)

And that gives us the tools to compare tasks across a combined runqueue. "

S_k+l reads like rq->core->vruntime, but it sounds like the equivalent of rq->core->vruntime is updated when we enter forced idle as opposed to all the time.

Yes, but actually computing and maintaining it is hella hard. Try it with the very first example in that email (the infeasible weight scenario) and tell me how it works for you ;-)

Also note that the text below (6) mentions dynamic, then look up the EEVDF paper which describes some of the dynamics -- the paper is incomplete and contains a bug, I forget if it ever got updated or if there's another paper that completes it (the BQF I/O scheduler started from that and fixed it).

I'm not saying it cannot be done, I'm just saying it is really rather involved and probably not worth it.

The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).

Also, I think the code as proposed here relies on SMT2 and is buggered for SMT3+. Now, that second link above describes means of making SMT3+ work, but we're not there yet.

#32 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Quoted 32 lines · Peter Zijlstra
Quoted 31 lines · Balbir Singh
Quoted 6 lines · Vineeth Pillai

Hi Balbir,

Quoted 3 lines · Balbir Singh

This seems cumbersome, is there no way to track the min_vruntime via rq->core->min_vruntime?

Quoted 8 lines

Do you mean to have a core wide min_vruntime? We had a similar approach from v3 to v7 and it had major issues which broke the assumptions of cfs. There were some lengthy discussions and Peter explained in-depth about the issues:

https://lwn.net/ml/linux-kernel/[email protected]/
https://lwn.net/ml/linux-kernel/[email protected]/

One of the equations in the link is

">From which immediately follows that:

          T_k + T_l
  S_k+l = ---------                                       (13)
          W_k + W_l

On which we can define a combined lag:

lag_k+l(i) := S_k+l - s_i (14)

And that gives us the tools to compare tasks across a combined runqueue. "

Quoted 8 lines
Quoted 3 lines

S_k+l reads like rq->core->vruntime, but it sounds like the equivalent of rq->core->vruntime is updated when we enter forced idle as opposed to all the time.

Yes, but actually computing and maintaining it is hella hard. Try it with the very first example in that email (the infeasible weight scenario) and tell me how it works for you ;-)

OK, I was hoping it could be done in the new RBTree's enqueue/dequeue, but yes I've not implemented it and I should go back and take a look at the first example again.

Quoted 5 lines

Also note that the text below (6) mentions dynamic, then look up the EEVDF paper which describes some of the dynamics -- the paper is incomplete and contains a bug, I forget if it ever got updated or if there's another paper that completes it (the BQF I/O scheduler started from that and fixed it).

I see, I am yet to read the EEVDF paper, but now I am out on a tangent :)

Quoted 4 lines

I'm not saying it cannot be done, I'm just saying it is really rather involved and probably not worth it.

Fair enough

Quoted 7 lines

The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).

I'll reread (6) and (7), I am trying to understand forced idle and contention together, from what I understand of the patches, there is

1. two tasks that are core scheduled, in that case vruntime works as expected on each CPU, but we need to compare their combined vrtuntime against other tasks on each CPU respectively for them to be selected/chosen?
2. When one of the tasks selected is a part of the core scheduling group and the other CPU does not select a core scheduled tasks, we need to ask ourselves if that CPU should force idle and that's where this logic comes into play?

Quoted 3 lines

Also, I think the code as proposed here relies on SMT2 and is buggered for SMT3+. Now, that second link above describes means of making SMT3+ work, but we're not there yet.

Thanks,
Balbir Singh.

maintainer
#33 lore raw

Re: [PATCH -tip 09/32] sched/fair: Snapshot the min_vruntime of CPUs on force idle

Quoted 10 lines · Balbir Singh
Quoted 7 lines · Peter Zijlstra

The basic observation the current approach relies on is that al that faffery basically boils down to the fact that vruntime only means something when there is contention. And that only the progression is important not the actual value. That is, this is all fundamentally a differential equation and our integration constant is meaningless (also embodied in (7)).

Quoted 2 lines

I'll reread (6) and (7), I am trying to understand forced idle and contention together, from what I understand of the patches, there is

When we force-idle there is contention by definition; there's a task that wanted to run, but couldn't.

Quoted 4 lines

1. two tasks that are core scheduled, in that case vruntime works as expected on each CPU, but we need to compare their combined vrtuntime against other tasks on each CPU respectively for them to be selected/chosen?

We need to compare across CPUs when the cookies don't match. This is required to avoid starving one or the other.

Quoted 4 lines

2. When one of the tasks selected is a part of the core scheduling group and the other CPU does not select a core scheduled tasks, we need to ask ourselves if that CPU should force idle and that's where this logic comes into play?

When one CPU selects a cookie task, and the other CPU cannot find a matching task, it must go idle (as idle matches everyone). This is the basic core-scheduling constraint.

So suppose you have two tasks, A and B, both with a cookie, but not matching.

Normal scheduling would run A and B concurrent on the two siblings. Core scheduling obviously cannot do this. When we pick A, the other CPU is not allowed to run B and thus will have to be forced idle and vice-versa.

The next problem is avoiding starvation. Assuming equal weight between the tasks, we'd want to end up running A and B in alternating cycles.

This means having to compare runtimes between A and B, but when they're on different runqueues the actual vruntime values can be wildly divergent and cannot be reasily compared (the integration constant is meaningless but really annoying ;-).

We also cannot use min_vruntime (which is the same as the task vruntime when there is only a single task), because then you cannot observe progress. The difference between min_vruntime and the task runtime is always 0, so you can't tell who just ran and who got starved.

This is where our snapshots come in play, we snapshot vruntime after task selection (before running), such that at the next pick we can tell who made progress and who got starved.

By marking the vruntime of both runqueues at the same point in time we basically normalize away that integration constant. You effectively reset the vruntime to 0 (through (7), but without iterating all the tasks and adjusting it).

Does that make sense?

Once you get this, read that second email linked.

#34 lore raw

PATCH 10/32 sched: Fix priority inversion of cookied task with sibling

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

---
 kernel/sched/core.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +16 -9
index 4ee4902c2cf5..53af817740c0 100644
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	need_sync = !!rq->core->core_cookie;
 
 	/* reset state */
+reset:
 	rq->core->core_cookie = 0UL;
 	if (rq->core->core_forceidle) {
 		need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				/*
 				 * If there weren't no cookies; we don't need to
 				 * bother with the other siblings.
-				 * If the rest of the core is not running a tagged
-				 * task, i.e.  need_sync == 0, and the current CPU
-				 * which called into the schedule() loop does not
-				 * have any tasks for this class, skip selecting for
-				 * other siblings since there's no point. We don't skip
-				 * for RT/DL because that could make CFS force-idle RT.
 				 */
-				if (i == cpu && !need_sync && class == &fair_sched_class)
+				if (i == cpu && !need_sync)
 					goto next_class;
 
 				continue;
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies
+					 * involved. We may have skipped
+					 * non-empty higher priority classes on
+					 * siblings, which are empty on this
+					 * CPU, so start over.
+					 */
+					need_sync = true;
+					goto reset;
+				}
+
 				next = p;
 				goto done;
 			}
@@ -5299,7 +5307,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 					 */
 					need_sync = true;
 				}
-
 			}
 		}
 next_class:;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#35 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

Quoted 25 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Quoted 7 lines

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

The changelog is hard to read, it refers to above if(), whereas there is no code snippet in the changelog. Also, from what I can see following the series, p->core_cookie is not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.

Balbir

#36 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

Quoted 7 lines
Quoted 4 lines

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

The changelog is hard to read, it refers to above if(), whereas there is no code snippet in the changelog.

Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]

I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coresched

Quoted 3 lines

Also, from what I can see following
the series, p->core_cookie is not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.

The interface patches for core_cookie are added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.

---8<-----------------------

From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with sibling

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Fix this issue by forcing need_sync and restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

---
 kernel/sched/core.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +16 -9
index 4ee4902c2cf5..53af817740c0 100644
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	need_sync = !!rq->core->core_cookie;
 
 	/* reset state */
+reset:
 	rq->core->core_cookie = 0UL;
 	if (rq->core->core_forceidle) {
 		need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				/*
 				 * If there weren't no cookies; we don't need to
 				 * bother with the other siblings.
-				 * If the rest of the core is not running a tagged
-				 * task, i.e.  need_sync == 0, and the current CPU
-				 * which called into the schedule() loop does not
-				 * have any tasks for this class, skip selecting for
-				 * other siblings since there's no point. We don't skip
-				 * for RT/DL because that could make CFS force-idle RT.
 				 */
-				if (i == cpu && !need_sync && class == &fair_sched_class)
+				if (i == cpu && !need_sync)
 					goto next_class;
 
 				continue;
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies
+					 * involved. We may have skipped
+					 * non-empty higher priority classes on
+					 * siblings, which are empty on this
+					 * CPU, so start over.
+					 */
+					need_sync = true;
+					goto reset;
+				}
+
 				next = p;
 				goto done;
 			}
@@ -5299,7 +5307,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 					 */
 					need_sync = true;
 				}
-
 			}
 		}
 next_class:;
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
#37 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

Quoted 44 lines · Joel Fernandes
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

Quoted 7 lines
Quoted 4 lines

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

The changelog is hard to read, it refers to above if(), whereas there is no code snippet in the changelog.

Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]

I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coresched

Quoted 8 lines
Quoted 3 lines

Also, from what I can see following
the series, p->core_cookie is not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.

The interface patches for core_cookie are added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.

Some comments below to help make the code easier to understand

Quoted 71 lines

---8<-----------------------

From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with sibling

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Fix this issue by forcing need_sync and restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

---
 kernel/sched/core.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +2 -7
index 4ee4902c2cf5..53af817740c0 100644
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	need_sync = !!rq->core->core_cookie;
 
 	/* reset state */
+reset:
 	rq->core->core_cookie = 0UL;
 	if (rq->core->core_forceidle) {
 		need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				/*
 				 * If there weren't no cookies; we don't need to
 				 * bother with the other siblings.
-				 * If the rest of the core is not running a tagged
-				 * task, i.e.  need_sync == 0, and the current CPU
-				 * which called into the schedule() loop does not
-				 * have any tasks for this class, skip selecting for
-				 * other siblings since there's no point. We don't skip
-				 * for RT/DL because that could make CFS force-idle RT.
 				 */
-				if (i == cpu && !need_sync && class == &fair_sched_class)
+				if (i == cpu && !need_sync)
 					goto next_class;
 
 				continue;

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261

Quoted 9 lines
kernel/sched/core.c +5 -1
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies

This is not entirely true, need_sync is a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has a core_cookie? The term cookie is quite overloaded when used in the context of core vs task.

Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?

Balbir Singh.

maintainer
#38 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

Quoted 7 lines · Balbir Singh
Quoted 7 lines
kernel/sched/core.c +2 -1
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
Quoted 8 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261

Quoted 3 lines
kernel/sched/core.c +3 -0
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies

This is not entirely true, need_sync is a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has a core_cookie? The term cookie is quite overloaded when used in the context of core vs task.

Nah, its the same. So each task gets a cookie to identify the 'group' of tasks (possibly just itself) it is allowed to share a core with.

When we to core task selection, the core gets assigned the cookie of the group it will run, same thing.

Quoted 4 lines

Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?

This whole patch is about eliding code-wide task selection when it is not required. IOW an optimization.

When there wasn't a core cookie (IOW, the previous task selection wasn't core wide and limited) and the task we just selected for our own CPU also didn't have a cookie (IOW it doesn't have to be core-wide) we can skip the core wide task selection and schedule just this CPU and call it a day.

The logic was subtly wrong, this patch fixes it. A next patch completely rewrites it again to make it far simpler again. Don't spend time trying to understand this patch (unless you're _that_ kind of person ;-) but instead apply the whole thing and look at the resulting pick_next_task() function.

#39 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

Quoted 37 lines · Peter Zijlstra
Quoted 7 lines · Balbir Singh
Quoted 7 lines
kernel/sched/core.c +2 -1
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
Quoted 8 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261

Quoted 3 lines
kernel/sched/core.c +3 -0
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies

This is not entirely true, need_sync is a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has a core_cookie? The term cookie is quite overloaded when used in the context of core vs task.

Nah, its the same. So each task gets a cookie to identify the 'group' of tasks (possibly just itself) it is allowed to share a core with.

When we to core task selection, the core gets assigned the cookie of the group it will run, same thing.

Quoted 4 lines

Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?

This whole patch is about eliding code-wide task selection when it is not required. IOW an optimization.

When there wasn't a core cookie (IOW, the previous task selection wasn't core wide and limited) and the task we just selected for our own CPU also didn't have a cookie (IOW it doesn't have to be core-wide) we can skip the core wide task selection and schedule just this CPU and call it a day.

Quoted 5 lines

The logic was subtly wrong, this patch fixes it. A next patch completely rewrites it again to make it far simpler again. Don't spend time trying to understand this patch (unless you're _that_ kind of person ;-) but instead apply the whole thing and look at the resulting pick_next_task() function.

Thanks, I'll look at the git tree and see what the final outcome looks like.

Balbir Singh.

#40 lore raw

Re: [PATCH -tip 10/32] sched: Fix priority inversion of cookied task with sibling

In reply to Balbir Singh
Quoted 142 lines · Balbir Singh
Quoted 44 lines · Joel Fernandes
Quoted 29 lines · Balbir Singh
Quoted 28 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

Say schedule() runs on rq0. Now, it will enter the above loop and pick_task(RT) will return NULL for 'p'. It will enter the above if() block and see that need_sync == false and will skip RT entirely.

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

Quoted 7 lines
Quoted 4 lines

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

The changelog is hard to read, it refers to above if(), whereas there is no code snippet in the changelog.

Yeah sorry, it comes from this email where I described the issue:
http://lore.kernel.org/r/[email protected]

I corrected the changelog and appended the patch below. Also pushed it to:
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=coresched

Quoted 8 lines
Quoted 3 lines

Also, from what I can see following
the series, p->core_cookie is not yet set anywhere (unless I missed it), so fixing it in here did not make sense just reading the series.

The interface patches for core_cookie are added later, that's how it is. The infrastructure comes first here. It would also not make sense to add interface first as well so I think the current ordering is fine.

Some comments below to help make the code easier to understand

Quoted 71 lines

---8<-----------------------

From: Peter Zijlstra <[email protected]>
Subject: [PATCH] sched: Fix priority inversion of cookied task with sibling

The rationale is as follows. In the core-wide pick logic, even if need_sync == false, we need to go look at other CPUs (non-local CPUs) to see if they could be running RT.

Say the RQs in a particular core look like this:
Let CFS1 and CFS2 be 2 tagged CFS tags. Let RT1 be an untagged RT task.

rq0 rq1
CFS1 (tagged) RT1 (not tag)
CFS2 (tagged)

The end result of the selection will be (say prio(CFS1) > prio(CFS2)):
rq0 rq1
CFS1 IDLE

When it should have selected:
rq0 r1
IDLE RT

Fix this issue by forcing need_sync and restarting the search if a cookied task was discovered. This will avoid this optimization from making incorrect picks.

Joel saw this issue on real-world usecases in ChromeOS where an RT task gets constantly force-idled and breaks RT. Lets cure it.

NOTE: This problem will be fixed differently in a later patch. It just
kept here for reference purposes about this issue, and to make applying later patches easier.

---
 kernel/sched/core.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
kernel/sched/core.c +2 -7
index 4ee4902c2cf5..53af817740c0 100644
@@ -5195,6 +5195,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	need_sync = !!rq->core->core_cookie;
 
 	/* reset state */
+reset:
 	rq->core->core_cookie = 0UL;
 	if (rq->core->core_forceidle) {
 		need_sync = true;
@@ -5242,14 +5243,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				/*
 				 * If there weren't no cookies; we don't need to
 				 * bother with the other siblings.
-				 * If the rest of the core is not running a tagged
-				 * task, i.e.  need_sync == 0, and the current CPU
-				 * which called into the schedule() loop does not
-				 * have any tasks for this class, skip selecting for
-				 * other siblings since there's no point. We don't skip
-				 * for RT/DL because that could make CFS force-idle RT.
 				 */
-				if (i == cpu && !need_sync && class == &fair_sched_class)
+				if (i == cpu && !need_sync)
 					goto next_class;
 
 				continue;

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5261

Quoted 9 lines
kernel/sched/core.c +5 -1
@@ -5259,7 +5254,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * Optimize the 'normal' case where there aren't any
 			 * cookies and we don't need to sync up.
 			 */
-			if (i == cpu && !need_sync && !p->core_cookie) {
+			if (i == cpu && !need_sync) {
+				if (p->core_cookie) {
+					/*
+					 * This optimization is only valid as
+					 * long as there are no cookies

This is not entirely true, need_sync is a function of core cookies, so I think this needs more clarification, it sounds like we enter this when the core has no cookies, but the task has a core_cookie? The term cookie is quite overloaded when used in the context of core vs task.

Quoted 4 lines

Effectively from what I understand this means that p wants to be coscheduled, but the core itself is not coscheduling anything at the moment, so we need to see if we should do a sync and that sync might cause p to get kicked out and a higher priority class to come in?

Yeah so about need_sync, it is basically a flag that says if the HT running the schedule() loop needs to bother with siblings.

need_sync is true only in following conditions:
- A cookied task is running on any HT on the core.
- Any HT in the core is force idled.

The above code comment you referred to is now reworked. That was for the case where we discovered during local selection that we found a task with a cookie so now we have to do a core-wide scan (need_sync = false before but now it becomes true and we start over). This optimization is done sligtly differently now, we run ->pick_task() on every class of the local CPU until we find something, if we find something with a cookie then we do core-wide selection.

The latest version of this code is now in Peter's branch:
https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/tree/kernel/sched/core.c?id=6288c0a49631ce6b53eeab7021a43e49c4c4d436

- Joel

#41 lore raw

PATCH 11/32 sched: Enqueue task into core queue only after vruntime is updated

In reply to Joel Fernandes (Google)

A waking task may have its vruntime adjusted. However, the code right now puts it into the core queue without the adjustment. This means the core queue may have a task with incorrect vruntime, potentially a very long one. This may cause a task to get artificially boosted during picking.

Fix it by enqueuing into the core queue only after the class-specific enqueue callback has been called. This ensures that for CFS tasks, the updated vruntime value is used when enqueuing the task into the core rbtree.

---
 kernel/sched/core.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/sched/core.c +3 -3
index 53af817740c0..6aa76de55ef2 100644
@@ -1753,9 +1753,6 @@ static inline void init_uclamp(void) { }
 
 static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 {
-	if (sched_core_enabled(rq))
-		sched_core_enqueue(rq, p);
-
 	if (!(flags & ENQUEUE_NOCLOCK))
 		update_rq_clock(rq);
 
@@ -1766,6 +1763,9 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
 
 	uclamp_rq_inc(rq, p);
 	p->sched_class->enqueue_task(rq, p, flags);
+
+	if (sched_core_enabled(rq))
+		sched_core_enqueue(rq, p);
 }
 
 static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#42 lore raw

PATCH 12/32 sched: Simplify the core pick loop for optimized case

In reply to Joel Fernandes (Google)

The core pick loop grew a lot of warts over time to support optimizations. Turns out that that directly doing a class pick before entering the core-wide pick is better for readability. Make the changes.

Since this is a relatively new patch, make it a separate patch so that it is easier to revert in case anyone reports an issue with it. Testing shows it to be working for me.

---
 kernel/sched/core.c | 73 ++++++++++++++++-----------------------------
 1 file changed, 26 insertions(+), 47 deletions(-)
kernel/sched/core.c +26 -47
index 6aa76de55ef2..12e8e6627ab3 100644
@@ -5180,6 +5180,15 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	put_prev_task_balance(rq, prev, rf);
 
 	smt_mask = cpu_smt_mask(cpu);
+	need_sync = !!rq->core->core_cookie;
+
+	/* reset state */
+	rq->core->core_cookie = 0UL;
+	if (rq->core->core_forceidle) {
+		need_sync = true;
+		fi_before = true;
+		rq->core->core_forceidle = false;
+	}
 
 	/*
 	 * core->core_task_seq, core->core_pick_seq, rq->core_sched_seq
@@ -5192,16 +5201,25 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 	 * 'Fix' this by also increasing @task_seq for every pick.
 	 */
 	rq->core->core_task_seq++;
-	need_sync = !!rq->core->core_cookie;
 
-	/* reset state */
-reset:
-	rq->core->core_cookie = 0UL;
-	if (rq->core->core_forceidle) {
+	/*
+	 * Optimize for common case where this CPU has no cookies
+	 * and there are no cookied tasks running on siblings.
+	 */
+	if (!need_sync) {
+		for_each_class(class) {
+			next = class->pick_task(rq);
+			if (next)
+				break;
+		}
+
+		if (!next->core_cookie) {
+			rq->core_pick = NULL;
+			goto done;
+		}
 		need_sync = true;
-		fi_before = true;
-		rq->core->core_forceidle = false;
 	}
+
 	for_each_cpu(i, smt_mask) {
 		struct rq *rq_i = cpu_rq(i);
 
@@ -5239,38 +5257,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * core.
 			 */
 			p = pick_task(rq_i, class, max);
-			if (!p) {
-				/*
-				 * If there weren't no cookies; we don't need to
-				 * bother with the other siblings.
-				 */
-				if (i == cpu && !need_sync)
-					goto next_class;
-
+			if (!p)
 				continue;
-			}
-
-			/*
-			 * Optimize the 'normal' case where there aren't any
-			 * cookies and we don't need to sync up.
-			 */
-			if (i == cpu && !need_sync) {
-				if (p->core_cookie) {
-					/*
-					 * This optimization is only valid as
-					 * long as there are no cookies
-					 * involved. We may have skipped
-					 * non-empty higher priority classes on
-					 * siblings, which are empty on this
-					 * CPU, so start over.
-					 */
-					need_sync = true;
-					goto reset;
-				}
-
-				next = p;
-				goto done;
-			}
 
 			rq_i->core_pick = p;
 
@@ -5298,18 +5286,9 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 						cpu_rq(j)->core_pick = NULL;
 					}
 					goto again;
-				} else {
-					/*
-					 * Once we select a task for a cpu, we
-					 * should not be doing an unconstrained
-					 * pick because it might starve a task
-					 * on a forced idle cpu.
-					 */
-					need_sync = true;
 				}
 			}
 		}
-next_class:;
 	}
 
 	rq->core->core_pick_seq = rq->core->core_task_seq;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#43 lore raw

Re: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case

Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7 -0
+	/*
+	 * Optimize for common case where this CPU has no cookies
+	 * and there are no cookied tasks running on siblings.
+	 */
+	if (!need_sync) {
+		for_each_class(class) {
+			next = class->pick_task(rq);

Review

Quoted 10 lines
(quoted diff) +8 -0
+			if (next)
+				break;
+		}
+
+		if (!next->core_cookie) {
+			rq->core_pick = NULL;
+			goto done;
+		}
 		need_sync = true;
 	}

This isn't what I send you here:

https://lkml.kernel.org/r/[email protected]

Specifically, you've lost the whole cfs-cgroup optimization.

What was wrong/not working with the below?

kernel/sched/core.c +5 -10
@@ -5225,8 +5227,6 @@ pick_next_task(struct rq *rq, struct tas
 		return next;
 	}
 
-	put_prev_task_balance(rq, prev, rf);
-
 	smt_mask = cpu_smt_mask(cpu);
 	need_sync = !!rq->core->core_cookie;
 
@@ -5255,17 +5255,14 @@ pick_next_task(struct rq *rq, struct tas
 	 * and there are no cookied tasks running on siblings.
 	 */
 	if (!need_sync) {
-		for_each_class(class) {
-			next = class->pick_task(rq);
-			if (next)
-				break;
-		}
-
+		next = __pick_next_task(rq, prev, rf);
 		if (!next->core_cookie) {
 			rq->core_pick = NULL;
-			goto done;
+			return next;
 		}
-		need_sync = true;
+		put_prev_task(next);
+	} else {
+		put_prev_task_balance(rq, prev, rf);
 	}
 
 	for_each_cpu(i, smt_mask) {
#44 lore raw

Re: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case

Hi Peter,

Quoted 21 lines · Peter Zijlstra
Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7 -0
+	/*
+	 * Optimize for common case where this CPU has no cookies
+	 * and there are no cookied tasks running on siblings.
+	 */
+	if (!need_sync) {
+		for_each_class(class) {
+			next = class->pick_task(rq);

Review

Quoted 10 lines
(quoted diff) +8 -0
+			if (next)
+				break;
+		}
+
+		if (!next->core_cookie) {
+			rq->core_pick = NULL;
+			goto done;
+		}
 		need_sync = true;
 	}

This isn't what I send you here:

Quoted 1 line

https://lkml.kernel.org/r/[email protected]

I had replied to it here with concerns about the effects of newly idle balancing not being reverseable, it was only a theoretical concern:
http://lore.kernel.org/r/[email protected]

Also I was trying to keep the logic the same as v8 for unconstrained pick (calling pick_task), considering that has been tested quite a bit.

Quoted 1 line

Specifically, you've lost the whole cfs-cgroup optimization.

Are you referring to this optimization in pick_next_task_fair() ?

/*
* Since we haven't yet done put_prev_entity and if the
* selected task
* is a different task than we started out with, try
* and touch the
* least amount of cfs_rqs.
*/

You are right, we wouldn't get that with just calling pick_task_fair(). We did not have this in v8 series either though.

Also, if the task is a cookied task, then I think you are doing more work with your patch due to the extra put_prev_task().

Quoted 1 line

What was wrong/not working with the below?

Other than the new idle balancing, IIRC it was also causing instability. Maybe we can considering this optimization in the future if that's Ok with you?

thanks,

- Joel

Quoted 34 lines
kernel/sched/core.c +5 -10
@@ -5225,8 +5227,6 @@ pick_next_task(struct rq *rq, struct tas
 		return next;
 	}
 
-	put_prev_task_balance(rq, prev, rf);
-
 	smt_mask = cpu_smt_mask(cpu);
 	need_sync = !!rq->core->core_cookie;
 
@@ -5255,17 +5255,14 @@ pick_next_task(struct rq *rq, struct tas
 	 * and there are no cookied tasks running on siblings.
 	 */
 	if (!need_sync) {
-		for_each_class(class) {
-			next = class->pick_task(rq);
-			if (next)
-				break;
-		}
-
+		next = __pick_next_task(rq, prev, rf);
 		if (!next->core_cookie) {
 			rq->core_pick = NULL;
-			goto done;
+			return next;
 		}
-		need_sync = true;
+		put_prev_task(next);
+	} else {
+		put_prev_task_balance(rq, prev, rf);
 	}
 
 	for_each_cpu(i, smt_mask) {
maintainer
#45 lore raw

Re: [PATCH -tip 12/32] sched: Simplify the core pick loop for optimized case

Quoted 24 lines · Joel Fernandes

Hi Peter,

Quoted 21 lines · Peter Zijlstra
Quoted 7 lines · Joel Fernandes (Google)
(quoted diff) +7 -0
+	/*
+	 * Optimize for common case where this CPU has no cookies
+	 * and there are no cookied tasks running on siblings.
+	 */
+	if (!need_sync) {
+		for_each_class(class) {
+			next = class->pick_task(rq);

Review

Quoted 10 lines
(quoted diff) +8 -0
+			if (next)
+				break;
+		}
+
+		if (!next->core_cookie) {
+			rq->core_pick = NULL;
+			goto done;
+		}
 		need_sync = true;
 	}

This isn't what I send you here:

Quoted 5 lines
Quoted 1 line

https://lkml.kernel.org/r/[email protected]

I had replied to it here with concerns about the effects of newly idle balancing not being reverseable, it was only a theoretical concern:
http://lore.kernel.org/r/[email protected]

Gah, missed that. I don't think that matters much see:
put_prev_task_balance() calling balance_fair().

Quoted 11 lines
Quoted 1 line

Specifically, you've lost the whole cfs-cgroup optimization.

Are you referring to this optimization in pick_next_task_fair() ?

/*
* Since we haven't yet done put_prev_entity and if the
* selected task
* is a different task than we started out with, try
* and touch the
* least amount of cfs_rqs.
*/

Yep, that. The giant FAIR_GROUP_SCHED hunk. The thing that makes all of pick_next_task() more complicated than it really wants to be.

Quoted 5 lines

You are right, we wouldn't get that with just calling pick_task_fair(). We did not have this in v8 series either though.

Also, if the task is a cookied task, then I think you are doing more work with your patch due to the extra put_prev_task().

Yes, but only if you mix cookie tasks with non-cookie tasks and schedule two non-cookie tasks back-to-back. I don't think we care overly much about that.

I think it makes more sense to ensure that if you have core-sched enabled on your machine and have a (core-aligned) parition with non-cookie tasks, scheduling has works as 'normal' as possible.

Quoted 5 lines
Quoted 1 line

What was wrong/not working with the below?

Other than the new idle balancing, IIRC it was also causing instability. Maybe we can considering this optimization in the future if that's Ok with you?

Hurmph.. you don't happen to remember what went splat?

#46 lore raw

PATCH 13/32 sched: Trivial forced-newidle balancer

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

---
 include/linux/sched.h |   1 +
 kernel/sched/core.c   | 130 +++++++++++++++++++++++++++++++++++++++++-
 kernel/sched/idle.c   |   1 +
 kernel/sched/sched.h  |   6 ++
 4 files changed, 137 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
index 344499ab29f2..7efce9c9d9cf 100644
@@ -688,6 +688,7 @@ struct task_struct {
 #ifdef CONFIG_SCHED_CORE
 	struct rb_node			core_node;
 	unsigned long			core_cookie;
+	unsigned int			core_occupation;
 #endif
 
 #ifdef CONFIG_CGROUP_SCHED
kernel/sched/core.c +129 -1

167 lines not on this page · raw message

kernel/sched/idle.c +1 -0
index 33864193a2f9..8bdb214eb78f 100644
@@ -404,6 +404,7 @@ static void set_next_task_idle(struct rq *rq, struct task_struct *next, bool fir
 {
 	update_idle_core(rq);
 	schedstat_inc(rq->sched_goidle);
+	queue_core_balance(rq);
 }
 
 #ifdef CONFIG_SMP
kernel/sched/sched.h +6 -0
index d934cc51acf1..e72942a9ee11 100644
@@ -1135,6 +1135,8 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+extern void queue_core_balance(struct rq *rq);
+
 #else /* !CONFIG_SCHED_CORE */
 
 static inline bool sched_core_enabled(struct rq *rq)
@@ -1147,6 +1149,10 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 	return &rq->__lock;
 }
 
+static inline void queue_core_balance(struct rq *rq)
+{
+}
+
 #endif /* CONFIG_SCHED_CORE */
 
 #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#47 lore raw

Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer

Quoted 8 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

...

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		if (p->core_occupation > dst->idle->core_occupation)
+			goto next;
+

I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.

Balbir Singh.

#48 lore raw

Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer

Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

...

Quoted 9 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		if (p->core_occupation > dst->idle->core_occupation)
+			goto next;
+

I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.

IIUC, this check prevents a task from keeping jumping among the cores forever.

For example, on a SMT2 platform:
- core0 runs taskA and taskB, core_occupation is 2
- core1 runs taskC, core_occupation is 1

Without this check, taskB could ping-pong between core0 and core1 by core load balance.

Thanks,
-Aubrey

#49 lore raw

Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer

Quoted 21 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

...

Quoted 9 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		if (p->core_occupation > dst->idle->core_occupation)
+			goto next;
+

I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.

Quoted 8 lines

IIUC, this check prevents a task from keeping jumping among the cores forever.

For example, on a SMT2 platform:
- core0 runs taskA and taskB, core_occupation is 2
- core1 runs taskC, core_occupation is 1

Without this check, taskB could ping-pong between core0 and core1 by core load balance.

But the comparison is p->core_occuption (as in tasks core occuptation, not sure what that means, can a task have a core_occupation of > 1?)

Balbir Singh.

#50 lore raw

Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer

Quoted 28 lines · Balbir Singh
Quoted 27 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

...

Quoted 9 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		if (p->core_occupation > dst->idle->core_occupation)
+			goto next;
+

I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.

IIUC, this check prevents a task from keeping jumping among the cores forever.

For example, on a SMT2 platform:
- core0 runs taskA and taskB, core_occupation is 2
- core1 runs taskC, core_occupation is 1

Quoted 6 lines
Quoted 2 lines

Without this check, taskB could ping-pong between core0 and core1 by core load balance.

But the comparison is p->core_occuption (as in tasks core occuptation, not sure what that means, can a task have a core_occupation of > 1?)

p->core_occupation is assigned to the core occupation in the last pick_next_task. (so yes, it can have a > 1 core_occupation).

Thanks,
-Aubrey

#51 lore raw

Re: [PATCH -tip 13/32] sched: Trivial forced-newidle balancer

Quoted 32 lines · Aubrey
Quoted 31 lines · Balbir Singh
Quoted 29 lines · Aubrey
Quoted 10 lines · Balbir Singh
Quoted 8 lines · Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

When a sibling is forced-idle to match the core-cookie; search for matching tasks to fill the core.

rcu_read_unlock() can incur an infrequent deadlock in sched_core_balance(). Fix this by using the RCU-sched flavor instead.

...

Quoted 9 lines

kernel/sched/core.c pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) · line 5407

Quoted 4 lines
kernel/sched/core.c +4 -0
+
+		if (p->core_occupation > dst->idle->core_occupation)
+			goto next;
+

I am unable to understand this check, a comment or clarification in the changelog will help. I presume we are looking at either one or two cpus to define the core_occupation and we expect to match it against the destination CPU.

IIUC, this check prevents a task from keeping jumping among the cores forever.

For example, on a SMT2 platform:
- core0 runs taskA and taskB, core_occupation is 2
- core1 runs taskC, core_occupation is 1

Without this check, taskB could ping-pong between core0 and core1 by core load balance.

Quoted 7 lines
Quoted 3 lines

But the comparison is p->core_occuption (as in tasks core occuptation, not sure what that means, can a task have a core_occupation of > 1?)

p->core_occupation is assigned to the core occupation in the last pick_next_task. (so yes, it can have a > 1 core_occupation).

Hmm.. I find that hard to interpret that. But I am happy to re-read the code again.

Balbir Singh.

#52 lore raw

PATCH 14/32 sched: migration changes for core scheduling

In reply to Joel Fernandes (Google)

From: Aubrey Li <[email protected]>

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 64 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 29 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +59 -5
index de82f88ba98c..ceb3906c9a8a 100644
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+#ifdef CONFIG_SCHED_CORE
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+#endif
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+#ifdef CONFIG_SCHED_CORE
+			/*
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu)) &&
+			    p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+				break;
+		}
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+#endif
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+#ifdef CONFIG_SCHED_CORE
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+#endif
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +29 -0
index e72942a9ee11..de553d39aa40 100644
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {
+			idle_core = false;
+			break;
+		}
+	}
+
+	/*
+	 * A CPU in an idle core is always the best choice for tasks with
+	 * cookies.
+	 */
+	return idle_core || rq->core->core_cookie == p->core_cookie;
+}
+
 extern void queue_core_balance(struct rq *rq);
 
 #else /* !CONFIG_SCHED_CORE */
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#53 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 38 lines · Joel Fernandes (Google)

From: Aubrey Li <[email protected]>

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 64 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 29 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +0 -0
index de82f88ba98c..ceb3906c9a8a 100644
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932

Quoted 10 lines
kernel/sched/fair.c +9 -0
 
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?

Quoted 112 lines
kernel/sched/fair.c +50 -5
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+#ifdef CONFIG_SCHED_CORE
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+#endif
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+#ifdef CONFIG_SCHED_CORE
+			/*
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu)) &&
+			    p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+				break;
+		}
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+#endif
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+#ifdef CONFIG_SCHED_CORE
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+#endif
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +7 -0
index e72942a9ee11..de553d39aa40 100644
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)

kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154

Quoted 10 lines
kernel/sched/sched.h +10 -0
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {

I was looking at this snippet and comparing this to is_core_idle(), the major difference is the check for vcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?

kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1166

Quoted 12 lines
kernel/sched/sched.h +12 -0
+			idle_core = false;
+			break;
+		}
+	}
+
+	/*
+	 * A CPU in an idle core is always the best choice for tasks with
+	 * cookies.
+	 */
+	return idle_core || rq->core->core_cookie == p->core_cookie;
+}
+

Balbir Singh.

#54 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 40 lines · Balbir Singh
Quoted 39 lines · Joel Fernandes (Google)

From: Aubrey Li <[email protected]>

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 64 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 29 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +0 -0
index de82f88ba98c..ceb3906c9a8a 100644
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
Quoted 13 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932

Quoted 9 lines
kernel/sched/fair.c +9 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 118 lines
Quoted 118 lines
kernel/sched/fair.c +50 -5
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+#ifdef CONFIG_SCHED_CORE
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+#endif
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+#ifdef CONFIG_SCHED_CORE
+			/*
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu)) &&
+			    p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+				break;
+		}
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+#endif
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+#ifdef CONFIG_SCHED_CORE
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+#endif
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +13 -0
index e72942a9ee11..de553d39aa40 100644
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
Quoted 8 lines

kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154

Quoted 4 lines
kernel/sched/sched.h +4 -0
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {

I was looking at this snippet and comparing this to is_core_idle(), the major difference is the check for vcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?

Yes, if there is a VCPU was preempted on this CPU, better not place task on this core as the VCPU may be holding a spinlock and wants to be executed again ASAP.

Thanks,
-Aubrey

maintainer
#55 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 13 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1932

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Quoted 5 lines

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
kernel/sched/sched.h +7 -0
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
Quoted 8 lines

kernel/sched/sched.h static inline raw_spinlock_t *rq_lockp(struct rq *rq) · line 1154

Quoted 4 lines
kernel/sched/sched.h +4 -0
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {

I was looking at this snippet and comparing this to is_core_idle(), the major difference is the check for vcpu_is_preempted(). Do we want to call the core as non idle if any vcpu was preempted on this CPU?

Quoted 3 lines

Yes, if there is a VCPU was preempted on this CPU, better not place task on this core as the VCPU may be holding a spinlock and wants to be executed again ASAP.

If you're doing core scheduling on vcpus, you deserve all the pain possible.

Tested-by Julien Desfossez
#56 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Aubrey
Quoted 13 lines

Review

Quoted 9 lines
(quoted diff) +9 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Quoted 9 lines
Quoted 5 lines

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

Thanks,
-Aubrey
======================================================================
From 18e4f4592c2a159fcbae637f3a422e37ad24cb5a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Wed, 25 Nov 2020 02:43:46 +0000
Subject: [PATCH 14/33] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 58 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 33 +++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 5 deletions(-)
kernel/sched/fair.c +53 -5
index de82f88ba98c..7eea5da6685a 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+#ifdef CONFIG_SCHED_CORE
+			/*
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu)) &&
+			    p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+				break;
+		}
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7551,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7588,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8821,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+#ifdef CONFIG_SCHED_CORE
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+#endif
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +33 -0
index e72942a9ee11..05b93787fe62 100644
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {
+			idle_core = false;
+			break;
+		}
+	}
+
+	/*
+	 * A CPU in an idle core is always the best choice for tasks with
+	 * cookies.
+	 */
+	return idle_core || rq->core->core_cookie == p->core_cookie;
+}
+
 extern void queue_core_balance(struct rq *rq);
 
 #else /* !CONFIG_SCHED_CORE */
@@ -1153,6 +1182,10 @@ static inline void queue_core_balance(struct rq *rq)
 {
 }
 
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	return true;
+}
 #endif /* CONFIG_SCHED_CORE */
 
 #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.17.1
#57 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

{
        return false;
}

#endif

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Balbir

Tested-by Julien Desfossez
#58 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 34 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
(quoted diff) +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

Quoted 8 lines
{
        return false;
}

#endif

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?

Thanks,
-Aubrey

From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 57 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +52 -5
index de82f88ba98c..70dd013dff1d 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+			/*
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu))) {
+				if (__cookie_match(cpu_rq(cpu), p))
+					break;
+			} else {
+				break;
+			}
+		}
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +43 -0

55 lines not on this page · raw message

Signature · 1 line
2.17.1
#59 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

{
        return false;
}

#endif

Quoted 7 lines
Quoted 3 lines

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?

Much better :)

Quoted 81 lines

Thanks,
-Aubrey

From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 57 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +15 -3
index de82f88ba98c..70dd013dff1d 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+			/*

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155

Quoted 10 lines
kernel/sched/fair.c +10 -0
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu))) {
+				if (__cookie_match(cpu_rq(cpu), p))
+					break;
+			} else {
+				break;
+			}
+		}

Isn't this better and equivalent?

	if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) &&
		sched_core_cookie_match(cpu_rq(cpu), p))
		break;
Quoted 40 lines
kernel/sched/fair.c +17 -2
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841

Quoted 10 lines
kernel/sched/fair.c +10 -0
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+

Again, I think this can be refactored because sched_core_cookie_match checks for sched_core_enabled()

	int i = 0;
	bool cookie_match = false;
	for_each_cpu(i, sched_group_span(group)) {
		if (sched_core_cookie_match(cpu_rq(i), p))
			break;
	}
	if (i >= nr_cpu_ids)
		continue;
Quoted 72 lines
kernel/sched/fair.c +4 -0
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
 
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
kernel/sched/sched.h +43 -0
index e72942a9ee11..8bb3b72d593c 100644
@@ -1135,6 +1135,40 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool __cookie_match(struct rq *rq, struct task_struct *p)
+{
+	return rq->core->core_cookie == p->core_cookie;
+}
+
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	bool idle_core = true;
+	int cpu;
+
+	/* Ignore cookie match if core scheduler is not enabled on the CPU. */
+	if (!sched_core_enabled(rq))
+		return true;
+
+	for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+		if (!available_idle_cpu(cpu)) {
+			idle_core = false;
+			break;
+		}
+	}
+
+	/*
+	 * A CPU in an idle core is always the best choice for tasks with
+	 * cookies.
+	 */
+	return idle_core || __cookie_match(rq, p);
+}
+
 extern void queue_core_balance(struct rq *rq);
 
 #else /* !CONFIG_SCHED_CORE */
@@ -1153,6 +1187,15 @@ static inline void queue_core_balance(struct rq *rq)
 {
 }
 
+static inline bool __cookie_match(struct rq *rq, struct task_struct *p)
+{
+	return true;
+}
+
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+	return true;
+}
 #endif /* CONFIG_SCHED_CORE */
 
 #ifdef CONFIG_SCHED_SMT
Signature · 1 line
2.17.1

Balbir Singh.

#60 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

{
        return false;
}

#endif

Quoted 7 lines
Quoted 3 lines

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?

Much better :)

Quoted 81 lines

Thanks,
-Aubrey

From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 57 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +15 -3
index de82f88ba98c..70dd013dff1d 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+			/*

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155

Quoted 10 lines
kernel/sched/fair.c +10 -0
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu))) {
+				if (__cookie_match(cpu_rq(cpu), p))
+					break;
+			} else {
+				break;
+			}
+		}

Isn't this better and equivalent?

Quoted 4 lines
	if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) &&
		sched_core_cookie_match(cpu_rq(cpu), p))
		break;

That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in sched_core_cookie_match.

Quoted 54 lines
Quoted 40 lines
kernel/sched/fair.c +17 -2
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841

Quoted 10 lines
kernel/sched/fair.c +10 -0
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+

Again, I think this can be refactored because sched_core_cookie_match checks for sched_core_enabled()

Quoted 8 lines
	int i = 0;
	bool cookie_match = false;
	for_each_cpu(i, sched_group_span(group)) {
		if (sched_core_cookie_match(cpu_rq(i), p))
			break;
	}
	if (i >= nr_cpu_ids)
		continue;

There is a loop here when CONFIG_SCHED_CORE=n, which is unwanted I guess.

Thanks,
-Aubrey

#61 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 152 lines · Aubrey
Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

{
        return false;
}

#endif

Quoted 7 lines
Quoted 3 lines

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?

Much better :)

Quoted 81 lines

Thanks,
-Aubrey

From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 57 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +15 -3
index de82f88ba98c..70dd013dff1d 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+			/*

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155

Quoted 10 lines
kernel/sched/fair.c +10 -0
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu))) {
+				if (__cookie_match(cpu_rq(cpu), p))
+					break;
+			} else {
+				break;
+			}
+		}

Isn't this better and equivalent?

Quoted 4 lines
	if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) &&
		sched_core_cookie_match(cpu_rq(cpu), p))
		break;
Quoted 3 lines

That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in sched_core_cookie_match.

I see, so we basically need a jump label, if sched_core_cookie_match can be inlined with a check for sched_core_enabled() upfront, it might solve a lot of the concern, readability of this section of code is not the best.

Quoted 64 lines
Quoted 55 lines
Quoted 41 lines
kernel/sched/fair.c +17 -2
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841

Quoted 10 lines
kernel/sched/fair.c +10 -0
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+

Again, I think this can be refactored because sched_core_cookie_match checks for sched_core_enabled()

Quoted 8 lines
	int i = 0;
	bool cookie_match = false;
	for_each_cpu(i, sched_group_span(group)) {
		if (sched_core_cookie_match(cpu_rq(i), p))
			break;
	}
	if (i >= nr_cpu_ids)
		continue;
Quoted 2 lines

There is a loop here when CONFIG_SCHED_CORE=n, which is unwanted I guess.

Yes, potentially, may be abstract the for_each_cpu into a function and then optimize out the case for SCHED_CORE=n, I feel all the extra checks in the various places make the code hard to read.

Balbir

#62 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 226 lines · Balbir Singh
Quoted 152 lines · Aubrey
Quoted 145 lines · Balbir Singh
Quoted 40 lines · Aubrey
Quoted 39 lines · Balbir Singh
Quoted 21 lines · Aubrey
Quoted 20 lines · Peter Zijlstra
Quoted 18 lines · Aubrey
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5 -0
+#ifdef CONFIG_SCHED_CORE
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
Quoted 8 lines

kernel/sched/fair.c static void task_numa_find_cpu(struct task_numa_env *env, · line 1930

Quoted 4 lines
kernel/sched/fair.c +4 -0
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+#endif
+

Any reason this is under an #ifdef? In sched_core_cookie_match() won't the check for sched_core_enabled() do the right thing even when CONFIG_SCHED_CORE is not enabed?>

Yes, sched_core_enabled works properly when CONFIG_SCHED_CORE is not enabled. But when CONFIG_SCHED_CORE is not enabled, it does not make sense to leave a core scheduler specific function here even at compile time. Also, for the cases in hot path, this saves CPU cycles to avoid a judgment.

Quoted 7 lines
Quoted 3 lines

No, that's nonsense. If it works, remove the #ifdef. Less (#ifdef) is more.

Okay, I pasted the refined patch here.
@Joel, please let me know if you want me to send it in a separated thread.

You still have a bunch of #ifdefs, can't we just do

#ifndef CONFIG_SCHED_CORE
static inline bool sched_core_enabled(struct rq *rq)

{
        return false;
}

#endif

Quoted 7 lines
Quoted 3 lines

and frankly I think even that is not needed because there is a jump label __sched_core_enabled that tells us if sched_core is enabled or not.

Hmm..., I need another wrapper for CONFIG_SCHED_CORE specific variables. How about this one?

Much better :)

Quoted 81 lines

Thanks,
-Aubrey

From 61dac9067e66b5b9ea26c684c8c8235714bab38a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 26 Nov 2020 03:08:04 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 57 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+), 5 deletions(-)
kernel/sched/fair.c +15 -3
index de82f88ba98c..70dd013dff1d 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,8 +6140,19 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-			break;
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+			/*

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6155

Quoted 10 lines
kernel/sched/fair.c +10 -0
+			 * If Core Scheduling is enabled, select this cpu
+			 * only if the process cookie matches core cookie.
+			 */
+			if (sched_core_enabled(cpu_rq(cpu))) {
+				if (__cookie_match(cpu_rq(cpu), p))
+					break;
+			} else {
+				break;
+			}
+		}

Isn't this better and equivalent?

Quoted 4 lines
	if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) &&
		sched_core_cookie_match(cpu_rq(cpu), p))
		break;
Quoted 3 lines

That's my previous implementation in the earlier version. But since here is the hot code path, we want to remove the idle core check in sched_core_cookie_match.

I see, so we basically need a jump label, if sched_core_cookie_match can be inlined with a check for sched_core_enabled() upfront, it might solve a lot of the concern, readability of this section of code is not the best.

Quoted 64 lines
Quoted 63 lines
Quoted 41 lines
kernel/sched/fair.c +17 -2
 	}
 
 	time = cpu_clock(this) - time;
@@ -7530,8 +7552,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7589,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8822,23 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		if (sched_core_enabled(cpu_rq(this_cpu))) {
+			int i = 0;
+			bool cookie_match = false;
+
+			for_each_cpu(i, sched_group_span(group)) {
+				struct rq *rq = cpu_rq(i);
+

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8841

Quoted 10 lines
kernel/sched/fair.c +10 -0
+				if (sched_core_cookie_match(rq, p)) {
+					cookie_match = true;
+					break;
+				}
+			}
+			/* Skip over this group if no cookie matched */
+			if (!cookie_match)
+				continue;
+		}
+

Again, I think this can be refactored because sched_core_cookie_match checks for sched_core_enabled()

	int i = 0;
	bool cookie_match = false;
	for_each_cpu(i, sched_group_span(group)) {
		if (sched_core_cookie_match(cpu_rq(i), p))
			break;
	}
	if (i >= nr_cpu_ids)
		continue;
Quoted 6 lines
Quoted 2 lines

There is a loop here when CONFIG_SCHED_CORE=n, which is unwanted I guess.

Yes, potentially, may be abstract the for_each_cpu into a function and then optimize out the case for SCHED_CORE=n, I feel all the extra checks in the various places make the code hard to read.

Okay, I see your point, let me try if I can make it better.

Thanks,
-Aubrey

Tested-by Julien Desfossez
#63 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Hi Balbir,

I still placed the patch embedded in this thread, welcome any comments.

Thanks,
-Aubrey
======================================================================

From d64455dcaf47329673903a68a9df1151400cdd7a Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Wed, 2 Dec 2020 13:53:30 +0000
Subject: [PATCH] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 33 +++++++++++++++++---
 kernel/sched/sched.h | 71 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 100 insertions(+), 4 deletions(-)
kernel/sched/fair.c +29 -4
index de82f88ba98c..b8657766b660 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,7 +6140,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
+
+		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu) &&
+		    sched_cpu_cookie_match(cpu_rq(cpu), p))
 			break;
 	}
 
@@ -7530,8 +7543,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7580,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8813,10 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		/* Skip over this group if no cookie matched */
+		if (!sched_group_cookie_match(cpu_rq(this_cpu), p, group))
+			continue;
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +71 -0

83 lines not on this page · raw message

Signature · 1 line
2.17.1
Tested-by Julien Desfossez
#64 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 3 lines · Aubrey

Hi Balbir,

I still placed the patch embedded in this thread, welcome any comments.

Sorry, this version needs more work, refined as below, and I realized I should place a version number to the patch, start from v2 now.

Thanks,
-Aubrey
======================================================================
From aff2919889635aa9311d15bac3e949af0300ddc1 Mon Sep 17 00:00:00 2001
From: Aubrey Li <[email protected]>
Date: Thu, 3 Dec 2020 00:51:18 +0000
Subject: [PATCH v2] sched: migration changes for core scheduling

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 33 +++++++++++++++++---
 kernel/sched/sched.h | 72 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 101 insertions(+), 4 deletions(-)
kernel/sched/fair.c +29 -4
index de82f88ba98c..afdfea70c58c 100644
@@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env,
 		if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
 			continue;
 
+		/*
+		 * Skip this cpu if source task's cookie does not match
+		 * with CPU's core cookie.
+		 */
+		if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+			continue;
+
 		env->dst_cpu = cpu;
 		if (task_numa_compare(env, taskimp, groupimp, maymove))
 			break;
@@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
 	/* Traverse only the allowed CPUs */
 	for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+		struct rq *rq = cpu_rq(i);
+
+		if (!sched_core_cookie_match(rq, p))
+			continue;
+
 		if (sched_idle_cpu(i))
 			return i;
 
 		if (available_idle_cpu(i)) {
-			struct rq *rq = cpu_rq(i);
 			struct cpuidle_state *idle = idle_get_state(rq);
 			if (idle && idle->exit_latency < min_exit_latency) {
 				/*
@@ -6129,7 +6140,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
 	for_each_cpu_wrap(cpu, cpus, target) {
 		if (!--nr)
 			return -1;
-		if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
+
+		if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) &&
+		    sched_cpu_cookie_match(cpu_rq(cpu), p))
 			break;
 	}
 
@@ -7530,8 +7543,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 	 * We do not migrate tasks that are:
 	 * 1) throttled_lb_pair, or
 	 * 2) cannot be migrated to this CPU due to cpus_ptr, or
-	 * 3) running (obviously), or
-	 * 4) are cache-hot on their current CPU.
+	 * 3) task's cookie does not match with this CPU's core cookie
+	 * 4) running (obviously), or
+	 * 5) are cache-hot on their current CPU.
 	 */
 	if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
 		return 0;
@@ -7566,6 +7580,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
 		return 0;
 	}
 
+	/*
+	 * Don't migrate task if the task's cookie does not match
+	 * with the destination CPU's core cookie.
+	 */
+	if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+		return 0;
+
 	/* Record that we found atleast one task that could run on dst_cpu */
 	env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8813,10 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
 					p->cpus_ptr))
 			continue;
 
+		/* Skip over this group if no cookie matched */
+		if (!sched_group_cookie_match(cpu_rq(this_cpu), p, group))
+			continue;
+
 		local_group = cpumask_test_cpu(this_cpu,
 					       sched_group_span(group));
 
kernel/sched/sched.h +72 -0

90 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#65 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

In reply to Joel Fernandes (Google)
Quoted 79 lines · Joel Fernandes (Google)

From: Aubrey Li <[email protected]>

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 64 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 29 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +18 -3
index de82f88ba98c..ceb3906c9a8a 100644
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,
                if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
                        continue;
 
+#ifdef CONFIG_SCHED_CORE
+               /*
+                * Skip this cpu if source task's cookie does not match
+                * with CPU's core cookie.
+                */
+               if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+                       continue;
+#endif
+
                env->dst_cpu = cpu;
                if (task_numa_compare(env, taskimp, groupimp, maymove))
                        break;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
        /* Traverse only the allowed CPUs */
        for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+               struct rq *rq = cpu_rq(i);
+
+#ifdef CONFIG_SCHED_CORE
+               if (!sched_core_cookie_match(rq, p))
+                       continue;
+#endif
+
                if (sched_idle_cpu(i))
                        return i;
 
                if (available_idle_cpu(i)) {
-                       struct rq *rq = cpu_rq(i);
                        struct cpuidle_state *idle = idle_get_state(rq);
                        if (idle && idle->exit_latency < min_exit_latency) {
                                /*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
        for_each_cpu_wrap(cpu, cpus, target) {
                if (!--nr)
                        return -1;
-               if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-                       break;
+
+               if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6158

Quoted 10 lines
kernel/sched/fair.c +10 -0
+#ifdef CONFIG_SCHED_CORE
+                       /*
+                        * If Core Scheduling is enabled, select this cpu
+                        * only if the process cookie matches core cookie.
+                        */
+                       if (sched_core_enabled(cpu_rq(cpu)) &&
+                           p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+                               break;
+               }

This makes code unreadable.
Put this coresched specific stuff in an inline function; You can have a look at what is done with asym_fits_capacity()

Quoted 43 lines
kernel/sched/fair.c +20 -2
        }
 
        time = cpu_clock(this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
         * We do not migrate tasks that are:
         * 1) throttled_lb_pair, or
         * 2) cannot be migrated to this CPU due to cpus_ptr, or
-        * 3) running (obviously), or
-        * 4) are cache-hot on their current CPU.
+        * 3) task's cookie does not match with this CPU's core cookie
+        * 4) running (obviously), or
+        * 5) are cache-hot on their current CPU.
         */
        if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
                return 0;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
                return 0;
        }
 
+#ifdef CONFIG_SCHED_CORE
+       /*
+        * Don't migrate task if the task's cookie does not match
+        * with the destination CPU's core cookie.
+        */
+       if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+               return 0;
+#endif
+
        /* Record that we found atleast one task that could run on dst_cpu */
        env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
                                        p->cpus_ptr))
                        continue;
 
+#ifdef CONFIG_SCHED_CORE
+               if (sched_core_enabled(cpu_rq(this_cpu))) {
+                       int i = 0;
+                       bool cookie_match = false;
+
+                       for_each_cpu(i, sched_group_span(group)) {
+                               struct rq *rq = cpu_rq(i);
+

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8847

Quoted 10 lines
kernel/sched/fair.c +10 -0
+                               if (sched_core_cookie_match(rq, p)) {
+                                       cookie_match = true;
+                                       break;
+                               }
+                       }
+                       /* Skip over this group if no cookie matched */
+                       if (!cookie_match)
+                               continue;
+               }
+#endif

same here, encapsulate this to keep find_idlest_group readable

Quoted 47 lines
kernel/sched/fair.c +1 -0
+
                local_group = cpumask_test_cpu(this_cpu,
                                               sched_group_span(group));
 
kernel/sched/sched.h +29 -1
index e72942a9ee11..de553d39aa40 100644
@@ -1135,6 +1135,35 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 
 bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
 
+/*
+ * Helper to check if the CPU's core cookie matches with the task's cookie
+ * when core scheduling is enabled.
+ * A special case is that the task's cookie always matches with CPU's core
+ * cookie if the CPU is in an idle core.
+ */
+static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
+{
+       bool idle_core = true;
+       int cpu;
+
+       /* Ignore cookie match if core scheduler is not enabled on the CPU. */
+       if (!sched_core_enabled(rq))
+               return true;
+
+       for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) {
+               if (!available_idle_cpu(cpu)) {
+                       idle_core = false;
+                       break;
+               }
+       }
+
+       /*
+        * A CPU in an idle core is always the best choice for tasks with
+        * cookies.
+        */
+       return idle_core || rq->core->core_cookie == p->core_cookie;
+}
+
 extern void queue_core_balance(struct rq *rq);
 
 #else /* !CONFIG_SCHED_CORE */
--

2.29.2.299.gdc1121823c-goog

#66 lore raw

Re: [PATCH -tip 14/32] sched: migration changes for core scheduling

Quoted 87 lines · Vincent Guittot
Quoted 85 lines · Joel Fernandes (Google)

From: Aubrey Li <[email protected]>

- Don't migrate if there is a cookie mismatch
Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU.

- Select cookie matched idle CPU
In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU.

- Find cookie matched idlest CPU
In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie

- Don't migrate task if cookie not match
For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie

---
 kernel/sched/fair.c  | 64 ++++++++++++++++++++++++++++++++++++++++----
 kernel/sched/sched.h | 29 ++++++++++++++++++++
 2 files changed, 88 insertions(+), 5 deletions(-)
kernel/sched/fair.c +24 -3
index de82f88ba98c..ceb3906c9a8a 100644
@@ -1921,6 +1921,15 @@ static void task_numa_find_cpu(struct task_numa_env *env,
                if (!cpumask_test_cpu(cpu, env->p->cpus_ptr))
                        continue;
 
+#ifdef CONFIG_SCHED_CORE
+               /*
+                * Skip this cpu if source task's cookie does not match
+                * with CPU's core cookie.
+                */
+               if (!sched_core_cookie_match(cpu_rq(cpu), env->p))
+                       continue;
+#endif
+
                env->dst_cpu = cpu;
                if (task_numa_compare(env, taskimp, groupimp, maymove))
                        break;
@@ -5867,11 +5876,17 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this
 
        /* Traverse only the allowed CPUs */
        for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) {
+               struct rq *rq = cpu_rq(i);
+
+#ifdef CONFIG_SCHED_CORE
+               if (!sched_core_cookie_match(rq, p))
+                       continue;
+#endif
+
                if (sched_idle_cpu(i))
                        return i;
 
                if (available_idle_cpu(i)) {
-                       struct rq *rq = cpu_rq(i);
                        struct cpuidle_state *idle = idle_get_state(rq);
                        if (idle && idle->exit_latency < min_exit_latency) {
                                /*
@@ -6129,8 +6144,18 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t
        for_each_cpu_wrap(cpu, cpus, target) {
                if (!--nr)
                        return -1;
-               if (available_idle_cpu(cpu) || sched_idle_cpu(cpu))
-                       break;
+
+               if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) {
+#ifdef CONFIG_SCHED_CORE
+                       /*
+                        * If Core Scheduling is enabled, select this cpu
+                        * only if the process cookie matches core cookie.
+                        */
+                       if (sched_core_enabled(cpu_rq(cpu)) &&
Quoted 9 lines

kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6158

Quoted 4 lines
kernel/sched/fair.c +4 -0
+                           p->core_cookie == cpu_rq(cpu)->core->core_cookie)
+#endif
+                               break;
+               }

This makes code unreadable.
Put this coresched specific stuff in an inline function; You can have a look at what is done with asym_fits_capacity()

This is done in a refined version. Sorry I pasted the version embedded in this thread, this is not the latest version.

Quoted 47 lines
Quoted 47 lines
kernel/sched/fair.c +24 -2
        }
 
        time = cpu_clock(this) - time;
@@ -7530,8 +7555,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
         * We do not migrate tasks that are:
         * 1) throttled_lb_pair, or
         * 2) cannot be migrated to this CPU due to cpus_ptr, or
-        * 3) running (obviously), or
-        * 4) are cache-hot on their current CPU.
+        * 3) task's cookie does not match with this CPU's core cookie
+        * 4) running (obviously), or
+        * 5) are cache-hot on their current CPU.
         */
        if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu))
                return 0;
@@ -7566,6 +7592,15 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)
                return 0;
        }
 
+#ifdef CONFIG_SCHED_CORE
+       /*
+        * Don't migrate task if the task's cookie does not match
+        * with the destination CPU's core cookie.
+        */
+       if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p))
+               return 0;
+#endif
+
        /* Record that we found atleast one task that could run on dst_cpu */
        env->flags &= ~LBF_ALL_PINNED;
 
@@ -8792,6 +8827,25 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
                                        p->cpus_ptr))
                        continue;
 
+#ifdef CONFIG_SCHED_CORE
+               if (sched_core_enabled(cpu_rq(this_cpu))) {
+                       int i = 0;
+                       bool cookie_match = false;
+
+                       for_each_cpu(i, sched_group_span(group)) {
+                               struct rq *rq = cpu_rq(i);
+
+                               if (sched_core_cookie_match(rq, p)) {
+                                       cookie_match = true;
+                                       break;
+                               }
Quoted 9 lines

kernel/sched/fair.c find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) · line 8847

Quoted 6 lines
kernel/sched/fair.c +6 -0
+                       }
+                       /* Skip over this group if no cookie matched */
+                       if (!cookie_match)
+                               continue;
+               }
+#endif

same here, encapsulate this to keep find_idlest_group readable

Okay, I'll try to refine this.

Thanks,
-Aubrey

#67 lore raw

PATCH 15/32 sched: Improve snapshotting of min_vruntime for CGroups

In reply to Joel Fernandes (Google)

A previous patch improved cross-cpu vruntime comparison opertations in pick_next_task(). Improve it further for tasks in CGroups.

In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).

A note about the min_vruntime snapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.

Which gives:

        fib     fi      update?
        0       0       1
        0       1       1
        1       0       1
        1       1       0

So the min_vruntime snapshot needs to be updated when: !(fib && fi).

Also, the cfs_prio_less() function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq's min_vruntime_fi in the hierarchy. So pass this information along via pick_task() -> prio_less().

---
 kernel/sched/core.c  | 61 +++++++++++++++++----------------
 kernel/sched/fair.c  | 80 ++++++++++++++++++++++++++++++++------------
 kernel/sched/sched.h |  7 +++-
 3 files changed, 97 insertions(+), 51 deletions(-)
kernel/sched/core.c +32 -29
index 3b373b592680..20125431af87 100644
@@ -101,7 +101,7 @@ static inline int __task_prio(struct task_struct *p)
  */
 
 /* real prio, less is less */
-static inline bool prio_less(struct task_struct *a, struct task_struct *b)
+static inline bool prio_less(struct task_struct *a, struct task_struct *b, bool in_fi)
 {
 
 	int pa = __task_prio(a), pb = __task_prio(b);
@@ -116,7 +116,7 @@ static inline bool prio_less(struct task_struct *a, struct task_struct *b)
 		return !dl_time_before(a->dl.deadline, b->dl.deadline);
 
 	if (pa == MAX_RT_PRIO + MAX_NICE)	/* fair */
-		return cfs_prio_less(a, b);
+		return cfs_prio_less(a, b, in_fi);
 
 	return false;
 }
@@ -130,7 +130,7 @@ static inline bool __sched_core_less(struct task_struct *a, struct task_struct *
 		return false;
 
 	/* flip prio, so high prio is leftmost */
-	if (prio_less(b, a))
+	if (prio_less(b, a, task_rq(a)->core->core_forceidle))
 		return true;
 
 	return false;
@@ -5101,7 +5101,7 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)
  * - Else returns idle_task.
  */
 static struct task_struct *
-pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *max)
+pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *max, bool in_fi)
 {
 	struct task_struct *class_pick, *cookie_pick;
 	unsigned long cookie = rq->core->core_cookie;
@@ -5116,7 +5116,7 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma
 		 * higher priority than max.
 		 */
 		if (max && class_pick->core_cookie &&
-		    prio_less(class_pick, max))
+		    prio_less(class_pick, max, in_fi))
 			return idle_sched_class.pick_task(rq);
 
 		return class_pick;
@@ -5135,13 +5135,15 @@ pick_task(struct rq *rq, const struct sched_class *class, struct task_struct *ma
 	 * the core (so far) and it must be selected, otherwise we must go with
 	 * the cookie pick in order to satisfy the constraint.
 	 */
-	if (prio_less(cookie_pick, class_pick) &&
-	    (!max || prio_less(max, class_pick)))
+	if (prio_less(cookie_pick, class_pick, in_fi) &&
+	    (!max || prio_less(max, class_pick, in_fi)))
 		return class_pick;
 
 	return cookie_pick;
 }
 
+extern void task_vruntime_update(struct rq *rq, struct task_struct *p, bool in_fi);
+
 static struct task_struct *
 pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 {
@@ -5230,9 +5232,14 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 
 		if (!next->core_cookie) {
 			rq->core_pick = NULL;
+			/*
+			 * For robustness, update the min_vruntime_fi for
+			 * unconstrained picks as well.
+			 */
+			WARN_ON_ONCE(fi_before);
+			task_vruntime_update(rq, next, false);
 			goto done;
 		}
-		need_sync = true;
 	}
 
 	for_each_cpu(i, smt_mask) {
@@ -5244,14 +5251,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			update_rq_clock(rq_i);
 	}
 
-	/* Reset the snapshot if core is no longer in force-idle. */
-	if (!fi_before) {
-		for_each_cpu(i, smt_mask) {
-			struct rq *rq_i = cpu_rq(i);
-			rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime;
-		}
-	}
-
 	/*
 	 * Try and select tasks for each sibling in decending sched_class
 	 * order.
@@ -5271,7 +5270,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 			 * highest priority task already selected for this
 			 * core.
 			 */
-			p = pick_task(rq_i, class, max);
+			p = pick_task(rq_i, class, max, fi_before);
 			if (!p)
 				continue;
 
@@ -5279,6 +5278,11 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				occ++;
 
 			rq_i->core_pick = p;
+			if (rq_i->idle == p && rq_i->nr_running) {
+				rq->core->core_forceidle = true;
+				if (!fi_before)
+					rq->core->core_forceidle_seq++;
+			}
 
 			/*
 			 * If this new candidate is of higher priority than the
@@ -5297,6 +5301,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 				max = p;
 
 				if (old_max) {
+					rq->core->core_forceidle = false;
 					for_each_cpu(j, smt_mask) {
 						if (j == i)
 							continue;
@@ -5338,10 +5343,16 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		if (!rq_i->core_pick)
 			continue;
 
-		if (is_task_rq_idle(rq_i->core_pick) && rq_i->nr_running &&
-		    !rq_i->core->core_forceidle) {
-			rq_i->core->core_forceidle = true;
-		}
+		/*
+		 * Update for new !FI->FI transitions, or if continuing to be in !FI:
+		 * fi_before     fi      update?
+		 *  0            0       1
+		 *  0            1       1
+		 *  1            0       1
+		 *  1            1       0
+		 */
+		if (!(fi_before && rq->core->core_forceidle))
+			task_vruntime_update(rq_i, rq_i->core_pick, rq->core->core_forceidle);
 
 		rq_i->core_pick->core_occupation = occ;
 
@@ -5361,14 +5372,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
 		resched_curr(rq_i);
 	}
 
-	/* Snapshot if core is in force-idle. */
-	if (!fi_before && rq->core->core_forceidle) {
-		for_each_cpu(i, smt_mask) {
-			struct rq *rq_i = cpu_rq(i);
-			rq_i->cfs.min_vruntime_fi = rq_i->cfs.min_vruntime;
-		}
-	}
-
 done:
 	set_next_task(rq, next);
 	return next;
kernel/sched/fair.c +59 -21

102 lines not on this page · raw message

kernel/sched/sched.h +6 -1
index de553d39aa40..5c258ab64052 100644
@@ -518,8 +518,10 @@ struct cfs_rq {
 	u64			exec_clock;
 	u64			min_vruntime;
 #ifdef CONFIG_SCHED_CORE
+	unsigned int		forceidle_seq;
 	u64			min_vruntime_fi;
 #endif
+
 #ifndef CONFIG_64BIT
 	u64			min_vruntime_copy;
 #endif
@@ -1078,6 +1080,7 @@ struct rq {
 	unsigned int		core_pick_seq;
 	unsigned long		core_cookie;
 	unsigned char		core_forceidle;
+	unsigned int		core_forceidle_seq;
 #endif
 };
 
@@ -1133,7 +1136,7 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq)
 	return &rq->__lock;
 }
 
-bool cfs_prio_less(struct task_struct *a, struct task_struct *b);
+bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool fi);
 
 /*
  * Helper to check if the CPU's core cookie matches with the task's cookie
@@ -1166,6 +1169,8 @@ static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p)
 
 extern void queue_core_balance(struct rq *rq);
 
+bool cfs_prio_less(struct task_struct *a, struct task_struct *b, bool fi);
+
 #else /* !CONFIG_SCHED_CORE */
 
 static inline bool sched_core_enabled(struct rq *rq)
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#68 lore raw

Re: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups

Quoted 21 lines · Joel Fernandes (Google)

A previous patch improved cross-cpu vruntime comparison opertations in pick_next_task(). Improve it further for tasks in CGroups.

In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).

A note about the min_vruntime snapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.

Which gives:

        fib     fi      update?
        0       0       1
        0       1       1
        1       0       1
        1       1       0
Quoted 6 lines

So the min_vruntime snapshot needs to be updated when: !(fib && fi).

Also, the cfs_prio_less() function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq's min_vruntime_fi in the hierarchy. So pass this information along via pick_task() -> prio_less().

Hurmph.. so I'm tempted to smash a bunch of patches together.

2 <- 3 (already done - bisection crashes are daft) 6 <- 11
7 <- {10, 12}
9 <- 15

I'm thinking that would result in an easier to read series, or do we want to preserve this history?

(fwiw, I pulled 15 before 13,14, as I think that makes more sense anyway).

Hmm?

#69 lore raw

Re: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups

Hi Peter,

Quoted 36 lines · Peter Zijlstra
Quoted 21 lines · Joel Fernandes (Google)

A previous patch improved cross-cpu vruntime comparison opertations in pick_next_task(). Improve it further for tasks in CGroups.

In particular, for cross-CPU comparisons, we were previously going to the root-level se(s) for both the task being compared. That was strange. This patch instead finds the se(s) for both tasks that have the same parent (which may be different from root).

A note about the min_vruntime snapshot and force idling:
Abbreviations: fi: force-idled now? ; fib: force-idled before? During selection:
When we're not fi, we need to update snapshot.
when we're fi and we were not fi, we must update snapshot. When we're fi and we were already fi, we must not update snapshot.

Which gives:

        fib     fi      update?
        0       0       1
        0       1       1
        1       0       1
        1       1       0
Quoted 6 lines

So the min_vruntime snapshot needs to be updated when: !(fib && fi).

Also, the cfs_prio_less() function needs to be aware of whether the core is in force idle or not, since it will be use this information to know whether to advance a cfs_rq's min_vruntime_fi in the hierarchy. So pass this information along via pick_task() -> prio_less().

Hurmph.. so I'm tempted to smash a bunch of patches together.

2 <- 3 (already done - bisection crashes are daft) 6 <- 11
7 <- {10, 12}
9 <- 15

Quoted 5 lines

I'm thinking that would result in an easier to read series, or do we want to preserve this history?

(fwiw, I pulled 15 before 13,14, as I think that makes more sense anyway).

Either way would be Ok with me, I would suggest retaining the history though so that the details in the changelog are preserved of the issues we faced, and in the future we can refer back to them.

thanks,

- Joel

maintainer
#70 lore raw

Re: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups

Quoted 3 lines · Joel Fernandes

Either way would be Ok with me, I would suggest retaining the history though so that the details in the changelog are preserved of the issues we faced, and in the future we can refer back to them.

Well, in part that's what we have mail archives for. My main concern is that there's a lot of back and forth in the patches as presented and that makes review really awkward.

I've done some of the patch munging proposed, I might do more if I find the motivation.

But first I should probably go read all that API crud you did on top, I've not read that before...

maintainer
#71 lore raw

Re: [PATCH -tip 15/32] sched: Improve snapshotting of min_vruntime for CGroups

In reply to Joel Fernandes (Google)

kernel/sched/fair.c static inline void task_tick_core(struct rq *rq, struct task_struct *curr) · line 10790

Quoted 5 lines · Joel Fernandes (Google)
kernel/sched/fair.c +3 -1
+static void se_fi_update(struct sched_entity *se, unsigned int fi_seq, bool forceidle)
 {
-	bool samecpu = task_cpu(a) == task_cpu(b);
+	bool root = true;
+	long old, new;

My compiler was not impressed by all those variable definitions.

Quoted 13 lines
kernel/sched/fair.c +13 -0
+
+	for_each_sched_entity(se) {
+		struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+		if (forceidle) {
+			if (cfs_rq->forceidle_seq == fi_seq)
+				break;
+			cfs_rq->forceidle_seq = fi_seq;
+		}
+
+		cfs_rq->min_vruntime_fi = cfs_rq->min_vruntime;
+	}
+}
#72 lore raw

PATCH 16/32 irq_work: Cleanup

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Get rid of the __call_single_node union and clean up the API a little to avoid external code relying on the structure layout as much.

(Needed for irq_work_is_busy() API in core-scheduling series).

---
 drivers/gpu/drm/i915/i915_request.c |  4 ++--
 include/linux/irq_work.h            | 33 ++++++++++++++++++-----------
 include/linux/irqflags.h            |  4 ++--
 kernel/bpf/stackmap.c               |  2 +-
 kernel/irq_work.c                   | 18 ++++++++--------
 kernel/printk/printk.c              |  6 ++----
 kernel/rcu/tree.c                   |  3 +--
 kernel/time/tick-sched.c            |  6 ++----
 kernel/trace/bpf_trace.c            |  2 +-
 9 files changed, 41 insertions(+), 37 deletions(-)
drivers/gpu/drm/i915/i915_request.c +2 -2
index 0e813819b041..5385b081a376 100644
@@ -197,7 +197,7 @@ __notify_execute_cb(struct i915_request *rq, bool (*fn)(struct irq_work *wrk))
 
 	llist_for_each_entry_safe(cb, cn,
 				  llist_del_all(&rq->execute_cb),
-				  work.llnode)
+				  work.node.llist)
 		fn(&cb->work);
 }
 
@@ -460,7 +460,7 @@ __await_execution(struct i915_request *rq,
 	 * callback first, then checking the ACTIVE bit, we serialise with
 	 * the completed/retired request.
 	 */
-	if (llist_add(&cb->work.llnode, &signal->execute_cb)) {
+	if (llist_add(&cb->work.node.llist, &signal->execute_cb)) {
 		if (i915_request_is_active(signal) ||
 		    __request_in_flight(signal))
 			__notify_execute_cb_imm(signal);
include/linux/irq_work.h +21 -12
index 30823780c192..ec2a47a81e42 100644
@@ -14,28 +14,37 @@
  */
 
 struct irq_work {
-	union {
-		struct __call_single_node node;
-		struct {
-			struct llist_node llnode;
-			atomic_t flags;
-		};
-	};
+	struct __call_single_node node;
 	void (*func)(struct irq_work *);
 };
 
+#define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){	\
+	.node = { .u_flags = (_flags), },			\
+	.func = (_func),					\
+}
+
+#define IRQ_WORK_INIT(_func) __IRQ_WORK_INIT(_func, 0)
+#define IRQ_WORK_INIT_LAZY(_func) __IRQ_WORK_INIT(_func, IRQ_WORK_LAZY)
+#define IRQ_WORK_INIT_HARD(_func) __IRQ_WORK_INIT(_func, IRQ_WORK_HARD_IRQ)
+
+#define DEFINE_IRQ_WORK(name, _f)				\
+	struct irq_work name = IRQ_WORK_INIT(_f)
+
 static inline
 void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *))
 {
-	atomic_set(&work->flags, 0);
-	work->func = func;
+	*work = IRQ_WORK_INIT(func);
 }
 
-#define DEFINE_IRQ_WORK(name, _f) struct irq_work name = {	\
-		.flags = ATOMIC_INIT(0),			\
-		.func  = (_f)					\
+static inline bool irq_work_is_pending(struct irq_work *work)
+{
+	return atomic_read(&work->node.a_flags) & IRQ_WORK_PENDING;
 }
 
+static inline bool irq_work_is_busy(struct irq_work *work)
+{
+	return atomic_read(&work->node.a_flags) & IRQ_WORK_BUSY;
+}
 
 bool irq_work_queue(struct irq_work *work);
 bool irq_work_queue_on(struct irq_work *work, int cpu);
include/linux/irqflags.h +2 -2
index 3ed4e8771b64..fef2d43a7a1d 100644
@@ -109,12 +109,12 @@ do {						\
 
 # define lockdep_irq_work_enter(__work)					\
 	  do {								\
-		  if (!(atomic_read(&__work->flags) & IRQ_WORK_HARD_IRQ))\
+		  if (!(atomic_read(&__work->node.a_flags) & IRQ_WORK_HARD_IRQ))\
 			current->irq_config = 1;			\
 	  } while (0)
 # define lockdep_irq_work_exit(__work)					\
 	  do {								\
-		  if (!(atomic_read(&__work->flags) & IRQ_WORK_HARD_IRQ))\
+		  if (!(atomic_read(&__work->node.a_flags) & IRQ_WORK_HARD_IRQ))\
 			current->irq_config = 0;			\
 	  } while (0)
 
kernel/bpf/stackmap.c +1 -1
index 06065fa27124..599041cd0c8a 100644
@@ -298,7 +298,7 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
 	if (irqs_disabled()) {
 		if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
 			work = this_cpu_ptr(&up_read_work);
-			if (atomic_read(&work->irq_work.flags) & IRQ_WORK_BUSY) {
+			if (irq_work_is_busy(&work->irq_work)) {
 				/* cannot queue more up_read, fallback */
 				irq_work_busy = true;
 			}
kernel/irq_work.c +9 -9
index eca83965b631..fbff25adb574 100644
@@ -31,7 +31,7 @@ static bool irq_work_claim(struct irq_work *work)
 {
 	int oflags;
 
-	oflags = atomic_fetch_or(IRQ_WORK_CLAIMED | CSD_TYPE_IRQ_WORK, &work->flags);
+	oflags = atomic_fetch_or(IRQ_WORK_CLAIMED | CSD_TYPE_IRQ_WORK, &work->node.a_flags);
 	/*
 	 * If the work is already pending, no need to raise the IPI.
 	 * The pairing atomic_fetch_andnot() in irq_work_run() makes sure
@@ -53,12 +53,12 @@ void __weak arch_irq_work_raise(void)
 static void __irq_work_queue_local(struct irq_work *work)
 {
 	/* If the work is "lazy", handle it from next tick if any */
-	if (atomic_read(&work->flags) & IRQ_WORK_LAZY) {
-		if (llist_add(&work->llnode, this_cpu_ptr(&lazy_list)) &&
+	if (atomic_read(&work->node.a_flags) & IRQ_WORK_LAZY) {
+		if (llist_add(&work->node.llist, this_cpu_ptr(&lazy_list)) &&
 		    tick_nohz_tick_stopped())
 			arch_irq_work_raise();
 	} else {
-		if (llist_add(&work->llnode, this_cpu_ptr(&raised_list)))
+		if (llist_add(&work->node.llist, this_cpu_ptr(&raised_list)))
 			arch_irq_work_raise();
 	}
 }
@@ -102,7 +102,7 @@ bool irq_work_queue_on(struct irq_work *work, int cpu)
 	if (cpu != smp_processor_id()) {
 		/* Arch remote IPI send/receive backend aren't NMI safe */
 		WARN_ON_ONCE(in_nmi());
-		__smp_call_single_queue(cpu, &work->llnode);
+		__smp_call_single_queue(cpu, &work->node.llist);
 	} else {
 		__irq_work_queue_local(work);
 	}
@@ -142,7 +142,7 @@ void irq_work_single(void *arg)
 	 * to claim that work don't rely on us to handle their data
 	 * while we are in the middle of the func.
 	 */
-	flags = atomic_fetch_andnot(IRQ_WORK_PENDING, &work->flags);
+	flags = atomic_fetch_andnot(IRQ_WORK_PENDING, &work->node.a_flags);
 
 	lockdep_irq_work_enter(work);
 	work->func(work);
@@ -152,7 +152,7 @@ void irq_work_single(void *arg)
 	 * no-one else claimed it meanwhile.
 	 */
 	flags &= ~IRQ_WORK_PENDING;
-	(void)atomic_cmpxchg(&work->flags, flags, flags & ~IRQ_WORK_BUSY);
+	(void)atomic_cmpxchg(&work->node.a_flags, flags, flags & ~IRQ_WORK_BUSY);
 }
 
 static void irq_work_run_list(struct llist_head *list)
@@ -166,7 +166,7 @@ static void irq_work_run_list(struct llist_head *list)
 		return;
 
 	llnode = llist_del_all(list);
-	llist_for_each_entry_safe(work, tmp, llnode, llnode)
+	llist_for_each_entry_safe(work, tmp, llnode, node.llist)
 		irq_work_single(work);
 }
 
@@ -198,7 +198,7 @@ void irq_work_sync(struct irq_work *work)
 {
 	lockdep_assert_irqs_enabled();
 
-	while (atomic_read(&work->flags) & IRQ_WORK_BUSY)
+	while (irq_work_is_busy(work))
 		cpu_relax();
 }
 EXPORT_SYMBOL_GPL(irq_work_sync);
kernel/printk/printk.c +2 -4
index fe64a49344bf..9ef23d4b07c7 100644
@@ -3025,10 +3025,8 @@ static void wake_up_klogd_work_func(struct irq_work *irq_work)
 		wake_up_interruptible(&log_wait);
 }
 
-static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) = {
-	.func = wake_up_klogd_work_func,
-	.flags = ATOMIC_INIT(IRQ_WORK_LAZY),
-};
+static DEFINE_PER_CPU(struct irq_work, wake_up_klogd_work) =
+	IRQ_WORK_INIT_LAZY(wake_up_klogd_work_func);
 
 void wake_up_klogd(void)
 {
kernel/rcu/tree.c +1 -2
index bd04b09b84b3..ed4941f0bd59 100644
@@ -1311,8 +1311,6 @@ static int rcu_implicit_dynticks_qs(struct rcu_data *rdp)
 		if (IS_ENABLED(CONFIG_IRQ_WORK) &&
 		    !rdp->rcu_iw_pending && rdp->rcu_iw_gp_seq != rnp->gp_seq &&
 		    (rnp->ffmask & rdp->grpmask)) {
-			init_irq_work(&rdp->rcu_iw, rcu_iw_handler);
-			atomic_set(&rdp->rcu_iw.flags, IRQ_WORK_HARD_IRQ);
 			rdp->rcu_iw_pending = true;
 			rdp->rcu_iw_gp_seq = rnp->gp_seq;
 			irq_work_queue_on(&rdp->rcu_iw, rdp->cpu);
@@ -3964,6 +3962,7 @@ int rcutree_prepare_cpu(unsigned int cpu)
 	rdp->cpu_no_qs.b.norm = true;
 	rdp->core_needs_qs = false;
 	rdp->rcu_iw_pending = false;
+	rdp->rcu_iw = IRQ_WORK_INIT_HARD(rcu_iw_handler);
 	rdp->rcu_iw_gp_seq = rdp->gp_seq - 1;
 	trace_rcu_grace_period(rcu_state.name, rdp->gp_seq, TPS("cpuonl"));
 	raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
kernel/time/tick-sched.c +2 -4
index 81632cd5e3b7..1b734070f028 100644
@@ -243,10 +243,8 @@ static void nohz_full_kick_func(struct irq_work *work)
 	/* Empty, the tick restart happens on tick_nohz_irq_exit() */
 }
 
-static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) = {
-	.func = nohz_full_kick_func,
-	.flags = ATOMIC_INIT(IRQ_WORK_HARD_IRQ),
-};
+static DEFINE_PER_CPU(struct irq_work, nohz_full_kick_work) =
+	IRQ_WORK_INIT_HARD(nohz_full_kick_func);
 
 /*
  * Kick this CPU if it's full dynticks in order to force it to
kernel/trace/bpf_trace.c +1 -1
index 4517c8b66518..a6903912f7a0 100644
@@ -1086,7 +1086,7 @@ static int bpf_send_signal_common(u32 sig, enum pid_type type)
 			return -EINVAL;
 
 		work = this_cpu_ptr(&send_signal_work);
-		if (atomic_read(&work->irq_work.flags) & IRQ_WORK_BUSY)
+		if (irq_work_is_busy(&work->irq_work))
 			return -EBUSY;
 
 		/* Add the current task, which is the target of sending signal,
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#73 lore raw

PATCH 17/32 arch/x86: Add a new TIF flag for untrusted tasks

In reply to Joel Fernandes (Google)

Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.

This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.

---
 arch/x86/include/asm/thread_info.h | 2 ++
 kernel/sched/sched.h               | 6 ++++++
 2 files changed, 8 insertions(+)
arch/x86/include/asm/thread_info.h +2 -0
index 93277a8d2ef0..ae4f6196e38c 100644
@@ -99,6 +99,7 @@ struct thread_info {
 #define TIF_SPEC_FORCE_UPDATE	23	/* Force speculation MSR update in context switch */
 #define TIF_FORCED_TF		24	/* true if TF in eflags artificially */
 #define TIF_BLOCKSTEP		25	/* set when we want DEBUGCTLMSR_BTF */
+#define TIF_UNSAFE_RET		26	/* On return to process/guest, perform safety checks. */
 #define TIF_LAZY_MMU_UPDATES	27	/* task is updating the mmu lazily */
 #define TIF_SYSCALL_TRACEPOINT	28	/* syscall tracepoint instrumentation */
 #define TIF_ADDR32		29	/* 32-bit address space on 64 bits */
@@ -127,6 +128,7 @@ struct thread_info {
 #define _TIF_SPEC_FORCE_UPDATE	(1 << TIF_SPEC_FORCE_UPDATE)
 #define _TIF_FORCED_TF		(1 << TIF_FORCED_TF)
 #define _TIF_BLOCKSTEP		(1 << TIF_BLOCKSTEP)
+#define _TIF_UNSAFE_RET 	(1 << TIF_UNSAFE_RET)
 #define _TIF_LAZY_MMU_UPDATES	(1 << TIF_LAZY_MMU_UPDATES)
 #define _TIF_SYSCALL_TRACEPOINT	(1 << TIF_SYSCALL_TRACEPOINT)
 #define _TIF_ADDR32		(1 << TIF_ADDR32)
kernel/sched/sched.h +6 -0
index 5c258ab64052..615092cb693c 100644
@@ -2851,3 +2851,9 @@ static inline bool is_per_cpu_kthread(struct task_struct *p)
 
 void swake_up_all_locked(struct swait_queue_head *q);
 void __prepare_to_swait(struct swait_queue_head *q, struct swait_queue *wait);
+
+#ifdef CONFIG_SCHED_CORE
+#ifndef TIF_UNSAFE_RET
+#define TIF_UNSAFE_RET (0)
+#endif
+#endif
Signature · 1 line
2.29.2.299.gdc1121823c-goog
Acked-by on v1 17/32
#74 lore raw

Re: [PATCH -tip 17/32] arch/x86: Add a new TIF flag for untrusted tasks

Quoted 11 lines · Joel Fernandes (Google)

Add a new TIF flag to indicate whether the kernel needs to be careful and take additional steps to mitigate micro-architectural issues during entry into user or guest mode.

This new flag will be used by the series to determine if waiting is needed or not, during exit to user or guest mode.

---

#75 lore raw

PATCH 18/32 kernel/entry: Add support for core-wide protection of kernel-mode

In reply to Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

---
 .../admin-guide/kernel-parameters.txt         |  11 +
 include/linux/entry-common.h                  |  12 +-
 include/linux/sched.h                         |  12 +
 kernel/entry/common.c                         |  28 +-
 kernel/sched/core.c                           | 241 ++++++++++++++++++
 kernel/sched/sched.h                          |   3 +
 6 files changed, 304 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
index bd1a5b87a5e2..b185c6ed4aba 100644
@@ -4678,6 +4678,17 @@
 
 	sbni=		[NET] Granch SBNI12 leased line adapter
 
+	sched_core_protect_kernel=
+			[SCHED_CORE] Pause SMT siblings of a core running in
+			user mode, if at least one of the siblings of the core
+			is running in kernel mode. This is to guarantee that
+			kernel data is not leaked to tasks which are not trusted
+			by the kernel. A value of 0 disables protection, 1
+			enables protection. The default is 1. Note that protection
+			depends on the arch defining the _TIF_UNSAFE_RET flag.
+			Further, for protecting VMEXIT, arch needs to call
+			KVM entry/exit hooks.
+
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
 	schedstats=	[KNL,X86] Enable or disable scheduled statistics.
include/linux/entry-common.h +11 -1
index 1a128baf3628..022e1f114157 100644
@@ -33,6 +33,10 @@
 # define _TIF_PATCH_PENDING		(0)
 #endif
 
+#ifndef _TIF_UNSAFE_RET
+# define _TIF_UNSAFE_RET		(0)
+#endif
+
 #ifndef _TIF_UPROBE
 # define _TIF_UPROBE			(0)
 #endif
@@ -74,7 +78,7 @@
 #define EXIT_TO_USER_MODE_WORK						\
 	(_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE |		\
 	 _TIF_NEED_RESCHED | _TIF_PATCH_PENDING | _TIF_NOTIFY_SIGNAL |	\
-	 ARCH_EXIT_TO_USER_MODE_WORK)
+	 _TIF_UNSAFE_RET | ARCH_EXIT_TO_USER_MODE_WORK)
 
 /**
  * arch_check_user_regs - Architecture specific sanity check for user mode regs
@@ -444,4 +448,10 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs);
  */
 void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state);
 
+/* entry_kernel_protected - Is kernel protection on entry/exit into kernel supported? */
+static inline bool entry_kernel_protected(void)
+{
+	return IS_ENABLED(CONFIG_SCHED_CORE) && sched_core_kernel_protected()
+		&& _TIF_UNSAFE_RET != 0;
+}
 #endif
include/linux/sched.h +12 -0
index 7efce9c9d9cf..a60868165590 100644
@@ -2076,4 +2076,16 @@ int sched_trace_rq_nr_running(struct rq *rq);
 
 const struct cpumask *sched_trace_rd_span(struct root_domain *rd);
 
+#ifdef CONFIG_SCHED_CORE
+void sched_core_unsafe_enter(void);
+void sched_core_unsafe_exit(void);
+bool sched_core_wait_till_safe(unsigned long ti_check);
+bool sched_core_kernel_protected(void);
+#else
+#define sched_core_unsafe_enter(ignore) do { } while (0)
+#define sched_core_unsafe_exit(ignore) do { } while (0)
+#define sched_core_wait_till_safe(ignore) do { } while (0)
+#define sched_core_kernel_protected(ignore) do { } while (0)
+#endif
+
 #endif
kernel/entry/common.c +26 -2
index bc75c114c1b3..9d9d926f2a1c 100644
@@ -28,6 +28,9 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 
 	instrumentation_begin();
 	trace_hardirqs_off_finish();
+
+	if (entry_kernel_protected())
+		sched_core_unsafe_enter();
 	instrumentation_end();
 }
 
@@ -145,6 +148,26 @@ static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work)
 	arch_do_signal_or_restart(regs, ti_work & _TIF_SIGPENDING);
 }
 
+static unsigned long exit_to_user_get_work(void)
+{
+	unsigned long ti_work = READ_ONCE(current_thread_info()->flags);
+
+	if (!entry_kernel_protected())
+		return ti_work;
+
+#ifdef CONFIG_SCHED_CORE
+	ti_work &= EXIT_TO_USER_MODE_WORK;
+	if ((ti_work & _TIF_UNSAFE_RET) == ti_work) {
+		sched_core_unsafe_exit();
+		if (sched_core_wait_till_safe(EXIT_TO_USER_MODE_WORK)) {
+			sched_core_unsafe_enter(); /* not exiting to user yet. */
+		}
+	}
+
+	return READ_ONCE(current_thread_info()->flags);
+#endif
+}
+
 static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 					    unsigned long ti_work)
 {
@@ -182,7 +205,7 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 		 * enabled above.
 		 */
 		local_irq_disable_exit_to_user();
-		ti_work = READ_ONCE(current_thread_info()->flags);
+		ti_work = exit_to_user_get_work();
 	}
 
 	/* Return the latest work state for arch_exit_to_user_mode() */
@@ -191,9 +214,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 
 static void exit_to_user_mode_prepare(struct pt_regs *regs)
 {
-	unsigned long ti_work = READ_ONCE(current_thread_info()->flags);
+	unsigned long ti_work;
 
 	lockdep_assert_irqs_disabled();
+	ti_work = exit_to_user_get_work();
 
 	if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
 		ti_work = exit_to_user_mode_loop(regs, ti_work);
kernel/sched/core.c +241 -0

259 lines not on this page · raw message

kernel/sched/sched.h +3 -0
index 615092cb693c..be6691337bbb 100644
@@ -1074,6 +1074,8 @@ struct rq {
 	unsigned int		core_enabled;
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
+	struct irq_work		core_irq_work; /* To force HT into kernel */
+	unsigned int		core_this_unsafe_nest;
 
 	/* shared state */
 	unsigned int		core_task_seq;
@@ -1081,6 +1083,7 @@ struct rq {
 	unsigned long		core_cookie;
 	unsigned char		core_forceidle;
 	unsigned int		core_forceidle_seq;
+	unsigned int		core_unsafe_nest;
 #endif
 };
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#76 lore raw

Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode

Quoted 13 lines · Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

Quoted 6 lines

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

Oh gawd; this is horrible...

Quoted 21 lines
kernel/sched/core.c +21 -0
+bool sched_core_wait_till_safe(unsigned long ti_check)
+{
+	bool restart = false;
+	struct rq *rq;
+	int cpu;
+
+	/* We clear the thread flag only at the end, so no need to check for it. */
+	ti_check &= ~_TIF_UNSAFE_RET;
+
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+
+	if (!sched_core_enabled(rq))
+		goto ret;
+
+	/* Down grade to allow interrupts to prevent stop_machine lockups.. */
+	preempt_disable();
+	local_irq_enable();
+
+	/*
+	 * Wait till the core of this HT is not in an unsafe state.

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5170

Quoted 10 lines
kernel/sched/core.c +10 -0
+	 *
+	 * Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit().
+	 */
+	while (smp_load_acquire(&rq->core->core_unsafe_nest) > 0) {
+		cpu_relax();
+		if (READ_ONCE(current_thread_info()->flags) & ti_check) {
+			restart = true;
+			break;
+		}
+	}

What's that ACQUIRE for?

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5181

Quoted 11 lines
kernel/sched/core.c +11 -0
+
+	/* Upgrade it back to the expectations of entry code. */
+	local_irq_disable();
+	preempt_enable();
+
+ret:
+	if (!restart)
+		clear_tsk_thread_flag(current, TIF_UNSAFE_RET);
+
+	return restart;
+}

So if TIF_NEED_RESCHED gets set, we'll break out and reschedule, cute.

Quoted 74 lines
kernel/sched/core.c +74 -0
+void sched_core_unsafe_enter(void)
+{
+	const struct cpumask *smt_mask;
+	unsigned long flags;
+	struct rq *rq;
+	int i, cpu;
+
+	if (!static_branch_likely(&sched_core_protect_kernel))
+		return;
+
+	local_irq_save(flags);
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+	if (!sched_core_enabled(rq))
+		goto ret;
+
+	/* Ensure that on return to user/guest, we check whether to wait. */
+	if (current->core_cookie)
+		set_tsk_thread_flag(current, TIF_UNSAFE_RET);
+
+	/* Count unsafe_enter() calls received without unsafe_exit() on this CPU. */
+	rq->core_this_unsafe_nest++;
+
+	/*
+	 * Should not nest: enter() should only pair with exit(). Both are done
+	 * during the first entry into kernel and the last exit from kernel.
+	 * Nested kernel entries (such as nested interrupts) will only trigger
+	 * enter() and exit() on the outer most kernel entry and exit.
+	 */
+	if (WARN_ON_ONCE(rq->core_this_unsafe_nest != 1))
+		goto ret;
+
+	raw_spin_lock(rq_lockp(rq));
+	smt_mask = cpu_smt_mask(cpu);
+
+	/*
+	 * Contribute this CPU's unsafe_enter() to the core-wide unsafe_enter()
+	 * count.  The raw_spin_unlock() release semantics pairs with the nest
+	 * counter's smp_load_acquire() in sched_core_wait_till_safe().
+	 */
+	WRITE_ONCE(rq->core->core_unsafe_nest, rq->core->core_unsafe_nest + 1);
+
+	if (WARN_ON_ONCE(rq->core->core_unsafe_nest == UINT_MAX))
+		goto unlock;
+
+	if (irq_work_is_busy(&rq->core_irq_work)) {
+		/*
+		 * Do nothing more since we are in an IPI sent from another
+		 * sibling to enforce safety. That sibling would have sent IPIs
+		 * to all of the HTs.
+		 */
+		goto unlock;
+	}
+
+	/*
+	 * If we are not the first ones on the core to enter core-wide unsafe
+	 * state, do nothing.
+	 */
+	if (rq->core->core_unsafe_nest > 1)
+		goto unlock;
+
+	/* Do nothing more if the core is not tagged. */
+	if (!rq->core->core_cookie)
+		goto unlock;
+
+	for_each_cpu(i, smt_mask) {
+		struct rq *srq = cpu_rq(i);
+
+		if (i == cpu || cpu_is_offline(i))
+			continue;
+
+		if (!srq->curr->mm || is_task_rq_idle(srq->curr))
+			continue;
+

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5272

Quoted 10 lines
kernel/sched/core.c +10 -0
+		/* Skip if HT is not running a tagged task. */
+		if (!srq->curr->core_cookie && !srq->core_pick)
+			continue;
+
+		/*
+		 * Force sibling into the kernel by IPI. If work was already
+		 * pending, no new IPIs are sent. This is Ok since the receiver
+		 * would already be in the kernel, or on its way to it.
+		 */
+		irq_work_queue_on(&srq->core_irq_work, i);

Why irq_work though? Why not smp_send_reschedule(i)?

Quoted 6 lines
kernel/sched/core.c +6 -0
+	}
+unlock:
+	raw_spin_unlock(rq_lockp(rq));
+ret:
+	local_irq_restore(flags);
+}
#77 lore raw

Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode

Hi Peter,

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

Quoted 8 lines
Quoted 6 lines

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

Oh gawd; this is horrible...

Yeah, its the same issue we discussed earlier this year :)

Quoted 25 lines
Quoted 25 lines
kernel/sched/core.c +25 -0
+bool sched_core_wait_till_safe(unsigned long ti_check)
+{
+	bool restart = false;
+	struct rq *rq;
+	int cpu;
+
+	/* We clear the thread flag only at the end, so no need to check for it. */
+	ti_check &= ~_TIF_UNSAFE_RET;
+
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+
+	if (!sched_core_enabled(rq))
+		goto ret;
+
+	/* Down grade to allow interrupts to prevent stop_machine lockups.. */
+	preempt_disable();
+	local_irq_enable();
+
+	/*
+	 * Wait till the core of this HT is not in an unsafe state.
+	 *
+	 * Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit().
+	 */
+	while (smp_load_acquire(&rq->core->core_unsafe_nest) > 0) {
Quoted 8 lines

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5170

Quoted 6 lines
kernel/sched/core.c +6 -0
+		cpu_relax();
+		if (READ_ONCE(current_thread_info()->flags) & ti_check) {
+			restart = true;
+			break;
+		}
+	}

What's that ACQUIRE for?

I was concerned about something like below for weakly-ordered arch:

Kernel Attacker
------ --------
write unsafe=1

kernel code does stores while (unsafe == 1); (ACQUIRE)
^ ^

   | needs to be ordered	 | needs to be ordered
   v                             v
write unsafe=0 (RELEASE)        Attacker code.

Here, I want the access made by kernel code to be ordered WRT the write to the unsafe nesting counter variable, so that the attacker code does not see those accesses later.

It could be argued its a theoretical concern, but I wanted to play it safe. In particular, the existing uarch buffer flushing before entering the Attacker code might make it sufficiently impossible for Attacker to do anything bad even without the additional memory barriers.

Quoted 5 lines
Quoted 5 lines
kernel/sched/core.c +5 -0
+
+	/* Upgrade it back to the expectations of entry code. */
+	local_irq_disable();
+	preempt_enable();
+
Quoted 8 lines

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5181

Quoted 6 lines
kernel/sched/core.c +6 -0
+ret:
+	if (!restart)
+		clear_tsk_thread_flag(current, TIF_UNSAFE_RET);
+
+	return restart;
+}

So if TIF_NEED_RESCHED gets set, we'll break out and reschedule, cute.

Thanks.

Quoted 17 lines
Quoted 17 lines
kernel/sched/core.c +17 -0
+	/* Do nothing more if the core is not tagged. */
+	if (!rq->core->core_cookie)
+		goto unlock;
+
+	for_each_cpu(i, smt_mask) {
+		struct rq *srq = cpu_rq(i);
+
+		if (i == cpu || cpu_is_offline(i))
+			continue;
+
+		if (!srq->curr->mm || is_task_rq_idle(srq->curr))
+			continue;
+
+		/* Skip if HT is not running a tagged task. */
+		if (!srq->curr->core_cookie && !srq->core_pick)
+			continue;
+
Quoted 8 lines

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5272

Quoted 6 lines
kernel/sched/core.c +6 -0
+		/*
+		 * Force sibling into the kernel by IPI. If work was already
+		 * pending, no new IPIs are sent. This is Ok since the receiver
+		 * would already be in the kernel, or on its way to it.
+		 */
+		irq_work_queue_on(&srq->core_irq_work, i);

Why irq_work though? Why not smp_send_reschedule(i)?

I tried this approach. The main issue is sending IPIs if a previous IPI was already sent. I needed some mechanism (shared variables to detect this) so that an IPI would not be sent if one was just sent.

With irq_work, I get the irq_work_is_busy() which helps with exactly that. With other approaches I tried, I ran into issues with CSD locking and it seems irq_work implements the CSD locking the way I wanted to. So to avoid reinventing the wheel, I stuck to irq_work and it worked well.

thanks,

- Joel

maintainer
#78 lore raw

Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode

In reply to Joel Fernandes (Google)
Quoted 40 lines · Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

 .../admin-guide/kernel-parameters.txt         |  11 +
 include/linux/entry-common.h                  |  12 +-
 include/linux/sched.h                         |  12 +
 kernel/entry/common.c                         |  28 +-
 kernel/sched/core.c                           | 241 ++++++++++++++++++
 kernel/sched/sched.h                          |   3 +
 6 files changed, 304 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +4 -0
index bd1a5b87a5e2..b185c6ed4aba 100644
@@ -4678,6 +4678,17 @@
 
 	sbni=		[NET] Granch SBNI12 leased line adapter
 
+	sched_core_protect_kernel=
+			[SCHED_CORE] Pause SMT siblings of a core running in
+			user mode, if at least one of the siblings of the core
+			is running in kernel mode. This is to guarantee that

Documentation/admin-guide/kernel-parameters.txt line 4694

Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +7 -0
+			kernel data is not leaked to tasks which are not trusted
+			by the kernel. A value of 0 disables protection, 1
+			enables protection. The default is 1. Note that protection
+			depends on the arch defining the _TIF_UNSAFE_RET flag.
+			Further, for protecting VMEXIT, arch needs to call
+			KVM entry/exit hooks.
+
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
 	schedstats=	[KNL,X86] Enable or disable scheduled statistics.

So I don't like the parameter name, it's too long. Also I don't like it because its a boolean.

You're adding syscall,irq,kvm under a single knob where they're all due to different flavours of broken. Different hardware might want/need different combinations.

Hardware without MDS but with L1TF wouldn't need the syscall hook, but you're not givng a choice here. And this is generic code, you can't assume stuff like this.

#79 lore raw

Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode

Quoted 50 lines · Peter Zijlstra
Quoted 49 lines · Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

 .../admin-guide/kernel-parameters.txt         |  11 +
 include/linux/entry-common.h                  |  12 +-
 include/linux/sched.h                         |  12 +
 kernel/entry/common.c                         |  28 +-
 kernel/sched/core.c                           | 241 ++++++++++++++++++
 kernel/sched/sched.h                          |   3 +
 6 files changed, 304 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
index bd1a5b87a5e2..b185c6ed4aba 100644
@@ -4678,6 +4678,17 @@
 
 	sbni=		[NET] Granch SBNI12 leased line adapter
 
+	sched_core_protect_kernel=
+			[SCHED_CORE] Pause SMT siblings of a core running in
+			user mode, if at least one of the siblings of the core
+			is running in kernel mode. This is to guarantee that
+			kernel data is not leaked to tasks which are not trusted
+			by the kernel. A value of 0 disables protection, 1
+			enables protection. The default is 1. Note that protection
+			depends on the arch defining the _TIF_UNSAFE_RET flag.
+			Further, for protecting VMEXIT, arch needs to call
+			KVM entry/exit hooks.
+
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
Quoted 4 lines

Documentation/admin-guide/kernel-parameters.txt line 4694

Quoted 1 line
Documentation/admin-guide/kernel-parameters.txt +0 -0
 	schedstats=	[KNL,X86] Enable or disable scheduled statistics.

So I don't like the parameter name, it's too long. Also I don't like it because its a boolean.

Maybe ht_protect= then?

Quoted 3 lines

You're adding syscall,irq,kvm under a single knob where they're all due to different flavours of broken. Different hardware might want/need different combinations.

Ok, I can try to make it ht_protect=irq,syscall,kvm etc. And conditionally enable the protection. Does that work for you?

Quoted 4 lines

Hardware without MDS but with L1TF wouldn't need the syscall hook, but you're not givng a choice here. And this is generic code, you can't assume stuff like this.

Got it.

thanks,

- Joel

#80 lore raw

Re: [PATCH -tip 18/32] kernel/entry: Add support for core-wide protection of kernel-mode

In reply to Joel Fernandes (Google)
Quoted 134 lines · Joel Fernandes (Google)

Core-scheduling prevents hyperthreads in usermode from attacking each other, but it does not do anything about one of the hyperthreads entering the kernel for any reason. This leaves the door open for MDS and L1TF attacks with concurrent execution sequences between hyperthreads.

This patch therefore adds support for protecting all syscall and IRQ kernel mode entries. Care is taken to track the outermost usermode exit and entry using per-cpu counters. In cases where one of the hyperthreads enter the kernel, no additional IPIs are sent. Further, IPIs are avoided when not needed - example: idle and non-cookie HTs do not need to be forced into kernel mode.

More information about attacks:
For MDS, it is possible for syscalls, IRQ and softirq handlers to leak data to either host or guest attackers. For L1TF, it is possible to leak to guest attackers. There is no possible mitigation involving flushing of buffers to avoid this since the execution of attacker and victims happen concurrently on 2 or more HTs.

---
 .../admin-guide/kernel-parameters.txt         |  11 +
 include/linux/entry-common.h                  |  12 +-
 include/linux/sched.h                         |  12 +
 kernel/entry/common.c                         |  28 +-
 kernel/sched/core.c                           | 241 ++++++++++++++++++
 kernel/sched/sched.h                          |   3 +
 6 files changed, 304 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +11 -0
index bd1a5b87a5e2..b185c6ed4aba 100644
@@ -4678,6 +4678,17 @@
 
 	sbni=		[NET] Granch SBNI12 leased line adapter
 
+	sched_core_protect_kernel=
+			[SCHED_CORE] Pause SMT siblings of a core running in
+			user mode, if at least one of the siblings of the core
+			is running in kernel mode. This is to guarantee that
+			kernel data is not leaked to tasks which are not trusted
+			by the kernel. A value of 0 disables protection, 1
+			enables protection. The default is 1. Note that protection
+			depends on the arch defining the _TIF_UNSAFE_RET flag.
+			Further, for protecting VMEXIT, arch needs to call
+			KVM entry/exit hooks.
+
 	sched_debug	[KNL] Enables verbose scheduler debug messages.
 
 	schedstats=	[KNL,X86] Enable or disable scheduled statistics.
include/linux/entry-common.h +11 -1
index 1a128baf3628..022e1f114157 100644
@@ -33,6 +33,10 @@
 # define _TIF_PATCH_PENDING		(0)
 #endif
 
+#ifndef _TIF_UNSAFE_RET
+# define _TIF_UNSAFE_RET		(0)
+#endif
+
 #ifndef _TIF_UPROBE
 # define _TIF_UPROBE			(0)
 #endif
@@ -74,7 +78,7 @@
 #define EXIT_TO_USER_MODE_WORK						\
 	(_TIF_SIGPENDING | _TIF_NOTIFY_RESUME | _TIF_UPROBE |		\
 	 _TIF_NEED_RESCHED | _TIF_PATCH_PENDING | _TIF_NOTIFY_SIGNAL |	\
-	 ARCH_EXIT_TO_USER_MODE_WORK)
+	 _TIF_UNSAFE_RET | ARCH_EXIT_TO_USER_MODE_WORK)
 
 /**
  * arch_check_user_regs - Architecture specific sanity check for user mode regs
@@ -444,4 +448,10 @@ irqentry_state_t noinstr irqentry_nmi_enter(struct pt_regs *regs);
  */
 void noinstr irqentry_nmi_exit(struct pt_regs *regs, irqentry_state_t irq_state);
 
+/* entry_kernel_protected - Is kernel protection on entry/exit into kernel supported? */
+static inline bool entry_kernel_protected(void)
+{
+	return IS_ENABLED(CONFIG_SCHED_CORE) && sched_core_kernel_protected()
+		&& _TIF_UNSAFE_RET != 0;
+}
 #endif
include/linux/sched.h +12 -0
index 7efce9c9d9cf..a60868165590 100644
@@ -2076,4 +2076,16 @@ int sched_trace_rq_nr_running(struct rq *rq);
 
 const struct cpumask *sched_trace_rd_span(struct root_domain *rd);
 
+#ifdef CONFIG_SCHED_CORE
+void sched_core_unsafe_enter(void);
+void sched_core_unsafe_exit(void);
+bool sched_core_wait_till_safe(unsigned long ti_check);
+bool sched_core_kernel_protected(void);
+#else
+#define sched_core_unsafe_enter(ignore) do { } while (0)
+#define sched_core_unsafe_exit(ignore) do { } while (0)
+#define sched_core_wait_till_safe(ignore) do { } while (0)
+#define sched_core_kernel_protected(ignore) do { } while (0)
+#endif
+
 #endif
kernel/entry/common.c +3 -0
index bc75c114c1b3..9d9d926f2a1c 100644
@@ -28,6 +28,9 @@ static __always_inline void enter_from_user_mode(struct pt_regs *regs)
 
 	instrumentation_begin();
 	trace_hardirqs_off_finish();
+
+	if (entry_kernel_protected())
+		sched_core_unsafe_enter();
 	instrumentation_end();
 }
 
@@ -145,6 +148,26 @@ static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work)
 	arch_do_signal_or_restart(regs, ti_work & _TIF_SIGPENDING);

kernel/entry/common.c static void handle_signal_work(struct pt_regs *regs, unsigned long ti_work) · line 158

Quoted 10 lines
kernel/entry/common.c +8 -0
 }
 
+static unsigned long exit_to_user_get_work(void)
+{
+	unsigned long ti_work = READ_ONCE(current_thread_info()->flags);
+
+	if (!entry_kernel_protected())
+		return ti_work;
+
+#ifdef CONFIG_SCHED_CORE

This #ifdef is not necessary, entry_kernel_protected() does this check no? The code should also compile anyway from what I can see so far.

Quoted 100 lines
kernel/entry/common.c +15 -2
+	ti_work &= EXIT_TO_USER_MODE_WORK;
+	if ((ti_work & _TIF_UNSAFE_RET) == ti_work) {
+		sched_core_unsafe_exit();
+		if (sched_core_wait_till_safe(EXIT_TO_USER_MODE_WORK)) {
+			sched_core_unsafe_enter(); /* not exiting to user yet. */
+		}
+	}
+
+	return READ_ONCE(current_thread_info()->flags);
+#endif
+}
+
 static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 					    unsigned long ti_work)
 {
@@ -182,7 +205,7 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 		 * enabled above.
 		 */
 		local_irq_disable_exit_to_user();
-		ti_work = READ_ONCE(current_thread_info()->flags);
+		ti_work = exit_to_user_get_work();
 	}
 
 	/* Return the latest work state for arch_exit_to_user_mode() */
@@ -191,9 +214,10 @@ static unsigned long exit_to_user_mode_loop(struct pt_regs *regs,
 
 static void exit_to_user_mode_prepare(struct pt_regs *regs)
 {
-	unsigned long ti_work = READ_ONCE(current_thread_info()->flags);
+	unsigned long ti_work;
 
 	lockdep_assert_irqs_disabled();
+	ti_work = exit_to_user_get_work();
 
 	if (unlikely(ti_work & EXIT_TO_USER_MODE_WORK))
 		ti_work = exit_to_user_mode_loop(regs, ti_work);
kernel/sched/core.c +49 -0
index 20125431af87..7f807a84cc30 100644
@@ -76,6 +76,27 @@ __read_mostly int scheduler_running;
 
 #ifdef CONFIG_SCHED_CORE
 
+DEFINE_STATIC_KEY_TRUE(sched_core_protect_kernel);
+static int __init set_sched_core_protect_kernel(char *str)
+{
+	unsigned long val = 0;
+
+	if (!str)
+		return 0;
+
+	if (!kstrtoul(str, 0, &val) && !val)
+		static_branch_disable(&sched_core_protect_kernel);
+
+	return 1;
+}
+__setup("sched_core_protect_kernel=", set_sched_core_protect_kernel);
+
+/* Is the kernel protected by core scheduling? */
+bool sched_core_kernel_protected(void)
+{
+	return static_branch_likely(&sched_core_protect_kernel);
+}
+
 DEFINE_STATIC_KEY_FALSE(__sched_core_enabled);
 
 /* kernel prio, less is more */
@@ -5092,6 +5113,225 @@ static inline bool cookie_match(struct task_struct *a, struct task_struct *b)
 	return a->core_cookie == b->core_cookie;
 }
 
+/*
+ * Handler to attempt to enter kernel. It does nothing because the exit to
+ * usermode or guest mode will do the actual work (of waiting if needed).
+ */
+static void sched_core_irq_work(struct irq_work *work)
+{
+}
+
+static inline void init_sched_core_irq_work(struct rq *rq)
+{
+	init_irq_work(&rq->core_irq_work, sched_core_irq_work);
+}
+
+/*
+ * sched_core_wait_till_safe - Pause the caller's hyperthread until the core
+ * exits the core-wide unsafe state. Obviously the CPU calling this function
+ * should not be responsible for the core being in the core-wide unsafe state
+ * otherwise it will deadlock.
+ *
+ * @ti_check: We spin here with IRQ enabled and preempt disabled. Break out of
+ *            the loop if TIF flags are set and notify caller about it.
+ *
+ * IRQs should be disabled.
+ */
+bool sched_core_wait_till_safe(unsigned long ti_check)
+{
+	bool restart = false;
+	struct rq *rq;

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5153

Quoted 10 lines
kernel/sched/core.c +10 -0
+	int cpu;
+
+	/* We clear the thread flag only at the end, so no need to check for it. */
+	ti_check &= ~_TIF_UNSAFE_RET;
+
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+
+	if (!sched_core_enabled(rq))
+		goto ret;

Why do we need to deal with ti_check if sched_core_enabled() is false (two statements above)

Quoted 40 lines
kernel/sched/core.c +40 -0
+
+	/* Down grade to allow interrupts to prevent stop_machine lockups.. */
+	preempt_disable();
+	local_irq_enable();
+
+	/*
+	 * Wait till the core of this HT is not in an unsafe state.
+	 *
+	 * Pair with raw_spin_lock/unlock() in sched_core_unsafe_enter/exit().
+	 */
+	while (smp_load_acquire(&rq->core->core_unsafe_nest) > 0) {
+		cpu_relax();
+		if (READ_ONCE(current_thread_info()->flags) & ti_check) {
+			restart = true;
+			break;
+		}
+	}
+
+	/* Upgrade it back to the expectations of entry code. */
+	local_irq_disable();
+	preempt_enable();
+
+ret:
+	if (!restart)
+		clear_tsk_thread_flag(current, TIF_UNSAFE_RET);
+
+	return restart;
+}
+
+/*
+ * Enter the core-wide IRQ state. Sibling will be paused if it is running
+ * 'untrusted' code, until sched_core_unsafe_exit() is called. Every attempt to
+ * avoid sending useless IPIs is made. Must be called only from hard IRQ
+ * context.
+ */
+void sched_core_unsafe_enter(void)
+{
+	const struct cpumask *smt_mask;
+	unsigned long flags;
+	struct rq *rq;

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5203

Quoted 10 lines
kernel/sched/core.c +10 -0
+	int i, cpu;
+
+	if (!static_branch_likely(&sched_core_protect_kernel))
+		return;
+
+	local_irq_save(flags);
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+	if (!sched_core_enabled(rq))
+		goto ret;

I am not sure about the rules of this check, do we have to do this with irq's disabled? Given that sched_core_enabled() can only change under stop_machine, can't we optimize this check?

Quoted 19 lines
kernel/sched/core.c +19 -0
+
+	/* Ensure that on return to user/guest, we check whether to wait. */
+	if (current->core_cookie)
+		set_tsk_thread_flag(current, TIF_UNSAFE_RET);
+
+	/* Count unsafe_enter() calls received without unsafe_exit() on this CPU. */
+	rq->core_this_unsafe_nest++;
+
+	/*
+	 * Should not nest: enter() should only pair with exit(). Both are done
+	 * during the first entry into kernel and the last exit from kernel.
+	 * Nested kernel entries (such as nested interrupts) will only trigger
+	 * enter() and exit() on the outer most kernel entry and exit.
+	 */
+	if (WARN_ON_ONCE(rq->core_this_unsafe_nest != 1))
+		goto ret;
+
+	raw_spin_lock(rq_lockp(rq));
+	smt_mask = cpu_smt_mask(cpu);

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5232

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+	/*
+	 * Contribute this CPU's unsafe_enter() to the core-wide unsafe_enter()
+	 * count.  The raw_spin_unlock() release semantics pairs with the nest
+	 * counter's smp_load_acquire() in sched_core_wait_till_safe().
+	 */
+	WRITE_ONCE(rq->core->core_unsafe_nest, rq->core->core_unsafe_nest + 1);
+
+	if (WARN_ON_ONCE(rq->core->core_unsafe_nest == UINT_MAX))
+		goto unlock;

I am sure this is quite unlikely unless your concerned about overflows, will this all eventually move to under SCHED_DEBUG?

Quoted 61 lines
kernel/sched/core.c +61 -0
+
+	if (irq_work_is_busy(&rq->core_irq_work)) {
+		/*
+		 * Do nothing more since we are in an IPI sent from another
+		 * sibling to enforce safety. That sibling would have sent IPIs
+		 * to all of the HTs.
+		 */
+		goto unlock;
+	}
+
+	/*
+	 * If we are not the first ones on the core to enter core-wide unsafe
+	 * state, do nothing.
+	 */
+	if (rq->core->core_unsafe_nest > 1)
+		goto unlock;
+
+	/* Do nothing more if the core is not tagged. */
+	if (!rq->core->core_cookie)
+		goto unlock;
+
+	for_each_cpu(i, smt_mask) {
+		struct rq *srq = cpu_rq(i);
+
+		if (i == cpu || cpu_is_offline(i))
+			continue;
+
+		if (!srq->curr->mm || is_task_rq_idle(srq->curr))
+			continue;
+
+		/* Skip if HT is not running a tagged task. */
+		if (!srq->curr->core_cookie && !srq->core_pick)
+			continue;
+
+		/*
+		 * Force sibling into the kernel by IPI. If work was already
+		 * pending, no new IPIs are sent. This is Ok since the receiver
+		 * would already be in the kernel, or on its way to it.
+		 */
+		irq_work_queue_on(&srq->core_irq_work, i);
+	}
+unlock:
+	raw_spin_unlock(rq_lockp(rq));
+ret:
+	local_irq_restore(flags);
+}
+
+/*
+ * Process any work need for either exiting the core-wide unsafe state, or for
+ * waiting on this hyperthread if the core is still in this state.
+ *
+ * @idle: Are we called from the idle loop?
+ */
+void sched_core_unsafe_exit(void)
+{
+	unsigned long flags;
+	unsigned int nest;
+	struct rq *rq;
+	int cpu;
+
+	if (!static_branch_likely(&sched_core_protect_kernel))

kernel/sched/core.c static inline bool cookie_match(struct task_struct *a, struct task_struct *b) · line 5303

Quoted 10 lines
kernel/sched/core.c +10 -0
+		return;
+
+	local_irq_save(flags);
+	cpu = smp_processor_id();
+	rq = cpu_rq(cpu);
+
+	/* Do nothing if core-sched disabled. */
+	if (!sched_core_enabled(rq))
+		goto ret;
+

Same as above

Quoted 56 lines
kernel/sched/core.c +32 -0
+	/*
+	 * Can happen when a process is forked and the first return to user
+	 * mode is a syscall exit. Either way, there's nothing to do.
+	 */
+	if (rq->core_this_unsafe_nest == 0)
+		goto ret;
+
+	rq->core_this_unsafe_nest--;
+
+	/* enter() should be paired with exit() only. */
+	if (WARN_ON_ONCE(rq->core_this_unsafe_nest != 0))
+		goto ret;
+
+	raw_spin_lock(rq_lockp(rq));
+	/*
+	 * Core-wide nesting counter can never be 0 because we are
+	 * still in it on this CPU.
+	 */
+	nest = rq->core->core_unsafe_nest;
+	WARN_ON_ONCE(!nest);
+
+	WRITE_ONCE(rq->core->core_unsafe_nest, nest - 1);
+	/*
+	 * The raw_spin_unlock release semantics pairs with the nest counter's
+	 * smp_load_acquire() in sched_core_wait_till_safe().
+	 */
+	raw_spin_unlock(rq_lockp(rq));
+ret:
+	local_irq_restore(flags);
+}
+
 // XXX fairness/fwd progress conditions
 /*
  * Returns
@@ -5497,6 +5737,7 @@ static inline void sched_core_cpu_starting(unsigned int cpu)
 			rq = cpu_rq(i);
 			if (rq->core && rq->core == rq)
 				core_rq = rq;
+			init_sched_core_irq_work(rq);
 		}
 
 		if (!core_rq)
kernel/sched/sched.h +2 -0
index 615092cb693c..be6691337bbb 100644
@@ -1074,6 +1074,8 @@ struct rq {
 	unsigned int		core_enabled;
 	unsigned int		core_sched_seq;
 	struct rb_root		core_tree;
+	struct irq_work		core_irq_work; /* To force HT into kernel */
+	unsigned int		core_this_unsafe_nest;
 
 	/* shared state */
 	unsigned int		core_task_seq;

kernel/sched/sched.h struct rq { · line 1089

Quoted 7 lines
kernel/sched/sched.h +1 -0
@@ -1081,6 +1083,7 @@ struct rq {
 	unsigned long		core_cookie;
 	unsigned char		core_forceidle;
 	unsigned int		core_forceidle_seq;
+	unsigned int		core_unsafe_nest;
 #endif
 };
 

Balbir Singh.

#81 lore raw

PATCH 19/32 entry/idle: Enter and exit kernel protection during idle entry and exit

In reply to Joel Fernandes (Google)

Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.

While at it, remove a stale RCU comment.

---
 include/linux/entry-common.h | 18 ++++++++++++++++++
 kernel/sched/idle.c          | 11 ++++++-----
 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/entry-common.h +18 -0
index 022e1f114157..8f34ae625f83 100644
@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)
 	return IS_ENABLED(CONFIG_SCHED_CORE) && sched_core_kernel_protected()
 		&& _TIF_UNSAFE_RET != 0;
 }
+
+/**
+ * generic_idle_enter - General tasks to perform during idle entry.
+ */
+static inline void generic_idle_enter(void)
+{
+	/* Entering idle ends the protected kernel region. */
+	sched_core_unsafe_exit();
+}
+
+/**
+ * generic_idle_exit  - General tasks to perform during idle exit.
+ */
+static inline void generic_idle_exit(void)
+{
+	/* Exiting idle (re)starts the protected kernel region. */
+	sched_core_unsafe_enter();
+}
 #endif
kernel/sched/idle.c +6 -5
index 8bdb214eb78f..ee4f91396c31 100644
@@ -8,6 +8,7 @@
  */
 #include "sched.h"
 
+#include <linux/entry-common.h>
 #include <trace/events/power.h>
 
 /* Linker adds these: start and end of __cpuidle functions */
@@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup);
 
 static noinline int __cpuidle cpu_idle_poll(void)
 {
+	generic_idle_enter();
 	trace_cpu_idle(0, smp_processor_id());
 	stop_critical_timings();
 	rcu_idle_enter();
@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)
 	rcu_idle_exit();
 	start_critical_timings();
 	trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id());
+	generic_idle_exit();
 
 	return 1;
 }
@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)
 		return;
 	}
 
-	/*
-	 * The RCU framework needs to be told that we are entering an idle
-	 * section, so no more rcu read side critical sections and one more
-	 * step to the grace period
-	 */
+	generic_idle_enter();
 
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();
@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)
 	 */
 	if (WARN_ON_ONCE(irqs_disabled()))
 		local_irq_enable();
+
+	generic_idle_exit();
 }
 
 /*
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#82 lore raw

Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit

Quoted 30 lines · Joel Fernandes (Google)

Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.

While at it, remove a stale RCU comment.

---
 include/linux/entry-common.h | 18 ++++++++++++++++++
 kernel/sched/idle.c          | 11 ++++++-----
 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/entry-common.h +9 -0
index 022e1f114157..8f34ae625f83 100644
@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)
 	return IS_ENABLED(CONFIG_SCHED_CORE) && sched_core_kernel_protected()
 		&& _TIF_UNSAFE_RET != 0;
 }
+
+/**
+ * generic_idle_enter - General tasks to perform during idle entry.
+ */
+static inline void generic_idle_enter(void)
+{
+	/* Entering idle ends the protected kernel region. */
+	sched_core_unsafe_exit();
+}

include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 475

Quoted 10 lines
include/linux/entry-common.h +9 -0
+
+/**
+ * generic_idle_exit  - General tasks to perform during idle exit.
+ */
+static inline void generic_idle_exit(void)
+{
+	/* Exiting idle (re)starts the protected kernel region. */
+	sched_core_unsafe_enter();
+}
 #endif

That naming is terrible..

Quoted 42 lines
kernel/sched/idle.c +4 -5
index 8bdb214eb78f..ee4f91396c31 100644
@@ -8,6 +8,7 @@
  */
 #include "sched.h"
 
+#include <linux/entry-common.h>
 #include <trace/events/power.h>
 
 /* Linker adds these: start and end of __cpuidle functions */
@@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup);
 
 static noinline int __cpuidle cpu_idle_poll(void)
 {
+	generic_idle_enter();
 	trace_cpu_idle(0, smp_processor_id());
 	stop_critical_timings();
 	rcu_idle_enter();
@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)
 	rcu_idle_exit();
 	start_critical_timings();
 	trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id());
+	generic_idle_exit();
 
 	return 1;
 }
@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)
 		return;
 	}
 
-	/*
-	 * The RCU framework needs to be told that we are entering an idle
-	 * section, so no more rcu read side critical sections and one more
-	 * step to the grace period
-	 */
+	generic_idle_enter();
 
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 229

Quoted 6 lines
kernel/sched/idle.c +2 -0
@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)
 	 */
 	if (WARN_ON_ONCE(irqs_disabled()))
 		local_irq_enable();
+
+	generic_idle_exit();
 }

I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?

#83 lore raw

Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit

Quoted 35 lines · Peter Zijlstra
Quoted 34 lines · Joel Fernandes (Google)

Add a generic_idle_{enter,exit} helper function to enter and exit kernel protection when entering and exiting idle, respectively.

While at it, remove a stale RCU comment.

---
 include/linux/entry-common.h | 18 ++++++++++++++++++
 kernel/sched/idle.c          | 11 ++++++-----
 2 files changed, 24 insertions(+), 5 deletions(-)
include/linux/entry-common.h +13 -0
index 022e1f114157..8f34ae625f83 100644
@@ -454,4 +454,22 @@ static inline bool entry_kernel_protected(void)
 	return IS_ENABLED(CONFIG_SCHED_CORE) && sched_core_kernel_protected()
 		&& _TIF_UNSAFE_RET != 0;
 }
+
+/**
+ * generic_idle_enter - General tasks to perform during idle entry.
+ */
+static inline void generic_idle_enter(void)
+{
+	/* Entering idle ends the protected kernel region. */
+	sched_core_unsafe_exit();
+}
+
+/**
+ * generic_idle_exit  - General tasks to perform during idle exit.
+ */
Quoted 8 lines

include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 475

Quoted 6 lines
include/linux/entry-common.h +5 -0
+static inline void generic_idle_exit(void)
+{
+	/* Exiting idle (re)starts the protected kernel region. */
+	sched_core_unsafe_enter();
+}
 #endif

That naming is terrible..

Yeah sorry :-\. The naming I chose was to be aligned with the CONFIG_GENERIC_ENTRY naming. I am open to ideas on that.

Quoted 39 lines
Quoted 39 lines
kernel/sched/idle.c +4 -5
index 8bdb214eb78f..ee4f91396c31 100644
@@ -8,6 +8,7 @@
  */
 #include "sched.h"
 
+#include <linux/entry-common.h>
 #include <trace/events/power.h>
 
 /* Linker adds these: start and end of __cpuidle functions */
@@ -54,6 +55,7 @@ __setup("hlt", cpu_idle_nopoll_setup);
 
 static noinline int __cpuidle cpu_idle_poll(void)
 {
+	generic_idle_enter();
 	trace_cpu_idle(0, smp_processor_id());
 	stop_critical_timings();
 	rcu_idle_enter();
@@ -66,6 +68,7 @@ static noinline int __cpuidle cpu_idle_poll(void)
 	rcu_idle_exit();
 	start_critical_timings();
 	trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id());
+	generic_idle_exit();
 
 	return 1;
 }
@@ -156,11 +159,7 @@ static void cpuidle_idle_call(void)
 		return;
 	}
 
-	/*
-	 * The RCU framework needs to be told that we are entering an idle
-	 * section, so no more rcu read side critical sections and one more
-	 * step to the grace period
-	 */
+	generic_idle_enter();
 
Quoted 12 lines

kernel/sched/idle.c static void cpuidle_idle_call(void) · line 229

Quoted 9 lines
kernel/sched/idle.c +2 -0
 	if (cpuidle_not_available(drv, dev)) {
 		tick_nohz_idle_stop_tick();
@@ -225,6 +224,8 @@ static void cpuidle_idle_call(void)
 	 */
 	if (WARN_ON_ONCE(irqs_disabled()))
 		local_irq_enable();
+
+	generic_idle_exit();
 }

I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?

The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?

thanks,

- Joel

maintainer
#84 lore raw

Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit

Quoted 10 lines · Joel Fernandes
Quoted 6 lines · Peter Zijlstra

include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 470

Quoted 2 lines · Joel Fernandes (Google)
include/linux/entry-common.h +2 -0
+static inline void generic_idle_enter(void)
+static inline void generic_idle_exit(void)

That naming is terrible..

Yeah sorry :-\. The naming I chose was to be aligned with the CONFIG_GENERIC_ENTRY naming. I am open to ideas on that.

entry_idle_{enter,exit}() ?

Quoted 6 lines
Quoted 2 lines

I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?

The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?

I didn't necessarily mean using those hooks, even placing your new hooks right next to them would've covered the exact same code with less lines modified.

#85 lore raw

Re: [PATCH -tip 19/32] entry/idle: Enter and exit kernel protection during idle entry and exit

Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Joel Fernandes
Quoted 6 lines · Peter Zijlstra

include/linux/entry-common.h static inline bool entry_kernel_protected(void) · line 470

Quoted 2 lines · Joel Fernandes (Google)
include/linux/entry-common.h +2 -0
+static inline void generic_idle_enter(void)
+static inline void generic_idle_exit(void)

That naming is terrible..

Quoted 4 lines
Quoted 2 lines

Yeah sorry :-\. The naming I chose was to be aligned with the CONFIG_GENERIC_ENTRY naming. I am open to ideas on that.

entry_idle_{enter,exit}() ?

Sounds good to me.

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

I'm confused.. arch_cpu_idle_{enter,exit}() weren't conveniently placed for you?

The way this patch series works, it does not depend on arch code as much as possible. Since there are other arch that may need this patchset such as ARM, it may be better to keep it in the generic entry code. Thoughts?

I didn't necessarily mean using those hooks, even placing your new hooks right next to them would've covered the exact same code with less lines modified.

Ok sure. I will improve it this way for next posting.

thanks,

- Joel

#86 lore raw

PATCH 20/32 entry/kvm: Protect the kernel when entering from guest

In reply to Joel Fernandes (Google)

From: Vineeth Pillai <[email protected]>

Similar to how user to kernel mode transitions are protected in earlier patches, protect the entry into kernel from guest mode as well.

---
 arch/x86/kvm/x86.c        |  2 ++
 include/linux/entry-kvm.h | 12 ++++++++++++
 kernel/entry/kvm.c        | 33 +++++++++++++++++++++++++++++++++
 3 files changed, 47 insertions(+)
arch/x86/kvm/x86.c +2 -0
index 447edc0d1d5a..a50be74f70f1 100644
@@ -8910,6 +8910,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 */
 	smp_mb__after_srcu_read_unlock();
 
+	kvm_exit_to_guest_mode();
 	/*
 	 * This handles the case where a posted interrupt was
 	 * notified with kvm_vcpu_kick.
@@ -9003,6 +9004,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		}
 	}
 
+	kvm_enter_from_guest_mode();
 	local_irq_enable();
 	preempt_enable();
 
include/linux/entry-kvm.h +12 -0
index 9b93f8584ff7..67da6dcf442b 100644
@@ -77,4 +77,16 @@ static inline bool xfer_to_guest_mode_work_pending(void)
 }
 #endif /* CONFIG_KVM_XFER_TO_GUEST_WORK */
 
+/**
+ * kvm_enter_from_guest_mode - Hook called just after entering kernel from guest.
+ * Caller should ensure interrupts are off.
+ */
+void kvm_enter_from_guest_mode(void);
+
+/**
+ * kvm_exit_to_guest_mode - Hook called just before entering guest from kernel.
+ * Caller should ensure interrupts are off.
+ */
+void kvm_exit_to_guest_mode(void);
+
 #endif
kernel/entry/kvm.c +33 -0
index 49972ee99aff..3b603e8bd5da 100644
@@ -50,3 +50,36 @@ int xfer_to_guest_mode_handle_work(struct kvm_vcpu *vcpu)
 	return xfer_to_guest_mode_work(vcpu, ti_work);
 }
 EXPORT_SYMBOL_GPL(xfer_to_guest_mode_handle_work);
+
+/**
+ * kvm_enter_from_guest_mode - Hook called just after entering kernel from guest.
+ * Caller should ensure interrupts are off.
+ */
+void kvm_enter_from_guest_mode(void)
+{
+	if (!entry_kernel_protected())
+		return;
+	sched_core_unsafe_enter();
+}
+EXPORT_SYMBOL_GPL(kvm_enter_from_guest_mode);
+
+/**
+ * kvm_exit_to_guest_mode - Hook called just before entering guest from kernel.
+ * Caller should ensure interrupts are off.
+ */
+void kvm_exit_to_guest_mode(void)
+{
+	if (!entry_kernel_protected())
+		return;
+	sched_core_unsafe_exit();
+
+	/*
+	 * Wait here instead of in xfer_to_guest_mode_handle_work(). The reason
+	 * is because in vcpu_run(), xfer_to_guest_mode_handle_work() is called
+	 * when a vCPU was either runnable or blocked. However, we only care
+	 * about the runnable case (VM entry/exit) which is handled by
+	 * vcpu_enter_guest().
+	 */
+	sched_core_wait_till_safe(XFER_TO_GUEST_MODE_WORK);
+}
+EXPORT_SYMBOL_GPL(kvm_exit_to_guest_mode);
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#87 lore raw

PATCH 21/32 sched: CGroup tagging interface for core scheduling

In reply to Joel Fernandes (Google)

From: Peter Zijlstra <[email protected]>

Marks all tasks in a cgroup as matching for core-scheduling.

A task will need to be moved into the core scheduler queue when the cgroup it belongs to is tagged to run with core scheduling. Similarly the task will need to be moved out of the core scheduler queue when the cgroup is untagged.

Also after we forked a task, its core scheduler queue's presence will need to be updated according to its new cgroup's status.

Use stop machine mechanism to update all tasks in a cgroup to prevent a new task from sneaking into the cgroup, and missed out from the update while we iterates through all the tasks in the cgroup. A more complicated scheme could probably avoid the stop machine. Such scheme will also need to resovle inconsistency between a task's cgroup core scheduling tag and residency in core scheduler queue.

We are opting for the simple stop machine mechanism for now that avoids such complications.

Core scheduler has extra overhead. Enable it only for core with more than one SMT hardware threads.

---
 kernel/sched/core.c  | 183 +++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/sched.h |   4 +
 2 files changed, 181 insertions(+), 6 deletions(-)
kernel/sched/core.c +177 -6
index 7f807a84cc30..b99a7493d590 100644
@@ -157,6 +157,37 @@ static inline bool __sched_core_less(struct task_struct *a, struct task_struct *
 	return false;
 }
 
+static bool sched_core_empty(struct rq *rq)
+{
+	return RB_EMPTY_ROOT(&rq->core_tree);
+}
+
+static bool sched_core_enqueued(struct task_struct *task)
+{
+	return !RB_EMPTY_NODE(&task->core_node);
+}
+
+static struct task_struct *sched_core_first(struct rq *rq)
+{
+	struct task_struct *task;
+
+	task = container_of(rb_first(&rq->core_tree), struct task_struct, core_node);
+	return task;
+}
+
+static void sched_core_flush(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+	struct task_struct *task;
+
+	while (!sched_core_empty(rq)) {
+		task = sched_core_first(rq);
+		rb_erase(&task->core_node, &rq->core_tree);
+		RB_CLEAR_NODE(&task->core_node);
+	}
+	rq->core->core_task_seq++;
+}
+
 static void sched_core_enqueue(struct rq *rq, struct task_struct *p)
 {
 	struct rb_node *parent, **node;
@@ -188,10 +219,11 @@ static void sched_core_dequeue(struct rq *rq, struct task_struct *p)
 {
 	rq->core->core_task_seq++;
 
-	if (!p->core_cookie)
+	if (!sched_core_enqueued(p))
 		return;
 
 	rb_erase(&p->core_node, &rq->core_tree);
+	RB_CLEAR_NODE(&p->core_node);
 }
 
 /*
@@ -255,8 +287,24 @@ static int __sched_core_stopper(void *data)
 	bool enabled = !!(unsigned long)data;
 	int cpu;
 
-	for_each_possible_cpu(cpu)
-		cpu_rq(cpu)->core_enabled = enabled;
+	for_each_possible_cpu(cpu) {
+		struct rq *rq = cpu_rq(cpu);
+
+		WARN_ON_ONCE(enabled == rq->core_enabled);
+
+		if (!enabled || (enabled && cpumask_weight(cpu_smt_mask(cpu)) >= 2)) {
+			/*
+			 * All active and migrating tasks will have already
+			 * been removed from core queue when we clear the
+			 * cgroup tags. However, dying tasks could still be
+			 * left in core queue. Flush them here.
+			 */
+			if (!enabled)
+				sched_core_flush(cpu);
+
+			rq->core_enabled = enabled;
+		}
+	}
 
 	return 0;
 }
@@ -266,7 +314,11 @@ static int sched_core_count;
 
 static void __sched_core_enable(void)
 {
-	// XXX verify there are no cookie tasks (yet)
+	int cpu;
+
+	/* verify there are no cookie tasks (yet) */
+	for_each_online_cpu(cpu)
+		BUG_ON(!sched_core_empty(cpu_rq(cpu)));
 
 	static_branch_enable(&__sched_core_enabled);
 	stop_machine(__sched_core_stopper, (void *)true, NULL);
@@ -274,8 +326,6 @@ static void __sched_core_enable(void)
 
 static void __sched_core_disable(void)
 {
-	// XXX verify there are no cookie tasks (left)
-
 	stop_machine(__sched_core_stopper, (void *)false, NULL);
 	static_branch_disable(&__sched_core_enabled);
 }
@@ -300,6 +350,7 @@ void sched_core_put(void)
 
 static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
 static inline void sched_core_dequeue(struct rq *rq, struct task_struct *p) { }
+static bool sched_core_enqueued(struct task_struct *task) { return false; }
 
 #endif /* CONFIG_SCHED_CORE */
 
@@ -3978,6 +4029,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 #ifdef CONFIG_SMP
 	plist_node_init(&p->pushable_tasks, MAX_PRIO);
 	RB_CLEAR_NODE(&p->pushable_dl_tasks);
+#endif
+#ifdef CONFIG_SCHED_CORE
+	RB_CLEAR_NODE(&p->core_node);
 #endif
 	return 0;
 }
@@ -7979,6 +8033,9 @@ void init_idle(struct task_struct *idle, int cpu)
 #ifdef CONFIG_SMP
 	sprintf(idle->comm, "%s/%d", INIT_TASK_COMM, cpu);
 #endif
+#ifdef CONFIG_SCHED_CORE
+	RB_CLEAR_NODE(&idle->core_node);
+#endif
 }
 
 #ifdef CONFIG_SMP
@@ -8995,6 +9052,15 @@ static void sched_change_group(struct task_struct *tsk, int type)
 	tg = container_of(task_css_check(tsk, cpu_cgrp_id, true),
 			  struct task_group, css);
 	tg = autogroup_task_group(tsk, tg);
+
+#ifdef CONFIG_SCHED_CORE
+	if ((unsigned long)tsk->sched_task_group == tsk->core_cookie)
+		tsk->core_cookie = 0UL;
+
+	if (tg->tagged /* && !tsk->core_cookie ? */)
+		tsk->core_cookie = (unsigned long)tg;
+#endif
+
 	tsk->sched_task_group = tg;
 
 #ifdef CONFIG_FAIR_GROUP_SCHED
@@ -9087,6 +9153,18 @@ static int cpu_cgroup_css_online(struct cgroup_subsys_state *css)
 	return 0;
 }
 
+static void cpu_cgroup_css_offline(struct cgroup_subsys_state *css)
+{
+#ifdef CONFIG_SCHED_CORE
+	struct task_group *tg = css_tg(css);
+
+	if (tg->tagged) {
+		sched_core_put();
+		tg->tagged = 0;
+	}
+#endif
+}
+
 static void cpu_cgroup_css_released(struct cgroup_subsys_state *css)
 {
 	struct task_group *tg = css_tg(css);
@@ -9652,6 +9730,82 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css,
 }
 #endif /* CONFIG_RT_GROUP_SCHED */
 
+#ifdef CONFIG_SCHED_CORE
+static u64 cpu_core_tag_read_u64(struct cgroup_subsys_state *css, struct cftype *cft)
+{
+	struct task_group *tg = css_tg(css);
+
+	return !!tg->tagged;
+}
+
+struct write_core_tag {
+	struct cgroup_subsys_state *css;
+	int val;
+};
+
+static int __sched_write_tag(void *data)
+{
+	struct write_core_tag *tag = (struct write_core_tag *) data;
+	struct cgroup_subsys_state *css = tag->css;
+	int val = tag->val;
+	struct task_group *tg = css_tg(tag->css);
+	struct css_task_iter it;
+	struct task_struct *p;
+
+	tg->tagged = !!val;
+
+	css_task_iter_start(css, 0, &it);
+	/*
+	 * Note: css_task_iter_next will skip dying tasks.
+	 * There could still be dying tasks left in the core queue
+	 * when we set cgroup tag to 0 when the loop is done below.
+	 */
+	while ((p = css_task_iter_next(&it))) {
+		p->core_cookie = !!val ? (unsigned long)tg : 0UL;
+
+		if (sched_core_enqueued(p)) {
+			sched_core_dequeue(task_rq(p), p);
+			if (!p->core_cookie)
+				continue;
+		}
+
+		if (sched_core_enabled(task_rq(p)) &&
+		    p->core_cookie && task_on_rq_queued(p))
+			sched_core_enqueue(task_rq(p), p);
+
+	}
+	css_task_iter_end(&it);
+
+	return 0;
+}
+
+static int cpu_core_tag_write_u64(struct cgroup_subsys_state *css, struct cftype *cft, u64 val)
+{
+	struct task_group *tg = css_tg(css);
+	struct write_core_tag wtag;
+
+	if (val > 1)
+		return -ERANGE;
+
+	if (!static_branch_likely(&sched_smt_present))
+		return -EINVAL;
+
+	if (tg->tagged == !!val)
+		return 0;
+
+	if (!!val)
+		sched_core_get();
+
+	wtag.css = css;
+	wtag.val = val;
+	stop_machine(__sched_write_tag, (void *) &wtag, NULL);
+	if (!val)
+		sched_core_put();
+
+	return 0;
+}
+#endif
+
 static struct cftype cpu_legacy_files[] = {
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	{
@@ -9688,6 +9842,14 @@ static struct cftype cpu_legacy_files[] = {
 		.write_u64 = cpu_rt_period_write_uint,
 	},
 #endif
+#ifdef CONFIG_SCHED_CORE
+	{
+		.name = "tag",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = cpu_core_tag_read_u64,
+		.write_u64 = cpu_core_tag_write_u64,
+	},
+#endif
 #ifdef CONFIG_UCLAMP_TASK_GROUP
 	{
 		.name = "uclamp.min",
@@ -9861,6 +10023,14 @@ static struct cftype cpu_files[] = {
 		.write_s64 = cpu_weight_nice_write_s64,
 	},
 #endif
+#ifdef CONFIG_SCHED_CORE
+	{
+		.name = "tag",
+		.flags = CFTYPE_NOT_ON_ROOT,
+		.read_u64 = cpu_core_tag_read_u64,
+		.write_u64 = cpu_core_tag_write_u64,
+	},
+#endif
 #ifdef CONFIG_CFS_BANDWIDTH
 	{
 		.name = "max",
@@ -9889,6 +10059,7 @@ static struct cftype cpu_files[] = {
 struct cgroup_subsys cpu_cgrp_subsys = {
 	.css_alloc	= cpu_cgroup_css_alloc,
 	.css_online	= cpu_cgroup_css_online,
+	.css_offline	= cpu_cgroup_css_offline,
 	.css_released	= cpu_cgroup_css_released,
 	.css_free	= cpu_cgroup_css_free,
 	.css_extra_stat_show = cpu_extra_stat_show,
kernel/sched/sched.h +4 -0
index be6691337bbb..3ba08973ed58 100644
@@ -377,6 +377,10 @@ struct cfs_bandwidth {
 struct task_group {
 	struct cgroup_subsys_state css;
 
+#ifdef CONFIG_SCHED_CORE
+	int			tagged;
+#endif
+
 #ifdef CONFIG_FAIR_GROUP_SCHED
 	/* schedulable entities of this group on each CPU */
 	struct sched_entity	**se;
Signature · 1 line
2.29.2.299.gdc1121823c-goog
#88 lore raw

PATCH 22/32 sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)

In order to prevent interference and clearly support both per-task and CGroup APIs, split the cookie into 2 and allow it to be set from either per-task, or CGroup API. The final cookie is the combined value of both and is computed when the stop-machine executes during a change of cookie.

Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.

While at it, refactor the CGroup code a bit. Future patches will introduce more APIs and support.

---
 include/linux/sched.h |   2 +
 kernel/sched/core.c   | 241 ++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/debug.c  |   4 +
 3 files changed, 236 insertions(+), 11 deletions(-)
include/linux/sched.h +2 -0
index a60868165590..c6a3b0fa952b 100644
@@ -688,6 +688,8 @@ struct task_struct {
 #ifdef CONFIG_SCHED_CORE
 	struct rb_node			core_node;
 	unsigned long			core_cookie;
+	unsigned long			core_task_cookie;
+	unsigned long			core_group_cookie;
 	unsigned int			core_occupation;
 #endif
 
kernel/sched/core.c +230 -11

271 lines not on this page · raw message

kernel/sched/debug.c +4 -0
index 60a922d3f46f..8c452b8010ad 100644
@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		__PS("clock-delta", t1-t0);
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	__PS("core_cookie", p->core_cookie);
+#endif
+
 	sched_show_numa(p, m);
 }
 
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#89 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 3 lines · Joel Fernandes (Google)

Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.

Might be useful to explain why exactly none of the many pid_t's are good enough.

#90 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Joel Fernandes (Google)

Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.

Might be useful to explain why exactly none of the many pid_t's are good enough.

I thought about this already and it does not seem a good fit. When 2 processes share, it is possible that more processes are added to that logical group. Then the original processes that share die, but if we hold on to the pid_t or task_struct, that would be awkward. It seemed introducing a new refcounted struct was the right way to go. I can add these details to the change log.

thanks!

- Joel

maintainer
#91 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779

Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13 -0
+void sched_core_tag_requeue(struct task_struct *p, unsigned long cookie, bool group)
+{
+	if (!p)
+		return;
+
+	if (group)
+		p->core_group_cookie = cookie;
+	else
+		p->core_task_cookie = cookie;
+
+	/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */
+	p->core_cookie = (p->core_task_cookie <<
+				(sizeof(unsigned long) * 4)) + p->core_group_cookie;

This seems dangerous; afaict there is nothing that prevents cookie collision.

#92 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 17 lines · Peter Zijlstra

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779

Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13 -0
+void sched_core_tag_requeue(struct task_struct *p, unsigned long cookie, bool group)
+{
+	if (!p)
+		return;
+
+	if (group)
+		p->core_group_cookie = cookie;
+	else
+		p->core_task_cookie = cookie;
+
+	/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */
+	p->core_cookie = (p->core_task_cookie <<
+				(sizeof(unsigned long) * 4)) + p->core_group_cookie;

This seems dangerous; afaict there is nothing that prevents cookie collision.

This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.

I'll squash it next time I post to prevent confusion. Thanks,

- Joel

maintainer
#93 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 19 lines · Joel Fernandes
Quoted 17 lines · Peter Zijlstra

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779

Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13 -0
+void sched_core_tag_requeue(struct task_struct *p, unsigned long cookie, bool group)
+{
+	if (!p)
+		return;
+
+	if (group)
+		p->core_group_cookie = cookie;
+	else
+		p->core_task_cookie = cookie;
+
+	/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */
+	p->core_cookie = (p->core_task_cookie <<
+				(sizeof(unsigned long) * 4)) + p->core_group_cookie;

This seems dangerous; afaict there is nothing that prevents cookie collision.

Quoted 2 lines

This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.

So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.

	COOKIE_CMP_RETURN(task_cookie);
	COOKIE_CMP_RETURN(group_cookie);
	COOKIE_CMP_RETURN(color);

So if task_cookie matches, we consider group_cookie, if that matches we consider color.

Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.

(I don't understand the color thing, but lets have that discussion in that subthread)

Which means you only need a single active cookie field.

IOW, you're just making things complicated and expensive.

#94 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 15 lines · Peter Zijlstra

So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.

        COOKIE_CMP_RETURN(task_cookie);
        COOKIE_CMP_RETURN(group_cookie);
        COOKIE_CMP_RETURN(color);

So if task_cookie matches, we consider group_cookie, if that matches we consider color.

Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.

Quoted 7 lines

(I don't understand the color thing, but lets have that discussion in that subthread)

Which means you only need a single active cookie field.

IOW, you're just making things complicated and expensive.

For the per-task interface, I believe we still want to prevent two tasks that share a task cookie from sharing an overall cookie if they are in two separately tagged groups (Joel please correct me if I'm mistaken there). That's why in Joel's older patch, the overall cookie was a combination of the task and group cookies. My concern about that was the potential cookie collision.

I followed up on the 'color' portion in the other thread.

Thanks,
Josh

maintainer
#95 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 24 lines · Josh Don
Quoted 15 lines · Peter Zijlstra

So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.

        COOKIE_CMP_RETURN(task_cookie);
        COOKIE_CMP_RETURN(group_cookie);
        COOKIE_CMP_RETURN(color);

So if task_cookie matches, we consider group_cookie, if that matches we consider color.

Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.

Quoted 7 lines

(I don't understand the color thing, but lets have that discussion in that subthread)

Which means you only need a single active cookie field.

IOW, you're just making things complicated and expensive.

Quoted 6 lines

For the per-task interface, I believe we still want to prevent two tasks that share a task cookie from sharing an overall cookie if they are in two separately tagged groups (Joel please correct me if I'm mistaken there). That's why in Joel's older patch, the overall cookie was a combination of the task and group cookies. My concern about that was the potential cookie collision.

Then disallow sharing a task cookie when the tasks are in different cgroups or disallow cgroup movement when they share a cookie.

#96 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 3 lines · Peter Zijlstra

Then disallow sharing a task cookie when the tasks are in different cgroups or disallow cgroup movement when they share a cookie.

Yes, we could restrict task cookie sharing to tasks that are in the same cgroup. Then the cookie easily just becomes a single value; either the task cookie or group cookie.

The advantage of the approach with the cookie struct is that it is easily extensible, and allows for trust models that don't conform exactly to the cgroup hierarchy (ie. our discussion on cookie color). The overhead of the approach seems tolerable, given that updates to a task's cookie are not in fast paths (ie. prctl, setting cgroup cookie, sched_move_task). Are you more concerned with the added complexity of maintaining the RB tree, refcounts, etc?

#97 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Peter Zijlstra

Hi Peter,

Quoted 26 lines · Peter Zijlstra
Quoted 19 lines · Joel Fernandes
Quoted 17 lines · Peter Zijlstra

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9779

Quoted 13 lines · Joel Fernandes (Google)
kernel/sched/core.c +13 -0
+void sched_core_tag_requeue(struct task_struct *p, unsigned long cookie, bool group)
+{
+	if (!p)
+		return;
+
+	if (group)
+		p->core_group_cookie = cookie;
+	else
+		p->core_task_cookie = cookie;
+
+	/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */
+	p->core_cookie = (p->core_task_cookie <<
+				(sizeof(unsigned long) * 4)) + p->core_group_cookie;

This seems dangerous; afaict there is nothing that prevents cookie collision.

Quoted 2 lines

This is fixed in a later patch by Josh "sched: Refactor core cookie into struct" where we are having independent fields for each type of cookie.

So I don't think that later patch is right... That is, it works, but afaict it's massive overkill.

Quoted 6 lines
	COOKIE_CMP_RETURN(task_cookie);
	COOKIE_CMP_RETURN(group_cookie);
	COOKIE_CMP_RETURN(color);

So if task_cookie matches, we consider group_cookie, if that matches we consider color.

Yes, the cookie is a compound one. A cookie structure is created here which has all 3 components. The final cookie value is a pointer to this compound structure.

Quoted 4 lines

Now, afaict that's semantically exactly the same as just using the narrowest cookie. That is, use the task cookie if there is, and then, walking the cgroup hierarchy (up) pick the first cgroup cookie.

[..]

Quoted 1 line

Which means you only need a single active cookie field.

Its not the same. The "compounded" cookie is needed to enforce the CGroup delegation model. This is exactly how both uclamp and cpusets work. For uclamp, if we take uclamp.min as an example, if the CGroup sets the uclamp.min of a group to 0, then even if you use the per-task interface (sched_setattr) for setting the task's clamp - the "effective uclamp.min" of the task as seen in /proc/pid/sched will still be 0.

Similar thing here, if 2 tasks belong to 2 different CGroups and each group is tagged with its own tag, then if you use the per-task interface to make the 2 tasks share a core, the "effective" sharing is still such that the tasks will not share a core -- because the CGroup decided to make it so.

I would like to really maintain this model. Doing it any other way is confusing - we have already discussed doing it this way before. With that way you end up failing one interface if another one was already used. Been there, done that. It sucks a lot.

Quoted 1 line

IOW, you're just making things complicated and expensive.

The cost of the additional comparisons you mentioned is only in the slow path (i.e. when someone joins or leaves a group). Once the task_struct's cookie field is set, the cost is not any more than what it was before.

Any other thoughts?

thanks,

- Joel

maintainer
#98 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)
Quoted 1 line · Joel Fernandes (Google)

+ * sched_core_tag_requeue - Common helper for all interfaces to set a cookie.

sched_core_set_cookie() would be a saner name, given that description, don't you think?

#99 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 6 lines · Peter Zijlstra
Quoted 1 line · Joel Fernandes (Google)

+ * sched_core_tag_requeue - Common helper for all interfaces to set a cookie.

sched_core_set_cookie() would be a saner name, given that description, don't you think?

Yeah, Josh is better than me at naming so he changed it to sched_core_update_cookie() already :-). Hopefully that's Ok too with you.

thanks,

- Joel

maintainer
#100 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)
Quoted 12 lines · Joel Fernandes (Google)
kernel/sched/core.c +12 -0
+/*
+ * Ensure that the task has been requeued. The stopper ensures that the task cannot
+ * be migrated to a different CPU while its core scheduler queue state is being updated.
+ * It also makes sure to requeue a task if it was running actively on another CPU.
+ */
+static int sched_core_task_join_stopper(void *data)
+{
+	struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data;
+	int i;
+
+	for (i = 0; i < 2; i++)
+		sched_core_tag_requeue(tag->tasks[i], tag->cookies[i], false /* !group */);

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9871

Quoted 10 lines
kernel/sched/core.c +8 -0
+
+	return 0;
+}
+
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
+{
 
+	stop_machine(sched_core_task_join_stopper, (void *)&wr, NULL);
 
+}

This is *REALLY* terrible...

#101 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 21 lines · Peter Zijlstra
Quoted 18 lines · Joel Fernandes (Google)
kernel/sched/core.c +18 -0
+/*
+ * Ensure that the task has been requeued. The stopper ensures that the task cannot
+ * be migrated to a different CPU while its core scheduler queue state is being updated.
+ * It also makes sure to requeue a task if it was running actively on another CPU.
+ */
+static int sched_core_task_join_stopper(void *data)
+{
+	struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data;
+	int i;
+
+	for (i = 0; i < 2; i++)
+		sched_core_tag_requeue(tag->tasks[i], tag->cookies[i], false /* !group */);
+
+	return 0;
+}
+
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
+{
Quoted 5 lines

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9959

Quoted 3 lines
kernel/sched/core.c +2 -0
+	stop_machine(sched_core_task_join_stopper, (void *)&wr, NULL);
 
+}

This is *REALLY* terrible...

I pulled this bit from your original patch. Are you concerned about the stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.

maintainer
#102 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 24 lines · Joel Fernandes
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Joel Fernandes (Google)
kernel/sched/core.c +19 -0
+/*
+ * Ensure that the task has been requeued. The stopper ensures that the task cannot
+ * be migrated to a different CPU while its core scheduler queue state is being updated.
+ * It also makes sure to requeue a task if it was running actively on another CPU.
+ */
+static int sched_core_task_join_stopper(void *data)
+{
+	struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data;
+	int i;
+
+	for (i = 0; i < 2; i++)
+		sched_core_tag_requeue(tag->tasks[i], tag->cookies[i], false /* !group */);
+
+	return 0;
+}
+
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
+{
 
+	stop_machine(sched_core_task_join_stopper, (void *)&wr, NULL);
Quoted 9 lines
Quoted 3 lines
Quoted 1 line

+}

This is *REALLY* terrible...

I pulled this bit from your original patch. Are you concerned about the stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.

Yeah.. so I can (and was planning on) remove stop_machine() from sched_core_{dis,en}able() before merging it.

(there's two options, one uses stop_cpus() with the SMT mask, the other RCU)

This though is exposing stop_machine() to joe user. Everybody is allowed to prctl() it's own task and set a cookie on himself. This means you just made giant unpriv DoS vector.

stop_machine is bad, really bad.

#103 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 35 lines · Peter Zijlstra
Quoted 24 lines · Joel Fernandes
Quoted 23 lines · Peter Zijlstra
Quoted 20 lines · Joel Fernandes (Google)
kernel/sched/core.c +19 -0
+/*
+ * Ensure that the task has been requeued. The stopper ensures that the task cannot
+ * be migrated to a different CPU while its core scheduler queue state is being updated.
+ * It also makes sure to requeue a task if it was running actively on another CPU.
+ */
+static int sched_core_task_join_stopper(void *data)
+{
+	struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data;
+	int i;
+
+	for (i = 0; i < 2; i++)
+		sched_core_tag_requeue(tag->tasks[i], tag->cookies[i], false /* !group */);
+
+	return 0;
+}
+
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
+{
 
+	stop_machine(sched_core_task_join_stopper, (void *)&wr, NULL);
Quoted 9 lines
Quoted 3 lines
Quoted 1 line

+}

This is *REALLY* terrible...

I pulled this bit from your original patch. Are you concerned about the stop_machine? Sharing a core is a slow path for our usecases (and as far as I know, for everyone else's). We can probably do something different if that requirement changes.

Quoted 5 lines

Yeah.. so I can (and was planning on) remove stop_machine() from sched_core_{dis,en}able() before merging it.

(there's two options, one uses stop_cpus() with the SMT mask, the other RCU)

Ok. What about changing the cookie of task T while holding the rq/pi locks, and then doing a resched_curr(rq) for that RQ?

The holding lock ensures no migration of task happens, and resched_curr() ensure that task T's rq will enter the scheduler to consider the task T's new cookie for scheduling. I believe this is analogous to what __sched_setscheduler() does when you switch a task from CFS to RT.

Quoted 5 lines

This though is exposing stop_machine() to joe user. Everybody is allowed to prctl() it's own task and set a cookie on himself. This means you just made giant unpriv DoS vector.

stop_machine is bad, really bad.

Agreed.

thanks,

- Joel

maintainer
#104 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)
Quoted 34 lines · Joel Fernandes (Google)
kernel/sched/core.c +34 -0
+/* Per-task interface */
+static unsigned long sched_core_alloc_task_cookie(void)
+{
+	struct sched_core_cookie *ptr =
+		kmalloc(sizeof(struct sched_core_cookie), GFP_KERNEL);
+
+	if (!ptr)
+		return 0;
+	refcount_set(&ptr->refcnt, 1);
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return (unsigned long)ptr;
+}
+
+static bool sched_core_get_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return refcount_inc_not_zero(&ptr->refcnt);
+}
+
+static void sched_core_put_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9865

Quoted 10 lines
kernel/sched/core.c +9 -0
+	if (refcount_dec_and_test(&ptr->refcnt))
+		kfree(ptr);
+}
 
+	/*
+	 * NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or
+	 *       sched_core_put_task_cookie(). However, sched_core_put() is done
+	 *       by this function *after* the stopper removes the tasks from the
+	 *       core queue, and not before. This is just to play it safe.
+	 */

So for no reason what so ever you've made the code more difficult?

#105 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 39 lines · Peter Zijlstra
Quoted 37 lines · Joel Fernandes (Google)
(quoted diff) +37 -0
+/* Per-task interface */
+static unsigned long sched_core_alloc_task_cookie(void)
+{
+	struct sched_core_cookie *ptr =
+		kmalloc(sizeof(struct sched_core_cookie), GFP_KERNEL);
+
+	if (!ptr)
+		return 0;
+	refcount_set(&ptr->refcnt, 1);
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return (unsigned long)ptr;
+}
+
+static bool sched_core_get_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return refcount_inc_not_zero(&ptr->refcnt);
+}
+
+static void sched_core_put_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	if (refcount_dec_and_test(&ptr->refcnt))
+		kfree(ptr);
+}
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+	/*
+	 * NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or
+	 *       sched_core_put_task_cookie(). However, sched_core_put() is done
+	 *       by this function *after* the stopper removes the tasks from the
+	 *       core queue, and not before. This is just to play it safe.
+	 */

So for no reason what so ever you've made the code more difficult?

You're right, I could just do sched_core_get() in the caller. I changed it as in the diff below:

---8<-----------------------

kernel/sched/coretag.c +7 -13
index 800c0f8bacfc..75e2edb53a48 100644
@@ -274,6 +274,7 @@ void sched_core_change_group(struct task_struct *p, struct task_group *new_tg)
 /* Per-task interface: Used by fork(2) and prctl(2). */
 static void sched_core_put_cookie_work(struct work_struct *ws);
 
+/* Caller has to call sched_core_get() if non-zero value is returned. */
 static unsigned long sched_core_alloc_task_cookie(void)
 {
 	struct sched_core_task_cookie *ck =
@@ -284,11 +285,6 @@ static unsigned long sched_core_alloc_task_cookie(void)
 	refcount_set(&ck->refcnt, 1);
 	INIT_WORK(&ck->work, sched_core_put_cookie_work);
 
-	/*
-	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
-	 * is done after the stopper runs.
-	 */
-	sched_core_get();
 	return (unsigned long)ck;
 }
 
@@ -354,12 +350,6 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 
 	mutex_lock(&sched_core_tasks_mutex);
 
-	/*
-	 * NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or
-	 *       sched_core_put_task_cookie(). However, sched_core_put() is done
-	 *       by this function *after* the stopper removes the tasks from the
-	 *       core queue, and not before. This is just to play it safe.
-	 */
 	if (!t2) {
 		if (t1->core_task_cookie) {
 			sched_core_put_task_cookie(t1->core_task_cookie);
@@ -370,7 +360,9 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 		/* Assign a unique per-task cookie solely for t1. */
 
 		cookie = sched_core_alloc_task_cookie();
-		if (!cookie)
+		if (cookie)
+			sched_core_get();
+		else
 			goto out_unlock;
 
 		if (t1->core_task_cookie) {
@@ -401,7 +393,9 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 
 		/* CASE 1. */
 		cookie = sched_core_alloc_task_cookie();
-		if (!cookie)
+		if (cookie)
+			sched_core_get();
+		else
 			goto out_unlock;
 
 		/* Add another reference for the other task. */
maintainer
#106 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9819

Quoted 10 lines · Joel Fernandes (Google)
kernel/sched/core.c +10 -0
+static bool sched_core_get_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return refcount_inc_not_zero(&ptr->refcnt);

See below, but afaict this should be refcount_inc().

Quoted 18 lines
kernel/sched/core.c +17 -0
+}
 
+	/*
+	 * 		t1		joining		t2
+	 * CASE 1:
+	 * before	0				0
+	 * after	new cookie			new cookie
+	 *
+	 * CASE 2:
+	 * before	X (non-zero)			0
+	 * after	0				0
+	 *
+	 * CASE 3:
+	 * before	0				X (non-zero)
+	 * after	X				X
+	 *
+	 * CASE 4:
+	 * before	Y (non-zero)			X (non-zero)

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9911

Quoted 10 lines
kernel/sched/core.c +10 -0
+	 * after	X				X
+	 */
+	if (!t1->core_task_cookie && !t2->core_task_cookie) {
+		/* CASE 1. */
+		cookie = sched_core_alloc_task_cookie();
+		if (!cookie)
+			goto out_unlock;
+
+		/* Add another reference for the other task. */
+		if (!sched_core_get_task_cookie(cookie)) {

afaict this should be refcount_inc(), as this can never fail and if it does, it's an error.

Quoted 8 lines
kernel/sched/core.c +8 -0
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		wr.tasks[0] = t1;
+		wr.tasks[1] = t2;
+		wr.cookies[0] = wr.cookies[1] = cookie;
+

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9929

Quoted 10 lines
kernel/sched/core.c +10 -0
+	} else if (t1->core_task_cookie && !t2->core_task_cookie) {
+		/* CASE 2. */
+		sched_core_put_task_cookie(t1->core_task_cookie);
+		sched_core_put_after_stopper = true;
+
+		wr.tasks[0] = t1; /* Reset cookie for t1. */
+
+	} else if (!t1->core_task_cookie && t2->core_task_cookie) {
+		/* CASE 3. */
+		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {

afaict this can never fail either, because you're calling in here with a reference on t2

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9939

Quoted 10 lines
kernel/sched/core.c +10 -0
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		wr.tasks[0] = t1;
+		wr.cookies[0] = t2->core_task_cookie;
+
+	} else {
+		/* CASE 4. */
+		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {

Same.

Quoted 9 lines
kernel/sched/core.c +9 -0
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+		sched_core_put_task_cookie(t1->core_task_cookie);
+		sched_core_put_after_stopper = true;
+
+		wr.tasks[0] = t1;
+		wr.cookies[0] = t2->core_task_cookie;
+	}
#107 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

Quoted 13 lines · Peter Zijlstra

Review

Quoted 10 lines · Joel Fernandes (Google)
(quoted diff) +10 -0
+static bool sched_core_get_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return refcount_inc_not_zero(&ptr->refcnt);

See below, but afaict this should be refcount_inc().

Fully agreed with all these. Updated with diff as below. Will test further and post next version soon. Thanks!

---8<-----------------------

kernel/sched/coretag.c +5 -14
index 2fb5544a4a18..8fce3f4b7cae 100644
@@ -288,12 +288,12 @@ static unsigned long sched_core_alloc_task_cookie(void)
 	return (unsigned long)ck;
 }
 
-static bool sched_core_get_task_cookie(unsigned long cookie)
+static void sched_core_get_task_cookie(unsigned long cookie)
 {
 	struct sched_core_task_cookie *ptr =
 		(struct sched_core_task_cookie *)cookie;
 
-	return refcount_inc_not_zero(&ptr->refcnt);
+	refcount_inc(&ptr->refcnt);
 }
 
 static void sched_core_put_task_cookie(unsigned long cookie)
@@ -392,10 +392,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 		sched_core_get(); /* For the alloc. */
 
 		/* Add another reference for the other task. */
-		if (!sched_core_get_task_cookie(cookie)) {
-			ret = -EINVAL;
-			goto out_unlock;
-		}
+		sched_core_get_task_cookie(cookie);
 		sched_core_get(); /* For the other task. */
 
 		wr.tasks[0] = t1;
@@ -411,10 +408,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 
 	} else if (!t1->core_task_cookie && t2->core_task_cookie) {
 		/* CASE 3. */
-		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {
-			ret = -EINVAL;
-			goto out_unlock;
-		}
+		sched_core_get_task_cookie(t2->core_task_cookie);
 		sched_core_get();
 
 		wr.tasks[0] = t1;
@@ -422,10 +416,7 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 
 	} else {
 		/* CASE 4. */
-		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {
-			ret = -EINVAL;
-			goto out_unlock;
-		}
+		sched_core_get_task_cookie(t2->core_task_cookie);
 		sched_core_get();
 
 		sched_core_put_task_cookie(t1->core_task_cookie);
Signature · 1 line
2.29.2.454.gaff20da3a2-goog
#108 lore raw

Re: [PATCH -tip 22/32] sched: Split the cookie and setup per-task cookie on fork

In reply to Joel Fernandes (Google)
Quoted 83 lines · Joel Fernandes (Google)

In order to prevent interference and clearly support both per-task and CGroup APIs, split the cookie into 2 and allow it to be set from either per-task, or CGroup API. The final cookie is the combined value of both and is computed when the stop-machine executes during a change of cookie.

Also, for the per-task cookie, it will get weird if we use pointers of any emphemeral objects. For this reason, introduce a refcounted object who's sole purpose is to assign unique cookie value by way of the object's pointer.

While at it, refactor the CGroup code a bit. Future patches will introduce more APIs and support.

---
 include/linux/sched.h |   2 +
 kernel/sched/core.c   | 241 ++++++++++++++++++++++++++++++++++++++++--
 kernel/sched/debug.c  |   4 +
 3 files changed, 236 insertions(+), 11 deletions(-)
include/linux/sched.h +2 -0
index a60868165590..c6a3b0fa952b 100644
@@ -688,6 +688,8 @@ struct task_struct {
 #ifdef CONFIG_SCHED_CORE
 	struct rb_node			core_node;
 	unsigned long			core_cookie;
+	unsigned long			core_task_cookie;
+	unsigned long			core_group_cookie;
 	unsigned int			core_occupation;
 #endif
 
kernel/sched/core.c +22 -0
index b99a7493d590..7ccca355623a 100644
@@ -346,11 +346,14 @@ void sched_core_put(void)
 	mutex_unlock(&sched_core_mutex);
 }
 
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2);
+
 #else /* !CONFIG_SCHED_CORE */
 
 static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
 static inline void sched_core_dequeue(struct rq *rq, struct task_struct *p) { }
 static bool sched_core_enqueued(struct task_struct *task) { return false; }
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) { }
 
 #endif /* CONFIG_SCHED_CORE */
 
@@ -4032,6 +4035,20 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
 #endif
 #ifdef CONFIG_SCHED_CORE
 	RB_CLEAR_NODE(&p->core_node);
+
+	/*
+	 * Tag child via per-task cookie only if parent is tagged via per-task
+	 * cookie. This is independent of, but can be additive to the CGroup tagging.
+	 */
+	if (current->core_task_cookie) {
+
+		/* If it is not CLONE_THREAD fork, assign a unique per-task tag. */
+		if (!(clone_flags & CLONE_THREAD)) {
+			return sched_core_share_tasks(p, p);
+               }
+		/* Otherwise share the parent's per-task tag. */
+		return sched_core_share_tasks(p, current);
+	}
 #endif
 	return 0;
 }
@@ -9731,6 +9748,217 @@ static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css,
 #endif /* CONFIG_RT_GROUP_SCHED */
 
 #ifdef CONFIG_SCHED_CORE
+/*
+ * A simple wrapper around refcount. An allocated sched_core_cookie's
+ * address is used to compute the cookie of the task.
+ */
+struct sched_core_cookie {

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9765

Quoted 10 lines
kernel/sched/core.c +10 -0
+	refcount_t refcnt;
+};
+
+/*
+ * sched_core_tag_requeue - Common helper for all interfaces to set a cookie.
+ * @p: The task to assign a cookie to.
+ * @cookie: The cookie to assign.
+ * @group: is it a group interface or a per-task interface.
+ *
+ * This function is typically called from a stop-machine handler.

Can you clarify if it is typically or always, what are the implications for locking?

Quoted 5 lines
kernel/sched/core.c +5 -0
+ */
+void sched_core_tag_requeue(struct task_struct *p, unsigned long cookie, bool group)
+{
+	if (!p)
+		return;

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9780

Quoted 10 lines
kernel/sched/core.c +10 -0
+
+	if (group)
+		p->core_group_cookie = cookie;
+	else
+		p->core_task_cookie = cookie;
+
+	/* Use up half of the cookie's bits for task cookie and remaining for group cookie. */
+	p->core_cookie = (p->core_task_cookie <<
+				(sizeof(unsigned long) * 4)) + p->core_group_cookie;
+

Always use masks to ensure this fits in the space we have, should we be concerned about overflows and the potential for collision of cookie values?

Quoted 43 lines
kernel/sched/core.c +43 -0
+	if (sched_core_enqueued(p)) {
+		sched_core_dequeue(task_rq(p), p);
+		if (!p->core_task_cookie)
+			return;
+	}
+
+	if (sched_core_enabled(task_rq(p)) &&
+			p->core_cookie && task_on_rq_queued(p))
+		sched_core_enqueue(task_rq(p), p);
+}
+
+/* Per-task interface */
+static unsigned long sched_core_alloc_task_cookie(void)
+{
+	struct sched_core_cookie *ptr =
+		kmalloc(sizeof(struct sched_core_cookie), GFP_KERNEL);
+
+	if (!ptr)
+		return 0;
+	refcount_set(&ptr->refcnt, 1);
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return (unsigned long)ptr;
+}
+
+static bool sched_core_get_task_cookie(unsigned long cookie)
+{
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	/*
+	 * NOTE: sched_core_put() is not done by put_task_cookie(). Instead, it
+	 * is done after the stopper runs.
+	 */
+	sched_core_get();
+	return refcount_inc_not_zero(&ptr->refcnt);
+}
+
+static void sched_core_put_task_cookie(unsigned long cookie)
+{

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9833

Quoted 10 lines
kernel/sched/core.c +10 -0
+	struct sched_core_cookie *ptr = (struct sched_core_cookie *)cookie;
+
+	if (refcount_dec_and_test(&ptr->refcnt))
+		kfree(ptr);
+}
+
+struct sched_core_task_write_tag {
+	struct task_struct *tasks[2];
+	unsigned long cookies[2];
+};

Use a better name instead of 2?

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9845

Quoted 12 lines
kernel/sched/core.c +12 -0
+
+/*
+ * Ensure that the task has been requeued. The stopper ensures that the task cannot
+ * be migrated to a different CPU while its core scheduler queue state is being updated.
+ * It also makes sure to requeue a task if it was running actively on another CPU.
+ */
+static int sched_core_task_join_stopper(void *data)
+{
+	struct sched_core_task_write_tag *tag = (struct sched_core_task_write_tag *)data;
+	int i;
+
+	for (i = 0; i < 2; i++)

Use ARRAY_SIZE(cookies) if you have to or ARRAY_SIZE(tasks)

kernel/sched/core.c static u64 cpu_rt_period_read_uint(struct cgroup_subsys_state *css, · line 9852

Quoted 7 lines
kernel/sched/core.c +7 -0
+		sched_core_tag_requeue(tag->tasks[i], tag->cookies[i], false /* !group */);
+
+	return 0;
+}
+
+static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
+{

Can you please explain how t1 and t2 are related, there is a table below, but I don't understand case#2, where the cookies get reset, is t2 the core leader and t2 leads what t1 and t2 collectively get?

May be just called t2 as parent?

Quoted 138 lines
kernel/sched/core.c +111 -11
+	struct sched_core_task_write_tag wr = {}; /* for stop machine. */
+	bool sched_core_put_after_stopper = false;
+	unsigned long cookie;
+	int ret = -ENOMEM;
+
+	mutex_lock(&sched_core_mutex);
+
+	/*
+	 * NOTE: sched_core_get() is done by sched_core_alloc_task_cookie() or
+	 *       sched_core_put_task_cookie(). However, sched_core_put() is done
+	 *       by this function *after* the stopper removes the tasks from the
+	 *       core queue, and not before. This is just to play it safe.
+	 */
+	if (t2 == NULL) {
+		if (t1->core_task_cookie) {
+			sched_core_put_task_cookie(t1->core_task_cookie);
+			sched_core_put_after_stopper = true;
+			wr.tasks[0] = t1; /* Keep wr.cookies[0] reset for t1. */
+		}
+	} else if (t1 == t2) {
+		/* Assign a unique per-task cookie solely for t1. */
+
+		cookie = sched_core_alloc_task_cookie();
+		if (!cookie)
+			goto out_unlock;
+
+		if (t1->core_task_cookie) {
+			sched_core_put_task_cookie(t1->core_task_cookie);
+			sched_core_put_after_stopper = true;
+		}
+		wr.tasks[0] = t1;
+		wr.cookies[0] = cookie;
+	} else
+	/*
+	 * 		t1		joining		t2
+	 * CASE 1:
+	 * before	0				0
+	 * after	new cookie			new cookie
+	 *
+	 * CASE 2:
+	 * before	X (non-zero)			0
+	 * after	0				0
+	 *
+	 * CASE 3:
+	 * before	0				X (non-zero)
+	 * after	X				X
+	 *
+	 * CASE 4:
+	 * before	Y (non-zero)			X (non-zero)
+	 * after	X				X
+	 */
+	if (!t1->core_task_cookie && !t2->core_task_cookie) {
+		/* CASE 1. */
+		cookie = sched_core_alloc_task_cookie();
+		if (!cookie)
+			goto out_unlock;
+
+		/* Add another reference for the other task. */
+		if (!sched_core_get_task_cookie(cookie)) {
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		wr.tasks[0] = t1;
+		wr.tasks[1] = t2;
+		wr.cookies[0] = wr.cookies[1] = cookie;
+
+	} else if (t1->core_task_cookie && !t2->core_task_cookie) {
+		/* CASE 2. */
+		sched_core_put_task_cookie(t1->core_task_cookie);
+		sched_core_put_after_stopper = true;
+
+		wr.tasks[0] = t1; /* Reset cookie for t1. */
+
+	} else if (!t1->core_task_cookie && t2->core_task_cookie) {
+		/* CASE 3. */
+		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+
+		wr.tasks[0] = t1;
+		wr.cookies[0] = t2->core_task_cookie;
+
+	} else {
+		/* CASE 4. */
+		if (!sched_core_get_task_cookie(t2->core_task_cookie)) {
+			ret = -EINVAL;
+			goto out_unlock;
+		}
+		sched_core_put_task_cookie(t1->core_task_cookie);
+		sched_core_put_after_stopper = true;
+
+		wr.tasks[0] = t1;
+		wr.cookies[0] = t2->core_task_cookie;
+	}
+
+	stop_machine(sched_core_task_join_stopper, (void *)&wr, NULL);
+
+	if (sched_core_put_after_stopper)
+		sched_core_put();
+
+	ret = 0;
+out_unlock:
+	mutex_unlock(&sched_core_mutex);
+	return ret;
+}
+
+/* CGroup interface */
 static u64 cpu_core_tag_read_u64(struct cgroup_subsys_state *css, struct cftype *cft)
 {
 	struct task_group *tg = css_tg(css);
@@ -9761,18 +9989,9 @@ static int __sched_write_tag(void *data)
 	 * when we set cgroup tag to 0 when the loop is done below.
 	 */
 	while ((p = css_task_iter_next(&it))) {
-		p->core_cookie = !!val ? (unsigned long)tg : 0UL;
-
-		if (sched_core_enqueued(p)) {
-			sched_core_dequeue(task_rq(p), p);
-			if (!p->core_cookie)
-				continue;
-		}
-
-		if (sched_core_enabled(task_rq(p)) &&
-		    p->core_cookie && task_on_rq_queued(p))
-			sched_core_enqueue(task_rq(p), p);
+		unsigned long cookie = !!val ? (unsigned long)tg : 0UL;
 
+		sched_core_tag_requeue(p, cookie, true /* group */);
 	}
 	css_task_iter_end(&it);
 
kernel/sched/debug.c +0 -0
index 60a922d3f46f..8c452b8010ad 100644

kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033

Quoted 10 lines
kernel/sched/debug.c +4 -0
@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		__PS("clock-delta", t1-t0);
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	__PS("core_cookie", p->core_cookie);
+#endif
+
 	sched_show_numa(p, m);
 }
 

Balbir Singh.

#109 lore raw

PATCH 23/32 sched: Add a per-thread core scheduling interface

In reply to Joel Fernandes (Google)

Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.

ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.

---
 include/linux/sched.h            |  1 +
 include/uapi/linux/prctl.h       |  3 ++
 kernel/sched/core.c              | 51 +++++++++++++++++++++++++++++---
 kernel/sys.c                     |  3 ++
 tools/include/uapi/linux/prctl.h |  3 ++
 5 files changed, 57 insertions(+), 4 deletions(-)
include/linux/sched.h +1 -0
index c6a3b0fa952b..79d76c78cc8e 100644
@@ -2083,6 +2083,7 @@ void sched_core_unsafe_enter(void);
 void sched_core_unsafe_exit(void);
 bool sched_core_wait_till_safe(unsigned long ti_check);
 bool sched_core_kernel_protected(void);
+int sched_core_share_pid(pid_t pid);
 #else
 #define sched_core_unsafe_enter(ignore) do { } while (0)
 #define sched_core_unsafe_exit(ignore) do { } while (0)
include/uapi/linux/prctl.h +3 -0
index c334e6a02e5f..217b0482aea1 100644
@@ -248,4 +248,7 @@ struct prctl_mm_map {
 #define PR_SET_IO_FLUSHER		57
 #define PR_GET_IO_FLUSHER		58
 
+/* Request the scheduler to share a core */
+#define PR_SCHED_CORE_SHARE		59
+
 #endif /* _LINUX_PRCTL_H */
kernel/sched/core.c +47 -4

78 lines not on this page · raw message

kernel/sys.c +3 -0
index a730c03ee607..61a3c98e36de 100644
@@ -2530,6 +2530,9 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 		error = (current->flags & PR_IO_FLUSHER) == PR_IO_FLUSHER;
 		break;
+	case PR_SCHED_CORE_SHARE:
+		error = sched_core_share_pid(arg2);
+		break;
 	default:
 		error = -EINVAL;
 		break;
tools/include/uapi/linux/prctl.h +3 -0
index 7f0827705c9a..4c45b7dcd92d 100644
@@ -247,4 +247,7 @@ struct prctl_mm_map {
 #define PR_SET_IO_FLUSHER		57
 #define PR_GET_IO_FLUSHER		58
 
+/* Request the scheduler to share a core */
+#define PR_SCHED_CORE_SHARE		59
+
 #endif /* _LINUX_PRCTL_H */
Signature · 1 line
2.29.2.299.gdc1121823c-goog
maintainer
#110 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

kernel/sched/core.c static int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2 · line 9973

Quoted 11 lines · Joel Fernandes (Google)
kernel/sched/core.c +11 -0
+/* Called from prctl interface: PR_SCHED_CORE_SHARE */
+int sched_core_share_pid(pid_t pid)
+{
+	struct task_struct *task;
+	int err;
+
+	if (pid == 0) { /* Recent current task's cookie. */
+		/* Resetting a cookie requires privileges. */
+		if (current->core_task_cookie)
+			if (!capable(CAP_SYS_ADMIN))
+				return -EPERM;

Coding-Style fail.

Also, why?!? I realize it is true for your case, because hardware fail. But in general this just isn't true. This wants to be some configurable policy.

#111 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 15 lines · Peter Zijlstra

Review

Quoted 11 lines · Joel Fernandes (Google)
(quoted diff) +11 -0
+/* Called from prctl interface: PR_SCHED_CORE_SHARE */
+int sched_core_share_pid(pid_t pid)
+{
+	struct task_struct *task;
+	int err;
+
+	if (pid == 0) { /* Recent current task's cookie. */
+		/* Resetting a cookie requires privileges. */
+		if (current->core_task_cookie)
+			if (!capable(CAP_SYS_ADMIN))
+				return -EPERM;

Coding-Style fail.

Quoted 3 lines

Also, why?!? I realize it is true for your case, because hardware fail. But in general this just isn't true. This wants to be some configurable policy.

True. I think me and you discussed eons ago though that it needs to privileged. For our case, actually we use seccomp so we don't let untrusted task set a cookie anyway, let alone reset it. We do it before we enter the seccomp sandbox. So we don't really need this security check here.

Since you dislike this part of the patch, I am Ok with just dropping it as below:

---8<-----------------------

kernel/sched/coretag.c +1 -5
index 8fce3f4b7cae..9b587a1245f5 100644
@@ -443,11 +443,7 @@ int sched_core_share_pid(pid_t pid)
 	struct task_struct *task;
 	int err;
 
-	if (pid == 0) { /* Recent current task's cookie. */
-		/* Resetting a cookie requires privileges. */
-		if (current->core_task_cookie)
-			if (!capable(CAP_SYS_ADMIN))
-				return -EPERM;
+	if (!pid) { /* Reset current task's cookie. */
 		task = NULL;
 	} else {
 		rcu_read_lock();
#112 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

In reply to Joel Fernandes (Google)
Quoted 7 lines · Joel Fernandes (Google)

Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.

ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.

Inline is a patch for comment to extend this interface to make it more useful. This patch would still need to provide doc and selftests updates as well.

-chrish

---8<-----------------------

From ec3d6506fee89022d93789e1ba44d49c1b1b04dd Mon Sep 17 00:00:00 2001
From: chris hyser <[email protected]>
Date: Tue, 10 Nov 2020 15:35:59 -0500
Subject: [PATCH] sched: Provide a more extensive prctl interface for core
scheduling.

The current prctl interface is a "from" only interface allowing a task to join an existing core scheduling group by getting the "cookie" from a specified task/pid.

Additionally, sharing from a task without an existing core scheduling group (cookie == 0) creates a new cookie shared between the two.

"From" functionality works well for programs modified to use the prctl(), but many applications will not be modified simply for core scheduling. Using a wrapper program to share the core scheduling cookie from another task followed by an "exec" can work, but there is no means to assign a cookie for an unmodified running task.

Simply inverting the interface to a "to"-only interface, i.e. the calling task shares it's cookie with the specified task/pid also has limitations, there being no means to enable tasks to join an existing core scheduling group for instance.

The solution is to allow both, or more specifically provide a flags argument to allow various core scheduling commands, currently FROM, TO, and CLEAR.

The combination of FROM and TO allow a helper program to share the core scheduling cookie of one task/core scheduling group, with additional tasks.

if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_pid, 0, 0) < 0)

	handle_error("src_pid sched_core failed");
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, dest_pid, 0, 0) < 0)
	handle_error("dest_pid sched_core failed");
include/linux/sched.h +2 -2
index c9efdf8..eed002e 100644
@@ -2084,14 +2084,14 @@ void sched_core_unsafe_enter(void);
 void sched_core_unsafe_exit(void);
 bool sched_core_wait_till_safe(unsigned long ti_check);
 bool sched_core_kernel_protected(void);
-int sched_core_share_pid(pid_t pid);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
 void sched_tsk_free(struct task_struct *tsk);
 #else
 #define sched_core_unsafe_enter(ignore) do { } while (0)
 #define sched_core_unsafe_exit(ignore) do { } while (0)
 #define sched_core_wait_till_safe(ignore) do { } while (0)
 #define sched_core_kernel_protected(ignore) do { } while (0)
-#define sched_core_share_pid(pid) do { } while (0)
+#define sched_core_share_pid(flags, pid) do { } while (0)
 #define sched_tsk_free(tsk) do { } while (0)
 #endif
 
include/uapi/linux/prctl.h +3 -0
index 217b048..f8e4e96 100644
@@ -250,5 +250,8 @@ struct prctl_mm_map {
 
 /* Request the scheduler to share a core */
 #define PR_SCHED_CORE_SHARE		59
+# define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+# define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+# define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
 
 #endif /* _LINUX_PRCTL_H */
kernel/sched/coretag.c +33 -27

79 lines not on this page · raw message

kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 
 #ifdef CONFIG_SCHED_CORE
 	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
 #endif
 
 	sched_show_numa(p, m);
kernel/sched/sched.h +1 -1
index b3b89bd..eafb399 100644
@@ -1202,7 +1202,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);
 void sched_core_get(void);
 void sched_core_put(void);
 
-int sched_core_share_pid(pid_t pid);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
 int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2);
 
 #ifdef CONFIG_CGROUP_SCHED
kernel/sys.c +5 -1
index 61a3c98..da52a0d 100644
@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 		error = (current->flags & PR_IO_FLUSHER) == PR_IO_FLUSHER;
 		break;
+#ifdef CONFIG_SCHED_CORE
 	case PR_SCHED_CORE_SHARE:
-		error = sched_core_share_pid(arg2);
+		if (arg4 || arg5)
+			return -EINVAL;
+		error = sched_core_share_pid(arg2, arg3);
 		break;
+#endif
 	default:
 		error = -EINVAL;
 		break;
#113 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 13 lines · Chris Hyser
kernel/sched/coretag.c +13 -0
+	get_task_struct(task);
+
+	/*
+	 * Check if this process has the right to modify the specified
+	 * process. Use the regular "ptrace_may_access()" checks.
+	 */
+	if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {
+		rcu_read_unlock();
+		err = -EPERM;
+		goto out;
+	}
+	rcu_read_unlock();
+

kernel/sched/coretag.c int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) · line 496

Quoted 10 lines
kernel/sched/coretag.c +10 -0
+	if (flags == PR_SCHED_CORE_CLEAR) {
+		dest = task;
+		src = NULL;
+	} else if (flags == PR_SCHED_CORE_SHARE_TO) {
+		dest = task;
+		src = current;
+	} else if (flags == PR_SCHED_CORE_SHARE_FROM) {
+		dest = current;
+		src = task;
+	}

I should have put in an else clause to catch bad input.

Quoted 5 lines
kernel/sched/coretag.c +2 -0
+
+	err = sched_core_share_tasks(dest, src);
  out:
  	if (task)
  		put_task_struct(task);
#114 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

In reply to Chris Hyser
Quoted 15 lines · Chris Hyser
Quoted 7 lines · Joel Fernandes (Google)

Add a per-thread core scheduling interface which allows a thread to share a core with another thread, or have a core exclusively for itself.

ChromeOS uses core-scheduling to securely enable hyperthreading. This cuts down the keypress latency in Google docs from 150ms to 50ms while improving the camera streaming frame rate by ~3%.

Inline is a patch for comment to extend this interface to make it more useful. This patch would still need to provide doc and selftests updates as well.

-chrish

---8<-----------------------

[..]

Quoted 83 lines
include/uapi/linux/prctl.h +3 -0
index 217b048..f8e4e96 100644
@@ -250,5 +250,8 @@ struct prctl_mm_map {
 
 /* Request the scheduler to share a core */
 #define PR_SCHED_CORE_SHARE		59
+# define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+# define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+# define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
 
 #endif /* _LINUX_PRCTL_H */
kernel/sched/coretag.c +21 -27
index 800c0f8..14feac1 100644
@@ -9,6 +9,7 @@
  */
 
 #include "sched.h"
+#include "linux/prctl.h"
 
 /*
  * Wrapper representing a complete cookie. The address of the cookie is used as
@@ -456,40 +457,45 @@ int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2)
 }
 
 /* Called from prctl interface: PR_SCHED_CORE_SHARE */
-int sched_core_share_pid(pid_t pid)
+int sched_core_share_pid(unsigned long flags, pid_t pid)
 {
+	struct task_struct *dest;
+	struct task_struct *src;
 	struct task_struct *task;
 	int err;
 
-	if (pid == 0) { /* Recent current task's cookie. */
-		/* Resetting a cookie requires privileges. */
-		if (current->core_task_cookie)
-			if (!capable(CAP_SYS_ADMIN))
-				return -EPERM;
-		task = NULL;
-	} else {
-		rcu_read_lock();
-		task = pid ? find_task_by_vpid(pid) : current;
-		if (!task) {
-			rcu_read_unlock();
-			return -ESRCH;
-		}
-
-		get_task_struct(task);
-
-		/*
-		 * Check if this process has the right to modify the specified
-		 * process. Use the regular "ptrace_may_access()" checks.
-		 */
-		if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {
-			rcu_read_unlock();
-			err = -EPERM;
-			goto out;
-		}
+	rcu_read_lock();
+	task = find_task_by_vpid(pid);
+	if (!task) {
 		rcu_read_unlock();
+		return -ESRCH;
 	}
 
-	err = sched_core_share_tasks(current, task);
+	get_task_struct(task);
+
+	/*
+	 * Check if this process has the right to modify the specified
+	 * process. Use the regular "ptrace_may_access()" checks.
+	 */
+	if (!ptrace_may_access(task, PTRACE_MODE_READ_REALCREDS)) {
+		rcu_read_unlock();
+		err = -EPERM;
+		goto out;
+	}
+	rcu_read_unlock();
+

kernel/sched/coretag.c int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2) · line 496

Quoted 10 lines
kernel/sched/coretag.c +10 -0
+	if (flags == PR_SCHED_CORE_CLEAR) {
+		dest = task;
+		src = NULL;
+	} else if (flags == PR_SCHED_CORE_SHARE_TO) {
+		dest = task;
+		src = current;
+	} else if (flags == PR_SCHED_CORE_SHARE_FROM) {
+		dest = current;
+		src = task;
+	}

Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?

Also looking forward to the docs/test updates.

thanks!

- Joel

Quoted 49 lines
kernel/sched/coretag.c +2 -0
+
+	err = sched_core_share_tasks(dest, src);
 out:
 	if (task)
 		put_task_struct(task);
kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 
 #ifdef CONFIG_SCHED_CORE
 	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
 #endif
 
 	sched_show_numa(p, m);
kernel/sched/sched.h +1 -1
index b3b89bd..eafb399 100644
@@ -1202,7 +1202,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);
 void sched_core_get(void);
 void sched_core_put(void);
 
-int sched_core_share_pid(pid_t pid);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
 int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2);
 
 #ifdef CONFIG_CGROUP_SCHED
kernel/sys.c +5 -1
index 61a3c98..da52a0d 100644
@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 		error = (current->flags & PR_IO_FLUSHER) == PR_IO_FLUSHER;
 		break;
+#ifdef CONFIG_SCHED_CORE
 	case PR_SCHED_CORE_SHARE:
-		error = sched_core_share_pid(arg2);
+		if (arg4 || arg5)
+			return -EINVAL;
+		error = sched_core_share_pid(arg2, arg3);
 		break;
+#endif
 	default:
 		error = -EINVAL;
 		break;
#115 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 3 lines · Joel Fernandes

Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?

yes.

Quoted 1 line

Also looking forward to the docs/test updates.

on it.

Thanks.

-chrish

#116 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

In reply to Joel Fernandes
Quoted 1 line · Joel Fernandes

...

Quoted 8 lines

Looks ok to me except the missing else { } clause you found. Also, maybe dest/src can be renamed to from/to to make meaning of variables more clear?

Also looking forward to the docs/test updates.

thanks!

- Joel

I fixed the else clause, made the name changes, added the documentation and fixed the selftests so they pass. Patch inline.

-chrish

 .../admin-guide/hw-vuln/core-scheduling.rst        | 98 ++++++++++++++++------
 include/linux/sched.h                              |  4 +-
 include/uapi/linux/prctl.h                         |  3 +
 kernel/sched/coretag.c                             | 59 ++++++++-----
 kernel/sched/debug.c                               |  1 +
 kernel/sched/sched.h                               |  2 +-
 kernel/sys.c                                       |  6 +-
 tools/include/uapi/linux/prctl.h                   |  3 +
 tools/testing/selftests/sched/test_coresched.c     | 14 +++-
 9 files changed, 136 insertions(+), 54 deletions(-)

-- >8 --
From beca9bae6750a66d8c30bbed1d6b8b26b2da05f4 Mon Sep 17 00:00:00 2001
From: chris hyser <[email protected]>
Date: Tue, 10 Nov 2020 15:35:59 -0500
Subject: [PATCH] sched: Provide a more extensive prctl interface for core
scheduling.

The current prctl interface is a "from" only interface allowing a task to join an existing core scheduling group by getting the "cookie" from a specified task/pid.

Additionally, sharing from a task without an existing core scheduling group (cookie == 0) creates a new cookie shared between the two.

"From" functionality works well for programs modified to use the prctl(), but many applications will not be modified simply for core scheduling. Using a wrapper program to share the core scheduling cookie from another task followed by an "exec" can work, but there is no means to assign a cookie for an unmodified running task.

Simply inverting the interface to a "to"-only interface, i.e. the calling task shares it's cookie with the specified task/pid also has limitations, there being no means to enable tasks to join an existing core scheduling group for instance.

The solution is to allow both, or more specifically provide a flags argument to allow various core scheduling commands, currently FROM, TO, and CLEAR.

The combination of FROM and TO allow a helper program to share the core scheduling cookie of one task/core scheduling group, with additional tasks.

if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_pid, 0, 0) < 0)

	handle_error("src_pid sched_core failed");
if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, dest_pid, 0, 0) < 0)
	handle_error("dest_pid sched_core failed");
Documentation/admin-guide/hw-vuln/core-scheduling.rst +74 -24
index c211a7b..0eefa9b 100644
@@ -100,33 +100,83 @@ The color is an 8-bit value allowing for up to 256 unique colors.
 
 prctl interface
 ###############
-A ``prtcl(2)`` command ``PR_SCHED_CORE_SHARE`` is available to a process to request
-sharing a core with another process.  For example, consider 2 processes ``P1``
-and ``P2`` with PIDs 100 and 200. If process ``P1`` calls
-``prctl(PR_SCHED_CORE_SHARE, 200)``, the kernel makes ``P1`` share a core with ``P2``.
-The kernel performs ptrace access mode checks before granting the request.
+A ``prtcl(2)`` command ``PR_SCHED_CORE_SHARE`` provides an interface for the
+creation of and admission and removal of tasks from core scheduling groups.
 
-.. note:: This operation is not commutative. P1 calling
-          ``prctl(PR_SCHED_CORE_SHARE, pidof(P2)`` is not the same as P2 calling the
-          same for P1. The former case is P1 joining P2's group of processes
-          (which P2 would have joined with ``prctl(2)`` prior to P1's ``prctl(2)``).
+::
 
-.. note:: The core-sharing granted with prctl(2) will be subject to
-          core-sharing restrictions specified by the CGroup interface. For example
-          if P1 and P2 are a part of 2 different tagged CGroups, then they will
-          not share a core even if a prctl(2) call is made. This is analogous
-          to how affinities are set using the cpuset interface.
+    #include <sys/prctl.h>
 
-It is important to note that, on a ``CLONE_THREAD`` ``clone(2)`` syscall, the child
-will be assigned the same tag as its parent and thus be allowed to share a core
-with them. This design choice is because, for the security usecase, a
-``CLONE_THREAD`` child can access its parent's address space anyway, so there's
-no point in not allowing them to share a core. If a different behavior is
-desired, the child thread can call ``prctl(2)`` as needed.  This behavior is
-specific to the ``prctl(2)`` interface. For the CGroup interface, the child of a
-fork always shares a core with its parent.  On the other hand, if a parent
-was previously tagged via ``prctl(2)`` and does a regular ``fork(2)`` syscall, the
-child will receive a unique tag.
+    int prctl(int option, unsigned long arg2, unsigned long arg3,
+            unsigned long arg4, unsigned long arg5);
+
+option:
+    ``PR_SCHED_CORE_SHARE``
+
+arg2:
+    - ``PR_SCHED_CORE_CLEAR            0  -- clear core_sched cookie of pid``
+    - ``PR_SCHED_CORE_SHARE_FROM       1  -- get core_sched cookie from pid``
+    - ``PR_SCHED_CORE_SHARE_TO         2  -- push core_sched cookie to pid``
+
+arg3:
+    ``tid`` of the task for which the operation applies
+
+arg4 and arg5:
+    MUST be equal to 0.
+
+Creation
+~~~~~~~~
+Creation is accomplished by sharing a ''cookie'' from a process not currently in
+a core scheduling group.
+
+::
+
+    if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, src_tid, 0, 0) < 0)
+            handle_error("src_tid sched_core failed");
+
+Removal
+~~~~~~~
+Removing a task from a core scheduling group is done by:
+
+::
+
+    if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_CLEAR, clr_tid, 0, 0) < 0)
+             handle_error("clr_tid sched_core failed");
+
+Cookie Transferal
+~~~~~~~~~~~~~~~~~
+Transferring a cookie between the current and other tasks is possible using
+PR_SCHED_CORE_SHARE_FROM and PR_SCHED_CORE_SHARE_TO to inherit a cookie from a
+specified task or a share a cookie with a task. In combination this allows a
+simple helper program to pull a cookie from a task in an existing core
+scheduling group and share it with already running tasks.
+
+::
+
+    if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, from_tid, 0, 0) < 0)
+            handle_error("from_tid sched_core failed");
+
+    if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_TO, to_tid, 0, 0) < 0)
+            handle_error("to_tid sched_core failed");
+
+
+.. note:: The core-sharing granted with ``prctl(2)`` will be subject to
+          core-sharing restrictions specified by the CGroup interface. For example,
+          if tasks T1 and T2 are a part of 2 different tagged CGroups, then they will
+          not share a core even if ``prctl(2)`` is called. This is analogous to how
+          affinities are set using the cpuset interface.
+
+It is important to note that, on a ``clone(2)`` syscall with ``CLONE_THREAD`` set,
+the child will be assigned the same ''cookie'' as its parent and thus in the
+same core scheduling group.  In the security usecase, a ``CLONE_THREAD`` child
+can access its parent's address space anyway (``CLONE_THREAD`` requires
+``CLONE_SIGHAND`` which requires ``CLONE_VM``), so there's no point in not
+allowing them to share a core. If a different behavior is desired, the child
+thread can call ``prctl(2)`` as needed.  This behavior is specific to the
+``prctl(2)`` interface. For the CGroup interface, the child of a fork always
+shares a core with its parent.  On the other hand, if a parent was previously
+tagged via ``prctl(2)`` and does a regular ``fork(2)`` syscall, the child will
+receive a unique tag.
 
 Design/Implementation
 ---------------------
include/linux/sched.h +2 -2
index c9efdf8..eed002e 100644
@@ -2084,14 +2084,14 @@ void sched_core_unsafe_enter(void);
 void sched_core_unsafe_exit(void);
 bool sched_core_wait_till_safe(unsigned long ti_check);
 bool sched_core_kernel_protected(void);
-int sched_core_share_pid(pid_t pid);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
 void sched_tsk_free(struct task_struct *tsk);
 #else
 #define sched_core_unsafe_enter(ignore) do { } while (0)
 #define sched_core_unsafe_exit(ignore) do { } while (0)
 #define sched_core_wait_till_safe(ignore) do { } while (0)
 #define sched_core_kernel_protected(ignore) do { } while (0)
-#define sched_core_share_pid(pid) do { } while (0)
+#define sched_core_share_pid(flags, pid) do { } while (0)
 #define sched_tsk_free(tsk) do { } while (0)
 #endif
 
include/uapi/linux/prctl.h +3 -0
index 217b048..f8e4e96 100644
@@ -250,5 +250,8 @@ struct prctl_mm_map {
 
 /* Request the scheduler to share a core */
 #define PR_SCHED_CORE_SHARE		59
+# define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+# define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+# define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
 
 #endif /* _LINUX_PRCTL_H */
kernel/sched/coretag.c +36 -23

78 lines not on this page · raw message

kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 
 #ifdef CONFIG_SCHED_CORE
 	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
 #endif
 
 	sched_show_numa(p, m);
kernel/sched/sched.h +1 -1
index 53f0f58..3e0b9df 100644
@@ -1229,7 +1229,7 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p);
 void sched_core_get(void);
 void sched_core_put(void);
 
-int sched_core_share_pid(pid_t pid);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
 int sched_core_share_tasks(struct task_struct *t1, struct task_struct *t2);
 
 #ifdef CONFIG_CGROUP_SCHED
kernel/sys.c +5 -1
index 61a3c98..da52a0d 100644
@@ -2530,9 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 		error = (current->flags & PR_IO_FLUSHER) == PR_IO_FLUSHER;
 		break;
+#ifdef CONFIG_SCHED_CORE
 	case PR_SCHED_CORE_SHARE:
-		error = sched_core_share_pid(arg2);
+		if (arg4 || arg5)
+			return -EINVAL;
+		error = sched_core_share_pid(arg2, arg3);
 		break;
+#endif
 	default:
 		error = -EINVAL;
 		break;
tools/include/uapi/linux/prctl.h +3 -0
index 4c45b7d..cd37bbf 100644
@@ -249,5 +249,8 @@ struct prctl_mm_map {
 
 /* Request the scheduler to share a core */
 #define PR_SCHED_CORE_SHARE		59
+# define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+# define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+# define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
 
 #endif /* _LINUX_PRCTL_H */
tools/testing/selftests/sched/test_coresched.c +11 -3
index 70ed275..6ce612e 100644
@@ -23,6 +23,9 @@
 
 #ifndef PR_SCHED_CORE_SHARE
 #define PR_SCHED_CORE_SHARE 59
+# define PR_SCHED_CORE_CLEAR            0
+# define PR_SCHED_CORE_SHARE_FROM       1
+# define PR_SCHED_CORE_SHARE_TO         2
 #endif
 
 #ifndef DEBUG_PRINT
@@ -391,9 +394,14 @@ struct task_state *add_task(char *p)
 
 	    pid = mem->pid_share;
 	    mem->pid_share = 0;
-	    if (pid == -1)
-		pid = 0;
-	    prctl(PR_SCHED_CORE_SHARE, pid);
+
+	    if (pid == -1) {
+		if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_CLEAR, getpid(), 0, 0))
+		    perror("prctl() PR_SCHED_CORE_CLEAR failed");
+	    } else {
+		if (prctl(PR_SCHED_CORE_SHARE, PR_SCHED_CORE_SHARE_FROM, pid, 0, 0))
+		    perror("prctl() PR_SCHED_CORE_SHARE_FROM failed");
+	    }
 	    pthread_mutex_unlock(&mem->m);
 	    pthread_cond_signal(&mem->cond_par);
 	}
#117 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1034

Quoted 10 lines
kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 
 #ifdef CONFIG_SCHED_CORE
 	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
 #endif

Hmm, so the final cookie of the task is always p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.

I will drop this hunk from your overall diff, but let me know if you disagree!

thanks,

- Joel

#118 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 15 lines · Joel Fernandes

kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1034

Quoted 10 lines
kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 
  #ifdef CONFIG_SCHED_CORE
  	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
  #endif

Hmm, so the final cookie of the task is always p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.

Quoted 2 lines

I will drop this hunk from your overall diff, but let me know if you disagree!

No problem. That was there primarily for debugging.

Thanks.

-chrish

#119 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 15 lines · chris hyser
Quoted 14 lines · Joel Fernandes

kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033

Quoted 9 lines
kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
  #ifdef CONFIG_SCHED_CORE
  	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
  #endif

Hmm, so the final cookie of the task is always p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.

Quoted 5 lines
Quoted 2 lines

I will drop this hunk from your overall diff, but let me know if you disagree!

No problem. That was there primarily for debugging.

Ok. I squashed Josh's changes into this patch and several of my fixups. So there'll be 3 patches:
1. CGroup + prctl (single patch as it is hell to split it)
2. Documentation
3. ksefltests

Below is the diff of #1. I still have to squash in the stop_machine removal and some more review changes. But other than that, please take a look and let me know anything that's odd. I will test further as well.

Also next series will only be interface as I want to see if I can get lucky enough to have Peter look at it before he leaves for PTO next week. For the other features, I will post different series as I prepare them. One series for interface, and another for kernel protection / migration changes.

---8<-----------------------

include/linux/sched.h +6 -0
index a60868165590..73baca11d743 100644
@@ -688,6 +688,8 @@ struct task_struct {
 #ifdef CONFIG_SCHED_CORE
 	struct rb_node			core_node;
 	unsigned long			core_cookie;
+	unsigned long			core_task_cookie;
+	unsigned long			core_group_cookie;
 	unsigned int			core_occupation;
 #endif
 
@@ -2081,11 +2083,15 @@ void sched_core_unsafe_enter(void);
 void sched_core_unsafe_exit(void);
 bool sched_core_wait_till_safe(unsigned long ti_check);
 bool sched_core_kernel_protected(void);
+int sched_core_share_pid(unsigned long flags, pid_t pid);
+void sched_tsk_free(struct task_struct *tsk);
 #else
 #define sched_core_unsafe_enter(ignore) do { } while (0)
 #define sched_core_unsafe_exit(ignore) do { } while (0)
 #define sched_core_wait_till_safe(ignore) do { } while (0)
 #define sched_core_kernel_protected(ignore) do { } while (0)
+#define sched_core_share_pid(flags, pid) do { } while (0)
+#define sched_tsk_free(tsk) do { } while (0)
 #endif
 
 #endif
include/uapi/linux/prctl.h +6 -0
index c334e6a02e5f..3752006842e1 100644
@@ -248,4 +248,10 @@ struct prctl_mm_map {
 #define PR_SET_IO_FLUSHER		57
 #define PR_GET_IO_FLUSHER		58
 
+/* Request the scheduler to share a core */
+#define PR_SCHED_CORE_SHARE		59
+#define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+#define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+#define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
+
 #endif /* _LINUX_PRCTL_H */
kernel/fork.c +1 -0
index 7199d359690c..5468c93829c5 100644
@@ -736,6 +736,7 @@ void __put_task_struct(struct task_struct *tsk)
 	exit_creds(tsk);
 	delayacct_tsk_free(tsk);
 	put_signal_struct(tsk->signal);
+	sched_tsk_free(tsk);
 
 	if (!profile_handoff_task(tsk))
 		free_task(tsk);
kernel/sched/Makefile +1 -0
index 5fc9c9b70862..c526c20adf9d 100644
@@ -36,3 +36,4 @@ obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o
 obj-$(CONFIG_MEMBARRIER) += membarrier.o
 obj-$(CONFIG_CPU_ISOLATION) += isolation.o
 obj-$(CONFIG_PSI) += psi.o
+obj-$(CONFIG_SCHED_CORE) += coretag.o
kernel/sched/core.c +120 -19

247 lines not on this page · raw message

kernel/sched/coretag.c new file +734 -0

734 lines not on this page · raw message

kernel/sched/debug.c +4 -0
index 60a922d3f46f..8c452b8010ad 100644
@@ -1024,6 +1024,10 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
 		__PS("clock-delta", t1-t0);
 	}
 
+#ifdef CONFIG_SCHED_CORE
+	__PS("core_cookie", p->core_cookie);
+#endif
+
 	sched_show_numa(p, m);
 }
 
kernel/sched/sched.h +57 -4

94 lines not on this page · raw message

kernel/sys.c +7 -0
index a730c03ee607..da52a0da24ef 100644
@@ -2530,6 +2530,13 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
 
 		error = (current->flags & PR_IO_FLUSHER) == PR_IO_FLUSHER;
 		break;
+#ifdef CONFIG_SCHED_CORE
+	case PR_SCHED_CORE_SHARE:
+		if (arg4 || arg5)
+			return -EINVAL;
+		error = sched_core_share_pid(arg2, arg3);
+		break;
+#endif
 	default:
 		error = -EINVAL;
 		break;
tools/include/uapi/linux/prctl.h +6 -0
index 7f0827705c9a..8fbf9d164ec4 100644
@@ -247,4 +247,10 @@ struct prctl_mm_map {
 #define PR_SET_IO_FLUSHER		57
 #define PR_GET_IO_FLUSHER		58
 
+/* Request the scheduler to share a core */
+#define PR_SCHED_CORE_SHARE		59
+#define PR_SCHED_CORE_CLEAR		0  /* clear core_sched cookie of pid */
+#define PR_SCHED_CORE_SHARE_FROM	1  /* get core_sched cookie from pid */
+#define PR_SCHED_CORE_SHARE_TO		2  /* push core_sched cookie to pid */
+
 #endif /* _LINUX_PRCTL_H */
Signature · 1 line
2.29.2.684.gfbc64c5ab5-goog
#120 lore raw

Re: [PATCH -tip 23/32] sched: Add a per-thread core scheduling interface

Quoted 28 lines · Joel Fernandes
Quoted 15 lines · chris hyser
Quoted 14 lines · Joel Fernandes

kernel/sched/debug.c void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, · line 1033

Quoted 9 lines
kernel/sched/debug.c +1 -0
index cffdfab..50c31f3 100644
@@ -1030,6 +1030,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns,
   #ifdef CONFIG_SCHED_CORE
   	__PS("core_cookie", p->core_cookie);
+	__PS("core_task_cookie", p->core_task_cookie);
   #endif

Hmm, so the final cookie of the task is always p->core_cookie. This is what the scheduler uses. All other fields are ingredients to derive the final cookie value.

Quoted 5 lines
Quoted 2 lines

I will drop this hunk from your overall diff, but let me know if you disagree!

No problem. That was there primarily for debugging.

Ok. I squashed Josh's changes into this patch and several of my fixups. So there'll be 3 patches:
1. CGroup + prctl (single patch as it is hell to split it)
2. Documentation
3. ksefltests

Quoted 3 lines

Below is the diff of #1. I still have to squash in the stop_machine removal and some more review changes. But other than that, please take a look and let me know anything that's odd. I will test further as well.

Will do. Looking at it now and just started a build.

-chrish