sched: Core Scheduling
Subsystems (5)
Peter Zijlstra
maintainercover sched: Core Scheduling
Hai,
This is an agressive fold of all the core-scheduling work so far. I've stripped a whole bunch of tags along the way (hopefully not too many, please yell if you feel I made a mistake), including tested-by. Please retest.
Changes since the last partial post is dropping all the cgroup stuff and PR_SCHED_CORE_CLEAR as well as that exec() behaviour in order to later resolve the cgroup issue.
Since we're really rather late for the coming merge window, my plan was to merge the lot right after the merge window.
Again, please test.
These patches should shortly be available in my queue.git.
--- b/kernel/sched/core_sched.c | 229 ++++++ b/tools/testing/selftests/sched/.gitignore | 1 b/tools/testing/selftests/sched/Makefile | 14 b/tools/testing/selftests/sched/config | 1 b/tools/testing/selftests/sched/cs_prctl_test.c | 338 +++++++++ include/linux/sched.h | 19 include/uapi/linux/prctl.h | 8 kernel/Kconfig.preempt | 6 kernel/fork.c | 4 kernel/sched/Makefile | 1 kernel/sched/core.c | 858 ++++++++++++++++++++++-- kernel/sched/cpuacct.c | 12 kernel/sched/deadline.c | 38 - kernel/sched/debug.c | 4 kernel/sched/fair.c | 276 +++++-- kernel/sched/idle.c | 13 kernel/sched/pelt.h | 2 kernel/sched/rt.c | 31 kernel/sched/sched.h | 393 ++++++++-- kernel/sched/stop_task.c | 14 kernel/sched/topology.c | 4 kernel/sys.c | 5 tools/include/uapi/linux/prctl.h | 8 23 files changed, 2057 insertions(+), 222 deletions(-)
Peter Zijlstra
maintainerPATCH 1/19 sched/fair: Add a few assertions
--- kernel/sched/fair.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/fair.c +9 -2
@@ -6246,6 +6246,11 @@ static int select_idle_sibling(struct ta task_util =uclamp_task_util (p); } +/* +* per-cpu select_idle_mask usage +*/ +lockdep_assert_irqs_disabled (); +if ((available_idle_cpu (target) ||sched_idle_cpu (target)) &&asym_fits_capacity (task_util, target))return target;
@@ -6711,8 +6716,6 @@ static int find_energy_efficient_cpu(str* certain conditions an idle sibling CPU if the domain has SD_WAKE_AFFINE set. * * Returns the target CPU number. -* -* preempt must be disabled. */ static int select_task_rq_fair (struct task_struct *p,int prev_cpu,int wake_flags)
@@ -6725,6 +6728,10 @@ select_task_rq_fair(struct task_struct */* SD_flags and WF_flags share the first nibble */ int sd_flag = wake_flags &0xF ; +/* +* required for stable ->cpus_allowed +*/ +lockdep_assert_held (&p->pi_lock);if (wake_flags &WF_TTWU ) {record_wakee (p);
[tip: sched/core] sched/fair: Add a few assertions
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9099a14708ce1dfecb6002605594a0daa319b555 Gitweb: https://git.kernel.org/tip/9099a14708ce1dfecb6002605594a0daa319b555 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 17 Nov 2020 18:19:35 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 12 May 2021 11:43:26 +02:00
sched/fair: Add a few assertions
--- kernel/sched/fair.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
kernel/sched/fair.c +9 -2
@@ -6288,6 +6288,11 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) task_util =uclamp_task_util (p); } +/* +* per-cpu select_idle_mask usage +*/ +lockdep_assert_irqs_disabled (); +if ((available_idle_cpu (target) ||sched_idle_cpu (target)) &&asym_fits_capacity (task_util, target))return target;
@@ -6781,8 +6786,6 @@ unlock:* certain conditions an idle sibling CPU if the domain has SD_WAKE_AFFINE set. * * Returns the target CPU number. -* -* preempt must be disabled. */ static int select_task_rq_fair (struct task_struct *p,int prev_cpu,int wake_flags)
@@ -6795,6 +6798,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)/* SD_flags and WF_flags share the first nibble */ int sd_flag = wake_flags &0xF ; +/* +* required for stable ->cpus_allowed +*/ +lockdep_assert_held (&p->pi_lock);if (wake_flags &WF_TTWU ) {record_wakee (p);
Re: [tip: sched/core] sched/fair: Add a few assertions
Quoted 11 lines · On 2021/5/12 18:28, tip-bot2 for Peter Zijlstra wrote:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 9099a14708ce1dfecb6002605594a0daa319b555 Gitweb: https://git.kernel.org/tip/9099a14708ce1dfecb6002605594a0daa319b555 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 17 Nov 2020 18:19:35 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 12 May 2021 11:43:26 +02:00sched/fair: Add a few assertions
Quoted 9 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Tested-by Don Hiatt <[email protected]>Tested-by Hongyu Ning <[email protected]>Tested-by Vincent Guittot <[email protected]>--- kernel/sched/fair.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
Add quick test results based on tip tree sched/core merge commit:
====TEST INFO====
- kernel under test:
-- tip tree sched/core merge: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=60208dac643e24cbc62317de4e486fdcbbf05215 -- coresched_v10 kernel source: https://github.com/digitalocean/linux-coresched/commits/coresched/v10-v5.10.y
- test machine setup:
CPU(s): 192
On-line CPU(s) list: 0-191
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
NUMA node(s): 4
- performance test workloads:
-- A. sysbench cpu (192 threads) + sysbench cpu (192 threads) -- B. sysbench cpu (192 threads) + sysbench mysql (192 threads) -- C. uperf netperf.xml (192 threads over TCP or UDP protocol separately) -- D. will-it-scale context_switch via pipe (192 threads)
- negative test:
-- A. continuously toggle coresched (enable/disable) thru prctl on task cookies of PGID, during full loading of uperf workload with coresched on -- B. continuously toggle smt (on/off) via /sys/devices/system/cpu/smt/control, during full loading of uperf workload with coresched on
====TEST RESULTS====
- performance change key info:
--workload B: coresched (cs_on), sysbench mysql performance drop around 20% vs coresched_v10
--workload C, coresched (cs_on), uperf performance increased almost double vs coresched_v10
--workload C, default (cs_off), uperf performance drop over 25% vs coresched_v10, same issue seen on v5.13-rc1 base (w/o coresched patchset) --workload D, coresched (cs_on), wis performance increased almost double vs coresched_v10
- negative test summary:
no platform hang or kernel panic observed for both test
- performance info of workloads, normalized based on coresched_v10 results -- performance workload A:
Note:
* no performance change compared to coresched_v10
+---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | | ** | coresched_tip_merge_base_v5.13-rc1 | coresched_tip_merge_base_v5.13-rc1 | ** | coresched_v10_base_v5.10.11 | coresched_v10_base_v5.10.11 | +=======================================+======+======================================+========================================+=======+===============================+=================================+ | workload | ** | sysbench cpu * 192 | sysbench cpu * 192 | ** | sysbench cpu * 192 | sysbench cpu * 192 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | prctl/cgroup | ** | prctl on workload cpu_0 | prctl on workload cpu_1 | ** | cg_sysbench_cpu_0 | cg_sysbench_cpu_1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) | ** | Tput_avg (events/s) | Tput_avg (events/s) | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | coresched normalized vs coresched_v10 | ** | 0.97 | 1.05 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | default normalized vs coresched_v10 | ** | 1.03 | 0.95 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | smtoff normalized vs coresched_v10 | ** | 0.96 | 1.04 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+
-- performance workload B:
Note:
* coresched (cs_on), sysbench mysql performance drop around 20% vs coresched_v10
+---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | | ** | coresched_tip_merge_base_v5.13-rc1 | coresched_tip_merge_base_v5.13-rc1 | ** | coresched_v10_base_v5.10.11 | coresched_v10_base_v5.10.11 | +=======================================+======+======================================+========================================+=======+===============================+=================================+ | workload | ** | sysbench cpu * 192 | sysbench mysql * 192 | ** | sysbench cpu * 192 | sysbench mysql * 192 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | prctl/cgroup | ** | prctl on workload cpu_0 | prctl on workload mysql_0 | ** | cg_sysbench_cpu_0 | cg_sysbench_mysql_0 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | record_item | ** | Tput_avg (events/s) | Tput_avg (events/s) | ** | Tput_avg (events/s) | Tput_avg (events/s) | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | coresched normalized vs coresched_v10 | ** | 1.02 | 0.81 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | default normalized vs coresched_v10 | ** | 1.01 | 0.94 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | smtoff normalized vs coresched_v10 | ** | 0.93 | 1.18 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+
-- performance workload C:
Note:
* coresched (cs_on), uperf performance increased almost double vs coresched_v10
* default (cs_off), uperf performance drop over 25% vs coresched_v10, same issue seen on v5.13-rc1 base (w/o coresched patchset)
+---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | | ** | coresched_tip_merge_base_v5.13-rc1 | coresched_tip_merge_base_v5.13-rc1 | ** | coresched_v10_base_v5.10.11 | coresched_v10_base_v5.10.11 | +=======================================+======+======================================+========================================+=======+===============================+=================================+ | workload | ** | uperf netperf TCP * 192 | uperf netperf UDP * 192 | ** | uperf netperf TCP * 192 | uperf netperf UDP * 192 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | prctl/cgroup | ** | prctl on workload uperf | prctl on workload uperf | ** | cg_uperf | cg_uperf | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | record_item | ** | Tput_avg (Gb/s) | Tput_avg (Gb/s) | ** | Tput_avg (Gb/s) | Tput_avg (Gb/s) | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | coresched normalized vs coresched_v10 | ** | 1.83 | 1.93 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | default normalized vs coresched_v10 | ** | 0.75 | 0.71 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+ | smtoff normalized vs coresched_v10 | ** | 1 | 1.06 | ** | 1 | 1 | +---------------------------------------+------+--------------------------------------+----------------------------------------+-------+-------------------------------+---------------------------------+
-- performance workload D:
Note:
* coresched (cs_on), wis performance increased almost double vs coresched_v10
* default (cs_off) and smtoff, wis performance is better vs coresched_v10
+---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | | ** | coresched_tip_merge_base_v5.13-rc1 | ** | coresched_v10_base_v5.10.11 | +=======================================+======+======================================+=======+===============================+ | workload | ** | will-it-scale * 192 | ** | will-it-scale * 192 | | | | (pipe based context_switch) | | (pipe based context_switch) | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | prctl/cgroup | ** | prctl on workload wis | ** | cg_wis | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | record_item | ** | threads_avg | ** | threads_avg | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | coresched normalized vs coresched_v10 | ** | 2.01 | ** | 1.00 | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | default normalized vs coresched_v10 | ** | 1.13 | ** | 1.00 | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+ | smtoff normalized vs coresched_v10 | ** | 1.29 | ** | 1.00 | +---------------------------------------+------+--------------------------------------+-------+-------------------------------+
-- notes on record_item:
* coresched normalized vs coresched_v10: smton, cs enabled, test result normalized by result of coresched_v10 under same config
* default normalized vs coresched_v10: smton, cs disabled, test result normalized by result of coresched_v10 under same config
* smtoff normalized vs coresched_v10: smtoff, test result normalized by result of coresched_v10 under same config
-- Hongyu Ning
Peter Zijlstra
maintainerPATCH 2/19 sched: Provide raw_spin_rq_*lock*() helpers
In reply to Peter ZijlstraIn prepration for playing games with rq->lock, add some rq_lock wrappers.
--- kernel/sched/core.c | 15 +++++++++++++++ kernel/sched/sched.h | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+)
kernel/sched/core.c +15 -0
@@ -184,6 +184,21 @@ int sysctl_sched_rt_runtime = 950000;* */ +void raw_spin_rq_lock_nested (struct rq *rq,int subclass) +{ +raw_spin_lock_nested (rq_lockp (rq), subclass); +} + +bool raw_spin_rq_trylock (struct rq *rq) +{ +return raw_spin_trylock (rq_lockp (rq)); +} + +void raw_spin_rq_unlock (struct rq *rq) +{ +raw_spin_unlock (rq_lockp (rq)); +} +/* * __task_rq_lock - lock the rq @p resides on. */
kernel/sched/sched.h +50 -0
56 lines not on this page · raw message
[tip: sched/core] sched: Provide raw_spin_rq_*lock*() helpers
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 39d371b7c0c299d489041884d005aacc4bba8c15 Gitweb: https://git.kernel.org/tip/39d371b7c0c299d489041884d005aacc4bba8c15 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 02 Mar 2021 12:13:13 +01:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 12 May 2021 11:43:26 +02:00
sched: Provide raw_spin_rq_*lock*() helpers
In prepration for playing games with rq->lock, add some rq_lock wrappers.
--- kernel/sched/core.c | 15 +++++++++++++- kernel/sched/sched.h | 50 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 65 insertions(+)
kernel/sched/core.c +15 -0
@@ -184,6 +184,21 @@ int sysctl_sched_rt_runtime = 950000;* */ +void raw_spin_rq_lock_nested (struct rq *rq,int subclass) +{ +raw_spin_lock_nested (rq_lockp (rq), subclass); +} + +bool raw_spin_rq_trylock (struct rq *rq) +{ +return raw_spin_trylock (rq_lockp (rq)); +} + +void raw_spin_rq_unlock (struct rq *rq) +{ +raw_spin_unlock (rq_lockp (rq)); +} +/* * __task_rq_lock - lock the rq @p resides on. */
kernel/sched/sched.h +50 -0
56 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH 3/19 sched: Wrap rq::lock access
In reply to Peter ZijlstraIn preparation of playing games with rq->lock, abstract the thing using an accessor.
--- kernel/sched/core.c | 70 ++++++++++++++++---------------- kernel/sched/cpuacct.c | 12 ++--- kernel/sched/deadline.c | 22 +++++----- kernel/sched/debug.c | 4 - kernel/sched/fair.c | 35 +++++++--------- kernel/sched/idle.c | 4 - kernel/sched/pelt.h | 2 kernel/sched/rt.c | 16 +++---- kernel/sched/sched.h | 103 +++++++++++++++++++++++------------------------- kernel/sched/topology.c | 4 - 10 files changed, 135 insertions(+), 137 deletions(-)
kernel/sched/core.c +35 -35
@@ -211,12 +211,12 @@ struct rq *__task_rq_lock(struct task_stfor (;;) { rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (likely (rq ==task_rq (p) && !task_on_rq_migrating (p))) {rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);while (unlikely (task_on_rq_migrating (p)))cpu_relax ();
@@ -235,7 +235,7 @@ struct rq *task_rq_lock(struct task_strufor (;;) {raw_spin_lock_irqsave (&p->pi_lock, rf->flags); rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);/* * move_queued_task() task_rq_lock() *
@@ -257,7 +257,7 @@ struct rq *task_rq_lock(struct task_strurq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags);while (unlikely (task_on_rq_migrating (p)))
@@ -327,7 +327,7 @@ void update_rq_clock(struct rq *rq) {s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (rq->clock_update_flags &RQCF_ACT_SKIP )return ;
@@ -626,7 +626,7 @@ void resched_curr(struct rq *rq)struct task_struct *curr = rq->curr;int cpu; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (test_tsk_need_resched (curr))return ;
@@ -650,10 +650,10 @@ void resched_cpu(int cpu)struct rq *rq =cpu_rq (cpu);unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (cpu_online (cpu) || cpu ==smp_processor_id ())resched_curr (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); }#ifdef CONFIG_SMP
@@ -1152,7 +1152,7 @@ static inline void uclamp_rq_inc_id(strustruct uclamp_se *uc_se = &p->uclamp[clamp_id];struct uclamp_bucket *bucket; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* Update task effective clamp */ p->uclamp[clamp_id] =uclamp_eff_get (p, clamp_id);
@@ -1192,7 +1192,7 @@ static inline void uclamp_rq_dec_id(struunsigned int bkt_clamp;unsigned int rq_clamp; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* * If sched_uclamp_used was enabled after task @p was enqueued,
@@ -1865,7 +1865,7 @@ static inline bool is_cpu_allowed(structstatic struct rq *move_queued_task (struct rq *rq,struct rq_flags *rf,struct task_struct *p,int new_cpu) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);deactivate_task (rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, new_cpu);
@@ -2039,7 +2039,7 @@ int push_cpu_stop(void *arg)struct task_struct *p = arg;raw_spin_lock_irq (&p->pi_lock); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (task_rq (p) != rq) goto out_unlock;
@@ -2069,7 +2069,7 @@ int push_cpu_stop(void *arg) out_unlock: rq->push_busy = false; -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irq (&p->pi_lock);put_task_struct (p);
@@ -2122,7 +2122,7 @@ __do_set_cpus_allowed(struct task_struct* Because __kthread_bind() calls this on blocked tasks without * holding rq->lock. */ -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);dequeue_task (rq, p,DEQUEUE_SAVE |DEQUEUE_NOCLOCK ); }if (running)
@@ -2463,7 +2463,7 @@ void set_task_cpu(struct task_struct *p,* task_rq_lock(). */ WARN_ON_ONCE (debug_locks && !(lockdep_is_held (&p->pi_lock) || -lockdep_is_held (&task_rq (p)->lock))); +lockdep_is_held (rq_lockp (task_rq (p)))));#endif /* * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
@@ -3005,7 +3005,7 @@ ttwu_do_activate(struct rq *rq, struct t {int en_flags =ENQUEUE_WAKEUP |ENQUEUE_NOCLOCK ; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (p->sched_contributes_to_load) rq->nr_uninterruptible--;
@@ -4016,7 +4016,7 @@ static void do_balance_callbacks(structvoid (*func)(struct rq *rq);struct callback_head *next; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);while (head) { func = (void (*)(struct rq *))head->func;
@@ -4039,7 +4039,7 @@ static inline struct callback_head *spli {struct callback_head *head = rq->balance_callback; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (head) rq->balance_callback =NULL ;
@@ -4056,9 +4056,9 @@ static inline void balance_callbacks(strunsigned long flags;if (unlikely (head)) { -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);do_balance_callbacks (rq, head); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); } }
@@ -4089,10 +4089,10 @@ prepare_lock_switch(struct rq *rq, struc* do an early lockdep release here: */ rq_unpin_lock (rq, rf); -spin_release (&rq->lock.dep_map, _THIS_IP_); +spin_release (&rq_lockp (rq)->dep_map, _THIS_IP_);#ifdef CONFIG_DEBUG_SPINLOCK /* this is a valid case when another task releases the spinlock */ - rq->lock.owner = next; +rq_lockp (rq)->owner = next;#endif }
@@ -4103,9 +4103,9 @@ static inline void finish_lock_switch(st* fix up the runqueue lock - which gets 'carried over' from * prev into current: */ -spin_acquire (&rq->lock.dep_map,0 ,0 , _THIS_IP_); +spin_acquire (&rq_lockp (rq)->dep_map,0 ,0 , _THIS_IP_);__balance_callbacks (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq); }/*
@@ -5165,7 +5165,7 @@ static void __sched notrace __schedule(brq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq); } }
@@ -5707,7 +5707,7 @@ void rt_mutex_setprio(struct task_structrq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);preempt_enable (); }
@@ -7456,7 +7456,7 @@ void init_idle(struct task_struct *idle,__sched_fork (0 , idle);raw_spin_lock_irqsave (&idle->pi_lock, flags); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
@@ -7494,7 +7494,7 @@ void init_idle(struct task_struct *idle,#ifdef CONFIG_SMP idle->on_cpu =1 ;#endif -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irqrestore (&idle->pi_lock, flags);/* Set the preempt count _outside_ the spinlocks! */
@@ -7660,7 +7660,7 @@ static void balance_push(struct rq *rq) {struct task_struct *push_task = rq->curr; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);SCHED_WARN_ON (rq->cpu !=smp_processor_id ());/*
@@ -7698,9 +7698,9 @@ static void balance_push(struct rq *rq)*/ if (!rq->nr_running && !rq_has_pinned_tasks (rq) &&rcuwait_active (&rq->hotplug_wait)) { -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);rcuwait_wake_up (&rq->hotplug_wait); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }return ; }
@@ -7710,7 +7710,7 @@ static void balance_push(struct rq *rq)* Temporarily drop rq->lock such that we can wake-up the stop task. * Both preemption and IRQs are still disabled. */ -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);stop_one_cpu_nowait (rq->cpu, __balance_push_cpu_stop, push_task,this_cpu_ptr (&push_work));/*
@@ -7718,7 +7718,7 @@ static void balance_push(struct rq *rq)* schedule(). The next pick is obviously going to be the stop task * which kthread_is_per_cpu() and will push this task away. */ -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }static void balance_push_set (int cpu,bool on)
@@ -8008,7 +8008,7 @@ static void dump_rq_tasks(struct rq *rq,struct task_struct *g, *p;int cpu =cpu_of (rq); -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);printk ("%sCPU%d enqueued tasks (%u total):\n" , loglvl, cpu, rq->nr_running);for_each_process_thread (g, p) {
@@ -8181,7 +8181,7 @@ void __init sched_init(void)struct rq *rq; rq =cpu_rq (i); -raw_spin_lock_init (&rq->lock); +raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; rq->calc_load_update = jiffies +LOAD_FREQ ;
kernel/sched/cpuacct.c +6 -6
@@ -112,7 +112,7 @@ static u64 cpuacct_cpuusage_read(struct/* * Take rq->lock to make 64-bit read safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif if (index ==CPUACCT_STAT_NSTATS ) {
@@ -126,7 +126,7 @@ static u64 cpuacct_cpuusage_read(struct }#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif return data;
@@ -141,14 +141,14 @@ static void cpuacct_cpuusage_write(struc/* * Take rq->lock to make 64-bit write safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) cpuusage->usages[i] = val;#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif }
@@ -253,13 +253,13 @@ static int cpuacct_all_seq_show(struct s* Take rq->lock to make 64-bit read safe on 32-bit * platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif seq_printf (m," %llu" , cpuusage->usages[index]);#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif }seq_puts (m,"\n" );
kernel/sched/deadline.c +11 -11
@@ -157,7 +157,7 @@ void __add_running_bw(u64 dl_bw, struct {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->running_bw += dl_bw;SCHED_WARN_ON (dl_rq->running_bw < old);/* overflow */ SCHED_WARN_ON (dl_rq->running_bw > dl_rq->this_bw);
@@ -170,7 +170,7 @@ void __sub_running_bw(u64 dl_bw, struct {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->running_bw -= dl_bw;SCHED_WARN_ON (dl_rq->running_bw > old);/* underflow */ if (dl_rq->running_bw > old)
@@ -184,7 +184,7 @@ void __add_rq_bw(u64 dl_bw, struct dl_rq {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->this_bw += dl_bw;SCHED_WARN_ON (dl_rq->this_bw < old);/* overflow */ }
@@ -194,7 +194,7 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->this_bw -= dl_bw;SCHED_WARN_ON (dl_rq->this_bw > old);/* underflow */ if (dl_rq->this_bw > old)
@@ -987,7 +987,7 @@ static int start_dl_timer(struct task_stktime_t now, act;s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* * We want the timer to fire at the deadline, but considering
@@ -1097,9 +1097,9 @@ static enum hrtimer_restart dl_task_time* If the runqueue is no longer available, migrate the * task elsewhere. This necessarily changes rq. */ -lockdep_unpin_lock (&rq->lock, rf.cookie); +lockdep_unpin_lock (rq_lockp (rq), rf.cookie); rq =dl_task_offline_migration (rq, p); - rf.cookie =lockdep_pin_lock (&rq->lock); + rf.cookie =lockdep_pin_lock (rq_lockp (rq));update_rq_clock (rq);/*
@@ -1731,7 +1731,7 @@ static void migrate_task_rq_dl(struct ta* from try_to_wake_up(). Hence, p->pi_lock is locked, but * rq->lock is not... So, lock it */ -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (p->dl.dl_non_contending) {sub_running_bw (&p->dl, &rq->dl); p->dl.dl_non_contending =0 ;
@@ -1746,7 +1746,7 @@ static void migrate_task_rq_dl(struct taput_task_struct (p); }sub_rq_bw (&p->dl, &rq->dl); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p)
@@ -2291,10 +2291,10 @@ static void pull_dl_task(struct rq *thisdouble_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq); } }
kernel/sched/debug.c +2 -2
@@ -576,7 +576,7 @@ void print_cfs_rq(struct seq_file *m, inSEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"exec_clock" ,SPLIT_NS (cfs_rq->exec_clock)); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (rb_first_cached (&cfs_rq->tasks_timeline)) MIN_vruntime = (__pick_first_entity (cfs_rq))->vruntime; last =__pick_last_entity (cfs_rq);
@@ -584,7 +584,7 @@ void print_cfs_rq(struct seq_file *m, in max_vruntime = last->vruntime; min_vruntime = cfs_rq->min_vruntime; rq0_min_vruntime =cpu_rq (0 )->cfs.min_vruntime; -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags);SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"MIN_vruntime" ,SPLIT_NS (MIN_vruntime)); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "min_vruntime",
kernel/sched/fair.c +17 -18
@@ -1107,7 +1107,7 @@ struct numa_group {static struct numa_group *deref_task_numa_group (struct task_struct *p) {return rcu_dereference_check (p->numa_group, p == current || - (lockdep_is_held (&task_rq (p)->lock) && !READ_ONCE (p->on_cpu))); + (lockdep_is_held (rq_lockp (task_rq (p))) && !READ_ONCE (p->on_cpu))); }static struct numa_group *deref_curr_numa_group (struct task_struct *p)
@@ -5328,7 +5328,7 @@ static void __maybe_unused update_runtim {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -5347,7 +5347,7 @@ static void __maybe_unused unthrottle_of {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -6874,7 +6874,7 @@ static void migrate_task_rq_fair(struct* In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old' * rq->lock and can modify state directly. */ -lockdep_assert_held (&task_rq (p)->lock); +lockdep_assert_rq_held (task_rq (p));detach_entity_cfs_rq (&p->se); }else {
@@ -7501,7 +7501,7 @@ static int task_hot(struct task_struct * {s64 delta; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);if (p->sched_class != &fair_sched_class)return 0 ;
@@ -7599,7 +7599,7 @@ int can_migrate_task(struct task_struct {int tsk_cache_hot; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);/* * We do not migrate tasks that are:
@@ -7688,7 +7688,7 @@ int can_migrate_task(struct task_struct*/ static void detach_task (struct task_struct *p,struct lb_env *env) { -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);deactivate_task (env->src_rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, env->dst_cpu);
@@ -7704,7 +7704,7 @@ static struct task_struct *detach_one_ta {struct task_struct *p; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);list_for_each_entry_reverse (p, &env->src_rq->cfs_tasks, se.group_node) {
@@ -7740,7 +7740,7 @@ static int detach_tasks(struct lb_env *estruct task_struct *p;int detached =0 ; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);/* * Source run queue has been emptied by another CPU, clear
@@ -7870,7 +7870,7 @@ static int detach_tasks(struct lb_env *e*/ static void attach_task (struct rq *rq,struct task_struct *p) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);BUG_ON (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );
@@ -9781,7 +9781,7 @@ static int load_balance(int this_cpu, stif (need_active_balance (&env)) {unsigned long flags; -raw_spin_lock_irqsave (&busiest->lock, flags); +raw_spin_rq_lock_irqsave (busiest, flags);/* * Don't kick the active_load_balance_cpu_stop,
@@ -9789,8 +9789,7 @@ static int load_balance(int this_cpu, st* moved to this_cpu: */ if (!cpumask_test_cpu (this_cpu, busiest->curr->cpus_ptr)) { -raw_spin_unlock_irqrestore (&busiest->lock, - flags); +raw_spin_rq_unlock_irqrestore (busiest, flags); goto out_one_pinned; }
@@ -9807,7 +9806,7 @@ static int load_balance(int this_cpu, st busiest->push_cpu = this_cpu; active_balance =1 ; } -raw_spin_unlock_irqrestore (&busiest->lock, flags); +raw_spin_rq_unlock_irqrestore (busiest, flags);if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest),
@@ -10624,7 +10623,7 @@ static int newidle_balance(struct rq *th goto out; } -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);update_blocked_averages (this_cpu);rcu_read_lock ();
@@ -10662,7 +10661,7 @@ static int newidle_balance(struct rq *th }rcu_read_unlock (); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq);if (curr_cost > this_rq->max_idle_balance_cost) this_rq->max_idle_balance_cost = curr_cost;
@@ -11143,9 +11142,9 @@ void unregister_fair_sched_group(struct rq =cpu_rq (cpu); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); } }
kernel/sched/idle.c +2 -2
@@ -455,10 +455,10 @@ struct task_struct *pick_next_task_idle(static void dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) { -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq);printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack (); -raw_spin_lock_irq (&rq->lock); +raw_spin_rq_lock_irq (rq); }/*
kernel/sched/pelt.h +1 -1
@@ -141,7 +141,7 @@ static inline void update_idle_rq_clock_static inline u64 rq_clock_pelt (struct rq *rq) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);assert_clock_updated (rq);return rq->clock_pelt - rq->lost_idle_time;
kernel/sched/rt.c +8 -8
@@ -888,7 +888,7 @@ static int do_sched_rt_period_timer(struif (skip)continue ; -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);update_rq_clock (rq);if (rt_rq->rt_time) {
@@ -926,7 +926,7 @@ static int do_sched_rt_period_timer(struif (enqueue)sched_rt_rq_enqueue (rt_rq); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }if (!throttled && (!rt_bandwidth_enabled () || rt_b->rt_runtime ==RUNTIME_INF ))
@@ -1894,10 +1894,10 @@ static int push_rt_task(struct rq *rq, b*/ push_task =get_push_task (rq);if (push_task) { -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);stop_one_cpu_nowait (rq->cpu, push_cpu_stop, push_task, &rq->push_work); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }return 0 ;
@@ -2122,10 +2122,10 @@ void rto_push_irq_work_func(struct irq_w* When it gets updated, a check is made if a push is possible. */ if (has_pushable_tasks (rq)) { -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);while (push_rt_task (rq, true)) ; -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }raw_spin_lock (&rd->rto_lock);
@@ -2243,10 +2243,10 @@ static void pull_rt_task(struct rq *thisdouble_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq); } }
kernel/sched/sched.h +51 -52
274 lines not on this page · raw message
kernel/sched/topology.c +2 -2
@@ -467,7 +467,7 @@ void rq_attach_root(struct rq *rq, strucstruct root_domain *old_rd =NULL ;unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (rq->rd) { old_rd = rq->rd;
@@ -493,7 +493,7 @@ void rq_attach_root(struct rq *rq, strucif (cpumask_test_cpu (rq->cpu, cpu_active_mask))set_rq_online (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags);if (old_rd)call_rcu (&old_rd->rcu, free_rootdomain);
[tip: sched/core] sched: Wrap rq::lock access
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5cb9eaa3d274f75539077a28cf01e3563195fa53 Gitweb: https://git.kernel.org/tip/5cb9eaa3d274f75539077a28cf01e3563195fa53 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 17 Nov 2020 18:19:31 -05:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 12 May 2021 11:43:26 +02:00
sched: Wrap rq::lock access
In preparation of playing games with rq->lock, abstract the thing using an accessor.
--- kernel/sched/core.c | 70 +++++++++++++------------- kernel/sched/cpuacct.c | 12 ++-- kernel/sched/deadline.c | 22 ++++---- kernel/sched/debug.c | 4 +- kernel/sched/fair.c | 35 ++++++------- kernel/sched/idle.c | 4 +- kernel/sched/pelt.h | 2 +- kernel/sched/rt.c | 16 +++--- kernel/sched/sched.h | 105 +++++++++++++++++++-------------------- kernel/sched/topology.c | 4 +- 10 files changed, 136 insertions(+), 138 deletions(-)
kernel/sched/core.c +35 -35
@@ -211,12 +211,12 @@ struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) { rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (likely (rq ==task_rq (p) && !task_on_rq_migrating (p))) {rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);while (unlikely (task_on_rq_migrating (p)))cpu_relax ();
@@ -235,7 +235,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)for (;;) {raw_spin_lock_irqsave (&p->pi_lock, rf->flags); rq =task_rq (p); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);/* * move_queued_task() task_rq_lock() *
@@ -257,7 +257,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)rq_pin_lock (rq, rf);return rq; } -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irqrestore (&p->pi_lock, rf->flags);while (unlikely (task_on_rq_migrating (p)))
@@ -327,7 +327,7 @@ void update_rq_clock(struct rq *rq) {s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (rq->clock_update_flags &RQCF_ACT_SKIP )return ;
@@ -625,7 +625,7 @@ void resched_curr(struct rq *rq)struct task_struct *curr = rq->curr;int cpu; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (test_tsk_need_resched (curr))return ;
@@ -649,10 +649,10 @@ void resched_cpu(int cpu)struct rq *rq =cpu_rq (cpu);unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (cpu_online (cpu) || cpu ==smp_processor_id ())resched_curr (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); }#ifdef CONFIG_SMP
@@ -1151,7 +1151,7 @@ static inline void uclamp_rq_inc_id(struct rq *rq, struct task_struct *p,struct uclamp_se *uc_se = &p->uclamp[clamp_id];struct uclamp_bucket *bucket; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* Update task effective clamp */ p->uclamp[clamp_id] =uclamp_eff_get (p, clamp_id);
@@ -1191,7 +1191,7 @@ static inline void uclamp_rq_dec_id(struct rq *rq, struct task_struct *p,unsigned int bkt_clamp;unsigned int rq_clamp; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* * If sched_uclamp_used was enabled after task @p was enqueued,
@@ -1864,7 +1864,7 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu)static struct rq *move_queued_task (struct rq *rq,struct rq_flags *rf,struct task_struct *p,int new_cpu) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);deactivate_task (rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, new_cpu);
@@ -2038,7 +2038,7 @@ int push_cpu_stop(void *arg)struct task_struct *p = arg;raw_spin_lock_irq (&p->pi_lock); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (task_rq (p) != rq) goto out_unlock;
@@ -2068,7 +2068,7 @@ int push_cpu_stop(void *arg) out_unlock: rq->push_busy = false; -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irq (&p->pi_lock);put_task_struct (p);
@@ -2121,7 +2121,7 @@ __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32* Because __kthread_bind() calls this on blocked tasks without * holding rq->lock. */ -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);dequeue_task (rq, p,DEQUEUE_SAVE |DEQUEUE_NOCLOCK ); }if (running)
@@ -2462,7 +2462,7 @@ void set_task_cpu(struct task_struct *p, unsigned int new_cpu)* task_rq_lock(). */ WARN_ON_ONCE (debug_locks && !(lockdep_is_held (&p->pi_lock) || -lockdep_is_held (&task_rq (p)->lock))); +lockdep_is_held (rq_lockp (task_rq (p)))));#endif /* * Clearly, migrating tasks to offline CPUs is a fairly daft thing.
@@ -3004,7 +3004,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags, {int en_flags =ENQUEUE_WAKEUP |ENQUEUE_NOCLOCK ; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (p->sched_contributes_to_load) rq->nr_uninterruptible--;
@@ -4015,7 +4015,7 @@ static void do_balance_callbacks(struct rq *rq, struct callback_head *head)void (*func)(struct rq *rq);struct callback_head *next; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);while (head) { func = (void (*)(struct rq *))head->func;
@@ -4038,7 +4038,7 @@ static inline struct callback_head *splice_balance_callbacks(struct rq *rq) {struct callback_head *head = rq->balance_callback; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);if (head) rq->balance_callback =NULL ;
@@ -4055,9 +4055,9 @@ static inline void balance_callbacks(struct rq *rq, struct callback_head *head)unsigned long flags;if (unlikely (head)) { -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);do_balance_callbacks (rq, head); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); } }
@@ -4088,10 +4088,10 @@ prepare_lock_switch(struct rq *rq, struct task_struct *next, struct rq_flags *rf* do an early lockdep release here: */ rq_unpin_lock (rq, rf); -spin_release (&rq->lock.dep_map, _THIS_IP_); +spin_release (&rq_lockp (rq)->dep_map, _THIS_IP_);#ifdef CONFIG_DEBUG_SPINLOCK /* this is a valid case when another task releases the spinlock */ - rq->lock.owner = next; +rq_lockp (rq)->owner = next;#endif }
@@ -4102,9 +4102,9 @@ static inline void finish_lock_switch(struct rq *rq)* fix up the runqueue lock - which gets 'carried over' from * prev into current: */ -spin_acquire (&rq->lock.dep_map,0 ,0 , _THIS_IP_); +spin_acquire (&rq_lockp (rq)->dep_map,0 ,0 , _THIS_IP_);__balance_callbacks (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq); }/*
@@ -5164,7 +5164,7 @@ static void __sched notrace __schedule(bool preempt)rq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq); } }
@@ -5706,7 +5706,7 @@ out_unlock:rq_unpin_lock (rq, &rf);__balance_callbacks (rq); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);preempt_enable (); }
@@ -7456,7 +7456,7 @@ void init_idle(struct task_struct *idle, int cpu)__sched_fork (0 , idle);raw_spin_lock_irqsave (&idle->pi_lock, flags); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); idle->state =TASK_RUNNING ; idle->se.exec_start =sched_clock ();
@@ -7494,7 +7494,7 @@ void init_idle(struct task_struct *idle, int cpu)#ifdef CONFIG_SMP idle->on_cpu =1 ;#endif -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);raw_spin_unlock_irqrestore (&idle->pi_lock, flags);/* Set the preempt count _outside_ the spinlocks! */
@@ -7660,7 +7660,7 @@ static void balance_push(struct rq *rq) {struct task_struct *push_task = rq->curr; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);SCHED_WARN_ON (rq->cpu !=smp_processor_id ());/*
@@ -7698,9 +7698,9 @@ static void balance_push(struct rq *rq)*/ if (!rq->nr_running && !rq_has_pinned_tasks (rq) &&rcuwait_active (&rq->hotplug_wait)) { -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);rcuwait_wake_up (&rq->hotplug_wait); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }return ; }
@@ -7710,7 +7710,7 @@ static void balance_push(struct rq *rq)* Temporarily drop rq->lock such that we can wake-up the stop task. * Both preemption and IRQs are still disabled. */ -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);stop_one_cpu_nowait (rq->cpu, __balance_push_cpu_stop, push_task,this_cpu_ptr (&push_work));/*
@@ -7718,7 +7718,7 @@ static void balance_push(struct rq *rq)* schedule(). The next pick is obviously going to be the stop task * which kthread_is_per_cpu() and will push this task away. */ -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }static void balance_push_set (int cpu,bool on)
@@ -8008,7 +8008,7 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)struct task_struct *g, *p;int cpu =cpu_of (rq); -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);printk ("%sCPU%d enqueued tasks (%u total):\n" , loglvl, cpu, rq->nr_running);for_each_process_thread (g, p) {
@@ -8181,7 +8181,7 @@ void __init sched_init(void)struct rq *rq; rq =cpu_rq (i); -raw_spin_lock_init (&rq->lock); +raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; rq->calc_load_update = jiffies +LOAD_FREQ ;
kernel/sched/cpuacct.c +6 -6
@@ -112,7 +112,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu,/* * Take rq->lock to make 64-bit read safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif if (index ==CPUACCT_STAT_NSTATS ) {
@@ -126,7 +126,7 @@ static u64 cpuacct_cpuusage_read(struct cpuacct *ca, int cpu, }#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif return data;
@@ -141,14 +141,14 @@ static void cpuacct_cpuusage_write(struct cpuacct *ca, int cpu, u64 val)/* * Take rq->lock to make 64-bit write safe on 32-bit platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif for (i =0 ; i <CPUACCT_STAT_NSTATS ; i++) cpuusage->usages[i] = val;#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif }
@@ -253,13 +253,13 @@ static int cpuacct_all_seq_show(struct seq_file *m, void *V)* Take rq->lock to make 64-bit read safe on 32-bit * platforms. */ -raw_spin_lock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_lock_irq (cpu_rq (cpu));#endif seq_printf (m," %llu" , cpuusage->usages[index]);#ifndef CONFIG_64BIT -raw_spin_unlock_irq (&cpu_rq (cpu)->lock); +raw_spin_rq_unlock_irq (cpu_rq (cpu));#endif }seq_puts (m,"\n" );
kernel/sched/deadline.c +11 -11
@@ -157,7 +157,7 @@ void __add_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->running_bw += dl_bw;SCHED_WARN_ON (dl_rq->running_bw < old);/* overflow */ SCHED_WARN_ON (dl_rq->running_bw > dl_rq->this_bw);
@@ -170,7 +170,7 @@ void __sub_running_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->running_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->running_bw -= dl_bw;SCHED_WARN_ON (dl_rq->running_bw > old);/* underflow */ if (dl_rq->running_bw > old)
@@ -184,7 +184,7 @@ void __add_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->this_bw += dl_bw;SCHED_WARN_ON (dl_rq->this_bw < old);/* overflow */ }
@@ -194,7 +194,7 @@ void __sub_rq_bw(u64 dl_bw, struct dl_rq *dl_rq) {u64 old = dl_rq->this_bw; -lockdep_assert_held (&(rq_of_dl_rq (dl_rq))->lock); +lockdep_assert_rq_held (rq_of_dl_rq (dl_rq)); dl_rq->this_bw -= dl_bw;SCHED_WARN_ON (dl_rq->this_bw > old);/* underflow */ if (dl_rq->this_bw > old)
@@ -987,7 +987,7 @@ static int start_dl_timer(struct task_struct *p)ktime_t now, act;s64 delta; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);/* * We want the timer to fire at the deadline, but considering
@@ -1097,9 +1097,9 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)* If the runqueue is no longer available, migrate the * task elsewhere. This necessarily changes rq. */ -lockdep_unpin_lock (&rq->lock, rf.cookie); +lockdep_unpin_lock (rq_lockp (rq), rf.cookie); rq =dl_task_offline_migration (rq, p); - rf.cookie =lockdep_pin_lock (&rq->lock); + rf.cookie =lockdep_pin_lock (rq_lockp (rq));update_rq_clock (rq);/*
@@ -1731,7 +1731,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unused* from try_to_wake_up(). Hence, p->pi_lock is locked, but * rq->lock is not... So, lock it */ -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);if (p->dl.dl_non_contending) {sub_running_bw (&p->dl, &rq->dl); p->dl.dl_non_contending =0 ;
@@ -1746,7 +1746,7 @@ static void migrate_task_rq_dl(struct task_struct *p, int new_cpu __maybe_unusedput_task_struct (p); }sub_rq_bw (&p->dl, &rq->dl); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }static void check_preempt_equal_dl (struct rq *rq,struct task_struct *p)
@@ -2291,10 +2291,10 @@ skip:double_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq); } }
kernel/sched/debug.c +2 -2
@@ -576,7 +576,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"exec_clock" ,SPLIT_NS (cfs_rq->exec_clock)); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (rb_first_cached (&cfs_rq->tasks_timeline)) MIN_vruntime = (__pick_first_entity (cfs_rq))->vruntime; last =__pick_last_entity (cfs_rq);
@@ -584,7 +584,7 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq) max_vruntime = last->vruntime; min_vruntime = cfs_rq->min_vruntime; rq0_min_vruntime =cpu_rq (0 )->cfs.min_vruntime; -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags);SEQ_printf (m," .%-30s: %Ld.%06ld\n" ,"MIN_vruntime" ,SPLIT_NS (MIN_vruntime)); SEQ_printf(m, " .%-30s: %Ld.%06ld\n", "min_vruntime",
kernel/sched/fair.c +17 -18
@@ -1107,7 +1107,7 @@ struct numa_group {static struct numa_group *deref_task_numa_group (struct task_struct *p) {return rcu_dereference_check (p->numa_group, p == current || - (lockdep_is_held (&task_rq (p)->lock) && !READ_ONCE (p->on_cpu))); + (lockdep_is_held (rq_lockp (task_rq (p))) && !READ_ONCE (p->on_cpu))); }static struct numa_group *deref_curr_numa_group (struct task_struct *p)
@@ -5328,7 +5328,7 @@ static void __maybe_unused update_runtime_enabled(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -5347,7 +5347,7 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq) {struct task_group *tg; -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);rcu_read_lock ();list_for_each_entry_rcu (tg, &task_groups, list) {
@@ -6891,7 +6891,7 @@ static void migrate_task_rq_fair(struct task_struct *p, int new_cpu)* In case of TASK_ON_RQ_MIGRATING we in fact hold the 'old' * rq->lock and can modify state directly. */ -lockdep_assert_held (&task_rq (p)->lock); +lockdep_assert_rq_held (task_rq (p));detach_entity_cfs_rq (&p->se); }else {
@@ -7518,7 +7518,7 @@ static int task_hot(struct task_struct *p, struct lb_env *env) {s64 delta; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);if (p->sched_class != &fair_sched_class)return 0 ;
@@ -7616,7 +7616,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) {int tsk_cache_hot; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);/* * We do not migrate tasks that are:
@@ -7705,7 +7705,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)*/ static void detach_task (struct task_struct *p,struct lb_env *env) { -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);deactivate_task (env->src_rq, p,DEQUEUE_NOCLOCK );set_task_cpu (p, env->dst_cpu);
@@ -7721,7 +7721,7 @@ static struct task_struct *detach_one_task(struct lb_env *env) {struct task_struct *p; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);list_for_each_entry_reverse (p, &env->src_rq->cfs_tasks, se.group_node) {
@@ -7757,7 +7757,7 @@ static int detach_tasks(struct lb_env *env)struct task_struct *p;int detached =0 ; -lockdep_assert_held (&env->src_rq->lock); +lockdep_assert_rq_held (env->src_rq);/* * Source run queue has been emptied by another CPU, clear
@@ -7887,7 +7887,7 @@ next:*/ static void attach_task (struct rq *rq,struct task_struct *p) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);BUG_ON (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );
@@ -9798,7 +9798,7 @@ more_balance:if (need_active_balance (&env)) {unsigned long flags; -raw_spin_lock_irqsave (&busiest->lock, flags); +raw_spin_rq_lock_irqsave (busiest, flags);/* * Don't kick the active_load_balance_cpu_stop,
@@ -9806,8 +9806,7 @@ more_balance:* moved to this_cpu: */ if (!cpumask_test_cpu (this_cpu, busiest->curr->cpus_ptr)) { -raw_spin_unlock_irqrestore (&busiest->lock, - flags); +raw_spin_rq_unlock_irqrestore (busiest, flags); goto out_one_pinned; }
@@ -9824,7 +9823,7 @@ more_balance: busiest->push_cpu = this_cpu; active_balance =1 ; } -raw_spin_unlock_irqrestore (&busiest->lock, flags); +raw_spin_rq_unlock_irqrestore (busiest, flags);if (active_balance) { stop_one_cpu_nowait(cpu_of(busiest),
@@ -10649,7 +10648,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) goto out; } -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);update_blocked_averages (this_cpu);rcu_read_lock ();
@@ -10688,7 +10687,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf) }rcu_read_unlock (); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq);if (curr_cost > this_rq->max_idle_balance_cost) this_rq->max_idle_balance_cost = curr_cost;
@@ -11175,9 +11174,9 @@ void unregister_fair_sched_group(struct task_group *tg) rq =cpu_rq (cpu); -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);list_del_leaf_cfs_rq (tg->cfs_rq[cpu]); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags); } }
kernel/sched/idle.c +2 -2
@@ -455,10 +455,10 @@ struct task_struct *pick_next_task_idle(struct rq *rq)static void dequeue_task_idle (struct rq *rq,struct task_struct *p,int flags) { -raw_spin_unlock_irq (&rq->lock); +raw_spin_rq_unlock_irq (rq);printk (KERN_ERR "bad: scheduling from the idle thread!\n" );dump_stack (); -raw_spin_lock_irq (&rq->lock); +raw_spin_rq_lock_irq (rq); }/*
kernel/sched/pelt.h +1 -1
@@ -141,7 +141,7 @@ static inline void update_idle_rq_clock_pelt(struct rq *rq)static inline u64 rq_clock_pelt (struct rq *rq) { -lockdep_assert_held (&rq->lock); +lockdep_assert_rq_held (rq);assert_clock_updated (rq);return rq->clock_pelt - rq->lost_idle_time;
kernel/sched/rt.c +8 -8
@@ -888,7 +888,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (skip)continue ; -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);update_rq_clock (rq);if (rt_rq->rt_time) {
@@ -926,7 +926,7 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)if (enqueue)sched_rt_rq_enqueue (rt_rq); -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }if (!throttled && (!rt_bandwidth_enabled () || rt_b->rt_runtime ==RUNTIME_INF ))
@@ -1894,10 +1894,10 @@ retry:*/ push_task =get_push_task (rq);if (push_task) { -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq);stop_one_cpu_nowait (rq->cpu, push_cpu_stop, push_task, &rq->push_work); -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq); }return 0 ;
@@ -2122,10 +2122,10 @@ void rto_push_irq_work_func(struct irq_work *work)* When it gets updated, a check is made if a push is possible. */ if (has_pushable_tasks (rq)) { -raw_spin_lock (&rq->lock); +raw_spin_rq_lock (rq);while (push_rt_task (rq, true)) ; -raw_spin_unlock (&rq->lock); +raw_spin_rq_unlock (rq); }raw_spin_lock (&rd->rto_lock);
@@ -2243,10 +2243,10 @@ skip:double_unlock_balance (this_rq, src_rq);if (push_task) { -raw_spin_unlock (&this_rq->lock); +raw_spin_rq_unlock (this_rq);stop_one_cpu_nowait (src_rq->cpu, push_cpu_stop, push_task, &src_rq->push_work); -raw_spin_lock (&this_rq->lock); +raw_spin_rq_lock (this_rq); } }
kernel/sched/sched.h +52 -53
275 lines not on this page · raw message
kernel/sched/topology.c +2 -2
@@ -467,7 +467,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)struct root_domain *old_rd =NULL ;unsigned long flags; -raw_spin_lock_irqsave (&rq->lock, flags); +raw_spin_rq_lock_irqsave (rq, flags);if (rq->rd) { old_rd = rq->rd;
@@ -493,7 +493,7 @@ void rq_attach_root(struct rq *rq, struct root_domain *rd)if (cpumask_test_cpu (rq->cpu, cpu_active_mask))set_rq_online (rq); -raw_spin_unlock_irqrestore (&rq->lock, flags); +raw_spin_rq_unlock_irqrestore (rq, flags);if (old_rd)call_rcu (&old_rd->rcu, free_rootdomain);
Peter Zijlstra
maintainerPATCH 4/19 sched: Prepare for Core-wide rq->lock
In reply to Peter ZijlstraWhen switching on core-sched, CPUs need to agree which lock to use for their RQ.
The new rule will be that rq->core_enabled will be toggled while holding all rq->__locks that belong to a core. This means we need to double check the rq->core_enabled value after each lock acquire and retry if it changed.
This also has implications for those sites that take multiple RQ locks, they need to be careful that the second lock doesn't end up being the first lock.
Verify the lock pointer after acquiring the first lock, because if they're on the same core, holding any of the rq->__lock instances will pin the core state.
While there, change the rq->__lock order to CPU number, instead of rq address, this greatly simplifies the next patch.
--- kernel/sched/core.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 41 +++++++++++------------------------------ 2 files changed, 57 insertions(+), 32 deletions(-)
kernel/sched/core.c +46 -2
@@ -186,12 +186,37 @@ int sysctl_sched_rt_runtime = 950000;void raw_spin_rq_lock_nested (struct rq *rq,int subclass) { -raw_spin_lock_nested (rq_lockp (rq), subclass); +raw_spinlock_t *lock; + +if (sched_core_disabled ()) { +raw_spin_lock_nested (&rq->__lock, subclass); +return ; + } + +for (;;) { + lock =rq_lockp (rq); +raw_spin_lock_nested (lock, subclass); +if (likely (lock ==rq_lockp (rq))) +return ; +raw_spin_unlock (lock); + } }bool raw_spin_rq_trylock (struct rq *rq) { -return raw_spin_trylock (rq_lockp (rq)); +raw_spinlock_t *lock; +bool ret; + +if (sched_core_disabled ()) +return raw_spin_trylock (&rq->__lock); + +for (;;) { + lock =rq_lockp (rq); + ret =raw_spin_trylock (lock); +if (!ret || (likely (lock ==rq_lockp (rq)))) +return ret; +raw_spin_unlock (lock); + } }void raw_spin_rq_unlock (struct rq *rq)
@@ -199,6 +224,25 @@ void raw_spin_rq_unlock(struct rq *rq)raw_spin_unlock (rq_lockp (rq)); } +#ifdef CONFIG_SMP +/* +* double_rq_lock - safely lock two runqueues +*/ +void double_rq_lock (struct rq *rq1,struct rq *rq2) +{ +lockdep_assert_irqs_disabled (); + +if (rq1->cpu > rq2->cpu) +swap (rq1, rq2); + +raw_spin_rq_lock (rq1); +if (rq_lockp (rq1) ==rq_lockp (rq2)) +return ; + +raw_spin_rq_lock_nested (rq2,SINGLE_DEPTH_NESTING ); +} +#endif +/* * __task_rq_lock - lock the rq @p resides on. */
kernel/sched/sched.h +11 -30
72 lines not on this page · raw message
Re: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
Hi Peter,
kernel/sched/core.c int sysctl_sched_rt_runtime = 950000; · line 191
Quoted 10 lines
kernel/sched/core.c +3
-1@@ -186,12 +186,37 @@ int sysctl_sched_rt_runtime = 950000;void raw_spin_rq_lock_nested (struct rq *rq,int subclass) { -raw_spin_lock_nested (rq_lockp (rq), subclass); +raw_spinlock_t *lock; + +if (sched_core_disabled ()) {
Nothing to stop sched_core from being enabled right here? Leading to us potentially taking the wrong lock.
Quoted 12 lines
kernel/sched/core.c +11
-0+raw_spin_lock_nested (&rq->__lock, subclass); +return ; + } + +for (;;) { + lock =rq_lockp (rq); +raw_spin_lock_nested (lock, subclass); +if (likely (lock ==rq_lockp (rq))) +return ; +raw_spin_unlock (lock); + } }
Peter Zijlstra
maintainerRe: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
Quoted 6 lines · Josh Don
Hi Peter,
Quoted 4 lines
kernel/sched/core.c +0
-0@@ -186,12 +186,37 @@ int sysctl_sched_rt_runtime = 950000;
Quoted 22 lines
kernel/sched/core.c int sysctl_sched_rt_runtime = 950000; · line 191
Quoted 6 lines
kernel/sched/core.c +3
-1void raw_spin_rq_lock_nested (struct rq *rq,int subclass) { -raw_spin_lock_nested (rq_lockp (rq), subclass); +raw_spinlock_t *lock; + +if (sched_core_disabled ()) {Nothing to stop
sched_corefrom being enabled right here? Leading to us potentially taking the wrong lock.Quoted 12 lines
kernel/sched/core.c +11
-0+raw_spin_lock_nested (&rq->__lock, subclass); +return ; + } + +for (;;) { + lock =rq_lockp (rq); +raw_spin_lock_nested (lock, subclass); +if (likely (lock ==rq_lockp (rq))) +return ; +raw_spin_unlock (lock); + } }
Very good; something like the below seems to be the best I can make of it..
kernel/sched/core.c +20 -4
@@ -290,6 +290,10 @@ static void sched_core_assert_empty(void)static void __sched_core_enable (void ) {static_branch_enable (&__sched_core_enabled); +/* +* Ensure raw_spin_rq_*lock*() have completed before flipping. +*/ +synchronize_sched ();__sched_core_flip (true);sched_core_assert_empty (); }
@@ -449,16 +453,22 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass) {raw_spinlock_t *lock; +preempt_disable ();if (sched_core_disabled ()) {raw_spin_lock_nested (&rq->__lock, subclass); +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; }for (;;) { lock =__rq_lockp (rq);raw_spin_lock_nested (lock, subclass); -if (likely (lock ==__rq_lockp (rq))) +if (likely (lock ==__rq_lockp (rq))) { +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; + }raw_spin_unlock (lock); } }
@@ -468,14 +478,20 @@ bool raw_spin_rq_trylock(struct rq *rq)raw_spinlock_t *lock;bool ret; -if (sched_core_disabled ()) -return raw_spin_trylock (&rq->__lock); +preempt_disable (); +if (sched_core_disabled ()) { + ret =raw_spin_trylock (&rq->__lock); +preempt_enable (); +return ret; + }for (;;) { lock =__rq_lockp (rq); ret =raw_spin_trylock (lock); -if (!ret || (likely (lock ==__rq_lockp (rq)))) +if (!ret || (likely (lock ==__rq_lockp (rq)))) { +preempt_enable ();return ret; + }raw_spin_unlock (lock); } }
Re: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
kernel/sched/core.c static void sched_core_assert_empty(void) · line 296
Quoted 12 lines
kernel/sched/core.c +4
-0@@ -290,6 +290,10 @@ static void sched_core_assert_empty(void)static void __sched_core_enable (void ) {static_branch_enable (&__sched_core_enabled); +/* +* Ensure raw_spin_rq_*lock*() have completed before flipping. +*/ +synchronize_sched ();
synchronize_rcu()
Quoted 4 lines
kernel/sched/core.c +0
-0__sched_core_flip (true);sched_core_assert_empty (); }
kernel/sched/core.c void raw_spin_rq_lock_nested(struct rq *rq, int subclass) · line 462
Quoted 10 lines
kernel/sched/core.c +3
-0@@ -449,16 +453,22 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass) {raw_spinlock_t *lock; +preempt_disable ();if (sched_core_disabled ()) {raw_spin_lock_nested (&rq->__lock, subclass); +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; }
This approach looks good to me. I'm guessing you went this route instead of doing the re-check after locking in order to optimize the disabled case?
Recommend a comment that the preempt_disable() here pairs with the synchronize_rcu() in __sched_core_enable().
Quoted 13 lines
kernel/sched/core.c +4
-1for (;;) { lock =__rq_lockp (rq);raw_spin_lock_nested (lock, subclass); -if (likely (lock ==__rq_lockp (rq))) +if (likely (lock ==__rq_lockp (rq))) { +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; + }raw_spin_unlock (lock); } }
Re: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
Quoted 23 lines · Josh Don
kernel/sched/core.c static void sched_core_assert_empty(void) · line 296
Quoted 13 lines
kernel/sched/core.c +4
-0@@ -290,6 +290,10 @@ static void sched_core_assert_empty(void)static void __sched_core_enable (void ) {static_branch_enable (&__sched_core_enabled); +/* +* Ensure raw_spin_rq_*lock*() have completed before flipping. +*/ +synchronize_sched ();
synchronize_rcu()Quoted 7 lines
kernel/sched/core.c +0
-0__sched_core_flip (true);sched_core_assert_empty (); }@@ -449,16 +453,22 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass) {raw_spinlock_t *lock;
Quoted 12 lines
kernel/sched/core.c void raw_spin_rq_lock_nested(struct rq *rq, int subclass) · line 462
Quoted 7 lines
kernel/sched/core.c +3
-0+preempt_disable ();if (sched_core_disabled ()) {raw_spin_lock_nested (&rq->__lock, subclass); +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; }This approach looks good to me. I'm guessing you went this route instead of doing the re-check after locking in order to optimize the disabled case?
Hi Josh and Peter,
I've been running into soft lookups and hard lockups when running a script that just cycles setting the cookie of a group of processes over and over again.
Unfortunately the only way I can reproduce this is by setting the cookies on qemu. I've tried sysbench, stress-ng but those seem to work just fine.
I'm running Peter's branch and even tried the suggested changes here but still see the same behavior. I enabled panic on hard lockup and here below is a snippet of the log.
Is there anything you'd like me to try or have any debugging you'd like me to do? I'd certainly like to get to the bottom of this.
Thanks and have a great day,
Don
[ 1164.129706] NMI backtrace for cpu 10 [ 1164.129707] CPU: 10 PID: 6994 Comm: CPU 2/KVM Kdump: loaded Tainted: G I L 5.12.0-rc8+ #4 [ 1164.129707] Hardware name: Dell Inc. PowerEdge R640/0W23H8, BIOS 2.9.4 11/06/2020 [ 1164.129708] RIP: 0010:native_queued_spin_lock_slowpath+0x69/0x1e0 [ 1164.129708] Code: 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 20 85 c0 75 0c b8 01 00 00 00 66 89 07 5d c3 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 eb ec f6 c4 01 75 04 c6 47 01 [ 1164.129709] RSP: 0018:ffffb35d591e0e20 EFLAGS: 00000002 [ 1164.129710] RAX: 0000000000200101 RBX: ffff9ecb0096c180 RCX: 0000000000000001 [ 1164.129710] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9ecb0096c180 [ 1164.129711] RBP: ffffb35d591e0e20 R08: 0000000000000000 R09: 000000010002e2a6 [ 1164.129711] R10: 0000000000000000 R11: ffffb35d591e0ff8 R12: ffff9ecb0096c180 [ 1164.129712] R13: 000000000000000a R14: 000000000000000a R15: ffff9e6e8dc88000 [ 1164.129712] FS: 00007f91bd79f700(0000) GS:ffff9ecb00940000(0000) knlGS:0000000000000000 [ 1164.129713] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1164.129713] CR2: 000000000255d05c CR3: 000000022eb8c005 CR4: 00000000007726e0 [ 1164.129714] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1164.129715] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1164.129715] PKRU: 55555554 [ 1164.129715] Call Trace: [ 1164.129716] <IRQ> [ 1164.129716] _raw_spin_lock+0x1f/0x30 [ 1164.129717] raw_spin_rq_lock_nested.part.93+0x2b/0x60 [ 1164.129717] raw_spin_rq_lock_nested.constprop.126+0x1a/0x20 [ 1164.129718] scheduler_tick+0x4c/0x240 [ 1164.129718] ? tick_sched_handle.isra.17+0x60/0x60 [ 1164.129719] update_process_times+0xab/0xc0 [ 1164.129719] tick_sched_handle.isra.17+0x21/0x60 [ 1164.129720] tick_sched_timer+0x6b/0x80 [ 1164.129720] __hrtimer_run_queues+0x10d/0x230 [ 1164.129721] hrtimer_interrupt+0xe7/0x230 [ 1164.129721] __sysvec_apic_timer_interrupt+0x62/0x100 [ 1164.129721] sysvec_apic_timer_interrupt+0x51/0xa0 [ 1164.129722] </IRQ> [ 1164.129722] asm_sysvec_apic_timer_interrupt+0x12/0x20 [ 1164.129722] RIP: 0010:smp_call_function_single+0xf6/0x130 [ 1164.129723] Code: 00 75 4d 48 83 c4 48 41 5a 5d 49 8d 62 f8 c3 48 89 75 c0 48 8d 75 b0 48 89 55 c8 e8 84 fe ff ff 8b 55 b8 83 e2 01 74 0a f3 90 <8b> 55 b8 83 e2 01 75 f6 eb c0 8b 05 4a d8 b0 01 85 c0 0f 85 67 ff [ 1164.129724] RSP: 0018:ffffb35d5e20fb20 EFLAGS: 00000202 [ 1164.129725] RAX: 0000000000000000 RBX: ffff9e6e8df12600 RCX: ffffb35d5f2ffcc0 [ 1164.129725] RDX: 0000000000000001 RSI: ffffb35d5e20fb20 RDI: ffffb35d5e20fb20 [ 1164.129725] RBP: ffffb35d5e20fb70 R08: 0000000000001000 R09: 0000000000001907 [ 1164.129726] R10: ffffb35d5e20fb98 R11: 00000000003d0900 R12: 000000000000000a [ 1164.129726] R13: 000000000000000c R14: 0000000000000000 R15: 0000000000000000 [ 1164.129727] ? vmclear_error+0x150/0x390 [kvm_intel] [ 1164.129727] vmx_vcpu_load_vmcs+0x1b2/0x330 [kvm_intel] [ 1164.129728] ? update_load_avg+0x1b8/0x640 [ 1164.129728] ? vmx_vcpu_load_vmcs+0x1b2/0x330 [kvm_intel] [ 1164.129728] ? load_fixmap_gdt+0x23/0x30 [ 1164.129729] vmx_vcpu_load_vmcs+0x309/0x330 [kvm_intel] [ 1164.129729] kvm_arch_vcpu_load+0x41/0x260 [kvm] [ 1164.129729] ? __switch_to_xtra+0xe3/0x4e0 [ 1164.129730] kvm_io_bus_write+0xbc/0xf0 [kvm] [ 1164.129730] ? kvm_io_bus_write+0xbc/0xf0 [kvm] [ 1164.129730] finish_task_switch+0x184/0x290 [ 1164.129731] __schedule+0x30a/0x12f0 [ 1164.129731] schedule+0x40/0xb0 [ 1164.129732] kvm_vcpu_block+0x57/0x4b0 [kvm] [ 1164.129732] kvm_arch_vcpu_ioctl_run+0x3fe/0x1b60 [kvm] [ 1164.129732] kvm_vcpu_kick+0x59c/0xa10 [kvm] [ 1164.129733] ? kvm_vcpu_kick+0x59c/0xa10 [kvm] [ 1164.129733] __x64_sys_ioctl+0x96/0xd0 [ 1164.129733] do_syscall_64+0x37/0x80 [ 1164.129734] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 1164.129734] RIP: 0033:0x7fb1c8a4c317 [ 1164.129735] Code: b3 66 90 48 8b 05 71 4b 2d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 41 4b 2d 00 f7 d8 64 89 01 48 [ 1164.129735] RSP: 002b:00007f91bd79e7e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 1164.129736] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007fb1c8a4c317 [ 1164.129737] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000001a [ 1164.129737] RBP: 0000000000000000 R08: 000055f3e6d027c0 R09: 00007f91a80040b8 [ 1164.129738] R10: 00007f91a80050a0 R11: 0000000000000246 R12: 00007fb1c4023010 [ 1164.129739] R13: 000055f3e569d880 R14: 00007fb1cb8e7000 R15: 00007ffd86806f70
Quoted 16 lines
Recommend a comment that the
preempt_disable()here pairs with thesynchronize_rcu()in__sched_core_enable().Quoted 13 lines
kernel/sched/core.c +4
-1for (;;) { lock =__rq_lockp (rq);raw_spin_lock_nested (lock, subclass); -if (likely (lock ==__rq_lockp (rq))) +if (likely (lock ==__rq_lockp (rq))) { +/* preempt *MUST* still be disabled here */ +preempt_enable_no_resched ();return ; + }raw_spin_unlock (lock); } }
Re: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
Quoted 8 lines · Don Hiatt
Hi Josh and Peter,
I've been running into soft lookups and hard lockups when running a script that just cycles setting the cookie of a group of processes over and over again.
Unfortunately the only way I can reproduce this is by setting the cookies on qemu. I've tried sysbench, stress-ng but those seem to work just fine.
Quoted 6 lines
I'm running Peter's branch and even tried the suggested changes here but still see the same behavior. I enabled panic on hard lockup and here below is a snippet of the log.
Is there anything you'd like me to try or have any debugging you'd like me to do? I'd certainly like to get to the bottom of this.
Hi Don,
I tried to repro using qemu, but did not generate a lockup. Could you provide more details on what your script is doing (or better yet, share the script directly)? I would have expected you to potentially hit a lockup if you were cycling sched_core being enabled and disabled, but it sounds like you are just recreating the cookie for a process group over and over?
Best,
Josh
Re: [PATCH 04/19] sched: Prepare for Core-wide rq->lock
Quoted 17 lines · Josh Don
Quoted 8 lines · Don Hiatt
Hi Josh and Peter,
I've been running into soft lookups and hard lockups when running a script that just cycles setting the cookie of a group of processes over and over again.
Unfortunately the only way I can reproduce this is by setting the cookies on qemu. I've tried sysbench, stress-ng but those seem to work just fine.
Quoted 6 lines
I'm running Peter's branch and even tried the suggested changes here but still see the same behavior. I enabled panic on hard lockup and here below is a snippet of the log.
Is there anything you'd like me to try or have any debugging you'd like me to do? I'd certainly like to get to the bottom of this.
Quoted 9 lines
Hi Don,
I tried to repro using qemu, but did not generate a lockup. Could you provide more details on what your script is doing (or better yet, share the script directly)? I would have expected you to potentially hit a lockup if you were cycling
sched_corebeing enabled and disabled, but it sounds like you are just recreating the cookie for a process group over and over?
I saw something similar on a bare metal hardware. Also tried the suggested patch here and no luck. Panic stack attached with softlockup_all_cpu_backtrace=1.
(sorry, my system has 192 cpus and somehow putting 184 cpus offline causes system hang without any message...)
My script created the core cookie for two different process groups. The one is for sysbench cpu, the other is for sysbench mysql, mysqld(cookie=0) is
also on the same machine. The number of tasks in each category is the same as the number of CPUs on the system. And cookie is created just during task startup.
Please let me know if the script is needed, I'll push it to github with some cleanup.
Thanks,
-Aubrey
��[ B E G I N ] 2 0 2 1 / 4 / 2 8 1 : 0 0 : 1 1