sched: Use lock guards, wave 1
Peter Zijlstra
maintainercover sched: Use lock guards, wave 1
Hi,
This is the first of two series converting kernel/sched/core.c to use the shiny new Scope-Based Resource Management machinery that has recently been merged:
https://lkml.kernel.org/r/20230612093537.614161713%40infradead.org
TL;DR:
void foo()
{
raw_spin_lock(&lock);
if (!cond)
goto unlock;
...
unlock:
raw_spin_unlock(&lock);
}can now be written like:
void foo()
{
guard(raw_spinlock)(&lock);
if (!cond)
return;
...
}or:
void foo()
{
scoped_guard (raw_spinlock, &lock) {
if (!cond)
break;
...
}
}Where guard() instantiates a 'cleanup' variable that is initialized by an expression that locks and returns the lock pointer. When this variable goes out of scope, the cleanup does the unlock. And scoped_guard() does the same but creates an explicit scope (using for).
By having the unlock be implicit, the code becomes simpler since it is no longer needed to manually track the various exit/error cases and many goto's just go away.
Specifically, every patch in this series is aimed at removing an out/unlock label and it's corresponding gotos.
If there are no objections / comments, I'm aiming to post the second batch at the end of the week.
Also available at:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/guards
Peter Zijlstra
maintainerPATCH 1/9 sched: Simplify get_nohz_timer_target()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
kernel/sched/core.c +6 -9
@@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask =housekeeping_cpumask (HK_TYPE_TIMER ); -rcu_read_lock (); +guard (rcu)(); +for_each_domain (cpu, sd) {for_each_cpu_and (i,sched_domain_span (sd), hk_mask) {if (cpu == i)continue ; -if (!idle_cpu (i)) { - cpu = i; - goto unlock; - } +if (!idle_cpu (i)) +return i; } }if (default_cpu ==-1 ) default_cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); - cpu = default_cpu; -unlock: -rcu_read_unlock (); -return cpu; + +return default_cpu; }/*
Re: [PATCH 1/9] sched: Simplify get_nohz_timer_target()
Quoted 30 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)kernel/sched/core.c +4
-5@@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask =housekeeping_cpumask (HK_TYPE_TIMER ); -rcu_read_lock (); +guard (rcu)(); +for_each_domain (cpu, sd) {for_each_cpu_and (i,sched_domain_span (sd), hk_mask) {if (cpu == i)continue ; -if (!idle_cpu (i)) { - cpu = i; - goto unlock; - } +if (!idle_cpu (i)) +return i; } }
kernel/sched/core.c int get_nohz_timer_target(void) · line 1116
Quoted 10 lines
kernel/sched/core.c +2
-4if (default_cpu ==-1 ) default_cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); - cpu = default_cpu; -unlock: -rcu_read_unlock (); -return cpu; + +return default_cpu; }
I haven't looked into the actual implementation of the guard stuff, but rcu_read_lock_guarded() is less of an eyesore to me than guard(rcu)(); TBH.
thanks,
- Joel
Peter Zijlstra
maintainerRe: [PATCH 1/9] sched: Simplify get_nohz_timer_target()
Quoted 44 lines · Joel Fernandes
Quoted 30 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)kernel/sched/core.c +4
-5@@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask =housekeeping_cpumask (HK_TYPE_TIMER ); -rcu_read_lock (); +guard (rcu)(); +for_each_domain (cpu, sd) {for_each_cpu_and (i,sched_domain_span (sd), hk_mask) {if (cpu == i)continue ; -if (!idle_cpu (i)) { - cpu = i; - goto unlock; - } +if (!idle_cpu (i)) +return i; } }kernel/sched/core.c int get_nohz_timer_target(void) · line 1116
Quoted 10 lines
kernel/sched/core.c +2
-4if (default_cpu ==-1 ) default_cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); - cpu = default_cpu; -unlock: -rcu_read_unlock (); -return cpu; + +return default_cpu; }Reviewed-by Joel Fernandes (Google) <[email protected]>
Quoted 3 lines
I haven't looked into the actual implementation of the guard stuff, but
rcu_read_lock_guarded()is less of an eyesore to me than guard(rcu)(); TBH.
I readily admit it isn't the prettiest construct, my brain is warped by many years of C++ and I can read it as: guard<rcu>(), but I'm not sure that's actually better :-)
The advantage of all this is that you also get:
scoped_guard (rcu) {
}for 'free'.
Re: [PATCH 1/9] sched: Simplify get_nohz_timer_target()
Quoted 54 lines · Peter Zijlstra
Quoted 44 lines · Joel Fernandes
Quoted 30 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)kernel/sched/core.c +4
-5@@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask =housekeeping_cpumask (HK_TYPE_TIMER ); -rcu_read_lock (); +guard (rcu)(); +for_each_domain (cpu, sd) {for_each_cpu_and (i,sched_domain_span (sd), hk_mask) {if (cpu == i)continue ; -if (!idle_cpu (i)) { - cpu = i; - goto unlock; - } +if (!idle_cpu (i)) +return i; } }kernel/sched/core.c int get_nohz_timer_target(void) · line 1116
Quoted 10 lines
kernel/sched/core.c +2
-4if (default_cpu ==-1 ) default_cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); - cpu = default_cpu; -unlock: -rcu_read_unlock (); -return cpu; + +return default_cpu; }Reviewed-by Joel Fernandes (Google) <[email protected]>Quoted 3 lines
I haven't looked into the actual implementation of the guard stuff, but
rcu_read_lock_guarded()is less of an eyesore to me than guard(rcu)(); TBH.I readily admit it isn't the prettiest construct, my brain is warped by many years of C++ and I can read it as: guard<rcu>(), but I'm not sure that's actually better :-)
Quoted 6 lines
The advantage of all this is that you also get:
scoped_guard (rcu) { }for 'free'.
Yes, overall the readability improvement is quite appealing. Thank you Peter!
- Joel
[tip: sched/core] sched: Simplify get_nohz_timer_target()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 7537b90c0036759e0b1b43dfbc6224dc5e900b13 Gitweb: https://git.kernel.org/tip/7537b90c0036759e0b1b43dfbc6224dc5e900b13 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:22 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:24 +02:00
sched: Simplify get_nohz_timer_target()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-)
kernel/sched/core.c +6 -9
@@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask =housekeeping_cpumask (HK_TYPE_TIMER ); -rcu_read_lock (); +guard (rcu)(); +for_each_domain (cpu, sd) {for_each_cpu_and (i,sched_domain_span (sd), hk_mask) {if (cpu == i)continue ; -if (!idle_cpu (i)) { - cpu = i; - goto unlock; - } +if (!idle_cpu (i)) +return i; } }if (default_cpu ==-1 ) default_cpu =housekeeping_any_cpu (HK_TYPE_TIMER ); - cpu = default_cpu; -unlock: -rcu_read_unlock (); -return cpu; + +return default_cpu; }/*
Peter Zijlstra
maintainerPATCH 2/9 sched: Simplify sysctl_sched_uclamp_handler()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/sched/core.c +4 -7
@@ -1801,7 +1801,8 @@ static int sysctl_sched_uclamp_handler(sint old_min, old_max, old_min_rt;int result; -mutex_lock (&uclamp_mutex); +guard (mutex)(&uclamp_mutex); + old_min = sysctl_sched_uclamp_util_min; old_max = sysctl_sched_uclamp_util_max; old_min_rt = sysctl_sched_uclamp_util_min_rt_default;
@@ -1810,7 +1811,7 @@ static int sysctl_sched_uclamp_handler(sif (result) goto undo;if (!write) - goto done; +return 0 ; if (sysctl_sched_uclamp_util_min > sysctl_sched_uclamp_util_max || sysctl_sched_uclamp_util_max > SCHED_CAPACITY_SCALE ||
@@ -1846,16 +1847,12 @@ static int sysctl_sched_uclamp_handler(s* Otherwise, keep it simple and do just a lazy update at each next * task enqueue time. */ - - goto done; +return 0 ; undo: sysctl_sched_uclamp_util_min = old_min; sysctl_sched_uclamp_util_max = old_max; sysctl_sched_uclamp_util_min_rt_default = old_min_rt; -done: -mutex_unlock (&uclamp_mutex); - return result; }#endif
[tip: sched/core] sched: Simplify sysctl_sched_uclamp_handler()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 0f92cdf36f848f1c077924f857a49789e00331c0 Gitweb: https://git.kernel.org/tip/0f92cdf36f848f1c077924f857a49789e00331c0 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:23 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:24 +02:00
sched: Simplify sysctl_sched_uclamp_handler()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/sched/core.c +4 -7
@@ -1827,7 +1827,8 @@ static int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,int old_min, old_max, old_min_rt;int result; -mutex_lock (&uclamp_mutex); +guard (mutex)(&uclamp_mutex); + old_min = sysctl_sched_uclamp_util_min; old_max = sysctl_sched_uclamp_util_max; old_min_rt = sysctl_sched_uclamp_util_min_rt_default;
@@ -1836,7 +1837,7 @@ static int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,if (result) goto undo;if (!write) - goto done; +return 0 ; if (sysctl_sched_uclamp_util_min > sysctl_sched_uclamp_util_max || sysctl_sched_uclamp_util_max > SCHED_CAPACITY_SCALE ||
@@ -1872,16 +1873,12 @@ static int sysctl_sched_uclamp_handler(struct ctl_table *table, int write,* Otherwise, keep it simple and do just a lazy update at each next * task enqueue time. */ - - goto done; +return 0 ; undo: sysctl_sched_uclamp_util_min = old_min; sysctl_sched_uclamp_util_max = old_max; sysctl_sched_uclamp_util_min_rt_default = old_min_rt; -done: -mutex_unlock (&uclamp_mutex); - return result; }#endif
Peter Zijlstra
maintainerPATCH 3/9 sched: Simplify: migrate_swap_stop()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 23 +++++++---------------- kernel/sched/sched.h | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 16 deletions(-)
kernel/sched/core.c +7 -16
@@ -3258,7 +3258,6 @@ static int migrate_swap_stop(void *data) {struct migration_swap_arg *arg = data;struct rq *src_rq, *dst_rq; -int ret = -EAGAIN ;if (!cpu_active (arg->src_cpu) || !cpu_active (arg->dst_cpu))return -EAGAIN ;
@@ -3266,33 +3265,25 @@ static int migrate_swap_stop(void *data) src_rq =cpu_rq (arg->src_cpu); dst_rq =cpu_rq (arg->dst_cpu); -double_raw_lock (&arg->src_task->pi_lock, - &arg->dst_task->pi_lock); -double_rq_lock (src_rq, dst_rq); +guard (double_raw_spinlock)(&arg->src_task->pi_lock, &arg->dst_task->pi_lock); +guard (double_rq_lock)(src_rq, dst_rq);if (task_cpu (arg->dst_task) != arg->dst_cpu) - goto unlock; +return -EAGAIN ;if (task_cpu (arg->src_task) != arg->src_cpu) - goto unlock; +return -EAGAIN ;if (!cpumask_test_cpu (arg->dst_cpu, arg->src_task->cpus_ptr)) - goto unlock; +return -EAGAIN ;if (!cpumask_test_cpu (arg->src_cpu, arg->dst_task->cpus_ptr)) - goto unlock; +return -EAGAIN ;__migrate_swap_task (arg->src_task, arg->dst_cpu);__migrate_swap_task (arg->dst_task, arg->src_cpu); - ret =0 ; - -unlock: -double_rq_unlock (src_rq, dst_rq); -raw_spin_unlock (&arg->dst_task->pi_lock); -raw_spin_unlock (&arg->src_task->pi_lock); - -return ret; +return 0 ; }/*
kernel/sched/sched.h +20 -0
@@ -2572,6 +2572,12 @@ static inline void double_rq_clock_clearstatic inline void double_rq_clock_clear_update (struct rq *rq1,struct rq *rq2) {}#endif +#define DEFINE_LOCK_GUARD_2 (name, type, _lock, _unlock, ...) \ +__DEFINE_UNLOCK_GUARD(name, type, _unlock, type *lock2; __VA_ARGS__) \ +static inline class_##name##_t class_##name##_constructor(type *lock, type *lock2) \ +{ class_##name##_t _t = { .lock = lock, .lock2 = lock2 }, *_T = &_t; \ + _lock; return _t; } +#ifdef CONFIG_SMP static inline bool rq_order_less (struct rq *rq1,struct rq *rq2)
@@ -2701,6 +2707,16 @@ static inline void double_raw_lock(raw_sraw_spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); } +static inline void double_raw_unlock (raw_spinlock_t *l1,raw_spinlock_t *l2) +{ +raw_spin_unlock (l1); +raw_spin_unlock (l2); +} + +DEFINE_LOCK_GUARD_2 (double_raw_spinlock, raw_spinlock_t, +double_raw_lock (_T->lock, _T->lock2), +double_raw_unlock (_T->lock, _T->lock2)) +/* * double_rq_unlock - safely unlock two runqueues *
@@ -2758,6 +2774,10 @@ static inline void double_rq_unlock(stru#endif +DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq, +double_rq_lock (_T->lock, _T->lock2), + double_rq_unlock(_T->lock, _T->lock2)) +extern struct sched_entity *__pick_first_entity (struct cfs_rq *cfs_rq);extern struct sched_entity *__pick_last_entity (struct cfs_rq *cfs_rq);
[tip: sched/core] sched: Simplify: migrate_swap_stop()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 5bb76f1ddf2a7dd98f5a89d7755600ed1b4a7fcd Gitweb: https://git.kernel.org/tip/5bb76f1ddf2a7dd98f5a89d7755600ed1b4a7fcd Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:24 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:25 +02:00
sched: Simplify: migrate_swap_stop()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 23 +++++++---------------- kernel/sched/sched.h | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 16 deletions(-)
kernel/sched/core.c +7 -16
@@ -3433,7 +3433,6 @@ static int migrate_swap_stop(void *data) {struct migration_swap_arg *arg = data;struct rq *src_rq, *dst_rq; -int ret = -EAGAIN ;if (!cpu_active (arg->src_cpu) || !cpu_active (arg->dst_cpu))return -EAGAIN ;
@@ -3441,33 +3440,25 @@ static int migrate_swap_stop(void *data) src_rq =cpu_rq (arg->src_cpu); dst_rq =cpu_rq (arg->dst_cpu); -double_raw_lock (&arg->src_task->pi_lock, - &arg->dst_task->pi_lock); -double_rq_lock (src_rq, dst_rq); +guard (double_raw_spinlock)(&arg->src_task->pi_lock, &arg->dst_task->pi_lock); +guard (double_rq_lock)(src_rq, dst_rq);if (task_cpu (arg->dst_task) != arg->dst_cpu) - goto unlock; +return -EAGAIN ;if (task_cpu (arg->src_task) != arg->src_cpu) - goto unlock; +return -EAGAIN ;if (!cpumask_test_cpu (arg->dst_cpu, arg->src_task->cpus_ptr)) - goto unlock; +return -EAGAIN ;if (!cpumask_test_cpu (arg->src_cpu, arg->dst_task->cpus_ptr)) - goto unlock; +return -EAGAIN ;__migrate_swap_task (arg->src_task, arg->dst_cpu);__migrate_swap_task (arg->dst_task, arg->src_cpu); - ret =0 ; - -unlock: -double_rq_unlock (src_rq, dst_rq); -raw_spin_unlock (&arg->dst_task->pi_lock); -raw_spin_unlock (&arg->src_task->pi_lock); - -return ret; +return 0 ; }/*
kernel/sched/sched.h +20 -0
@@ -2614,6 +2614,12 @@ static inline void double_rq_clock_clear_update(struct rq *rq1, struct rq *rq2)static inline void double_rq_clock_clear_update (struct rq *rq1,struct rq *rq2) {}#endif +#define DEFINE_LOCK_GUARD_2 (name, type, _lock, _unlock, ...) \ +__DEFINE_UNLOCK_GUARD(name, type, _unlock, type *lock2; __VA_ARGS__) \ +static inline class_##name##_t class_##name##_constructor(type *lock, type *lock2) \ +{ class_##name##_t _t = { .lock = lock, .lock2 = lock2 }, *_T = &_t; \ + _lock; return _t; } +#ifdef CONFIG_SMP static inline bool rq_order_less (struct rq *rq1,struct rq *rq2)
@@ -2743,6 +2749,16 @@ static inline void double_raw_lock(raw_spinlock_t *l1, raw_spinlock_t *l2)raw_spin_lock_nested (l2,SINGLE_DEPTH_NESTING ); } +static inline void double_raw_unlock (raw_spinlock_t *l1,raw_spinlock_t *l2) +{ +raw_spin_unlock (l1); +raw_spin_unlock (l2); +} + +DEFINE_LOCK_GUARD_2 (double_raw_spinlock, raw_spinlock_t, +double_raw_lock (_T->lock, _T->lock2), +double_raw_unlock (_T->lock, _T->lock2)) +/* * double_rq_unlock - safely unlock two runqueues *
@@ -2800,6 +2816,10 @@ static inline void double_rq_unlock(struct rq *rq1, struct rq *rq2)#endif +DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq, +double_rq_lock (_T->lock, _T->lock2), + double_rq_unlock(_T->lock, _T->lock2)) +extern struct sched_entity *__pick_first_entity (struct cfs_rq *cfs_rq);extern struct sched_entity *__pick_last_entity (struct cfs_rq *cfs_rq);
Peter Zijlstra
maintainerPATCH 4/9 sched: Simplify wake_up_if_idle()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 20 ++++++-------------- kernel/sched/sched.h | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 14 deletions(-)
kernel/sched/core.c +6 -14
@@ -3872,21 +3872,13 @@ static void __ttwu_queue_wakelist(structvoid wake_up_if_idle (int cpu) {struct rq *rq =cpu_rq (cpu); -struct rq_flags rf; -rcu_read_lock (); - -if (!is_idle_task (rcu_dereference (rq->curr))) - goto out; - -rq_lock_irqsave (rq, &rf); -if (is_idle_task (rq->curr)) -resched_curr (rq); -/* Else CPU is not idle, do nothing here: */ -rq_unlock_irqrestore (rq, &rf); - -out: -rcu_read_unlock (); +guard (rcu)(); +if (is_idle_task (rcu_dereference (rq->curr))) { +guard (rq_lock_irqsave)(rq); +if (is_idle_task (rq->curr)) +resched_curr (rq); + } }bool cpus_share_cache (int this_cpu,int that_cpu)
kernel/sched/sched.h +15 -0
@@ -1678,6 +1678,21 @@ rq_unlock(struct rq *rq, struct rq_flagsraw_spin_rq_unlock (rq); } +DEFINE_LOCK_GUARD_1(rq_lock, struct rq, +rq_lock (_T->lock, &_T->rf), +rq_unlock (_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irq, struct rq, +rq_lock_irq (_T->lock, &_T->rf), +rq_unlock_irq (_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irqsave, struct rq, +rq_lock_irqsave (_T->lock, &_T->rf), +rq_unlock_irqrestore (_T->lock, &_T->rf), + struct rq_flags rf) +static inline struct rq *this_rq_lock_irq (struct rq_flags *rf) __acquires(rq->lock)
[tip: sched/core] sched: Simplify wake_up_if_idle()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4eb054f92b066ec0a0cba6896ee8eff4c91dfc9e Gitweb: https://git.kernel.org/tip/4eb054f92b066ec0a0cba6896ee8eff4c91dfc9e Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:25 +02:00
sched: Simplify wake_up_if_idle()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 20 ++++++-------------- kernel/sched/sched.h | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 14 deletions(-)
kernel/sched/core.c +6 -14
@@ -3939,21 +3939,13 @@ static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flagsvoid wake_up_if_idle (int cpu) {struct rq *rq =cpu_rq (cpu); -struct rq_flags rf; - -rcu_read_lock (); -if (!is_idle_task (rcu_dereference (rq->curr))) - goto out; - -rq_lock_irqsave (rq, &rf); -if (is_idle_task (rq->curr)) -resched_curr (rq); -/* Else CPU is not idle, do nothing here: */ -rq_unlock_irqrestore (rq, &rf); - -out: -rcu_read_unlock (); +guard (rcu)(); +if (is_idle_task (rcu_dereference (rq->curr))) { +guard (rq_lock_irqsave)(rq); +if (is_idle_task (rq->curr)) +resched_curr (rq); + } }bool cpus_share_cache (int this_cpu,int that_cpu)
kernel/sched/sched.h +15 -0
@@ -1705,6 +1705,21 @@ rq_unlock(struct rq *rq, struct rq_flags *rf)raw_spin_rq_unlock (rq); } +DEFINE_LOCK_GUARD_1(rq_lock, struct rq, +rq_lock (_T->lock, &_T->rf), +rq_unlock (_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irq, struct rq, +rq_lock_irq (_T->lock, &_T->rf), +rq_unlock_irq (_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irqsave, struct rq, +rq_lock_irqsave (_T->lock, &_T->rf), +rq_unlock_irqrestore (_T->lock, &_T->rf), + struct rq_flags rf) +static inline struct rq *this_rq_lock_irq (struct rq_flags *rf) __acquires(rq->lock)
Peter Zijlstra
maintainerPATCH 5/9 sched: Simplify ttwu()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 221 +++++++++++++++++++++++++--------------------------- 1 file changed, 109 insertions(+), 112 deletions(-)
kernel/sched/core.c +109 -112
@@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpustruct sched_domain *sd;__schedstat_inc (p->stats.nr_wakeups_remote); -rcu_read_lock (); + +guard (rcu)();for_each_domain (rq->cpu, sd) {if (cpumask_test_cpu (cpu,sched_domain_span (sd))) {__schedstat_inc (sd->ttwu_wake_remote);break ; } } -rcu_read_unlock (); }if (wake_flags &WF_MIGRATED )
@@ -4172,10 +4172,9 @@ bool ttwu_state_match(struct task_structstatic int try_to_wake_up (struct task_struct *p,unsigned int state,int wake_flags) { -unsigned long flags; +guard (preempt)();int cpu, success =0 ; -preempt_disable ();if (p == current) {/* * We're waking current, this means 'p->on_rq' and 'task_cpu(p)
@@ -4202,129 +4201,127 @@ try_to_wake_up(struct task_struct *p, un* reordered with p->state check below. This pairs with smp_store_mb() * in set_current_state() that the waiting thread does. */ -raw_spin_lock_irqsave (&p->pi_lock, flags); -smp_mb__after_spinlock (); -if (!ttwu_state_match (p, state, &success)) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { +smp_mb__after_spinlock (); +if (!ttwu_state_match (p, state, &success)) +break ; -trace_sched_waking (p); +trace_sched_waking (p); -/* -* Ensure we load p->on_rq _after_ p->state, otherwise it would -* be possible to, falsely, observe p->on_rq == 0 and get stuck -* in smp_cond_load_acquire() below. -* -* sched_ttwu_pending() try_to_wake_up() -* STORE p->on_rq = 1 LOAD p->state -* UNLOCK rq->lock -* -* __schedule() (switch to task 'p') -* LOCK rq->lock smp_rmb(); -* smp_mb__after_spinlock(); -* UNLOCK rq->lock -* -* [task p] -* STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq -* -* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in -* __schedule(). See the comment for smp_mb__after_spinlock(). -* -* A similar smb_rmb() lives in try_invoke_on_locked_down_task(). -*/ -smp_rmb (); -if (READ_ONCE (p->on_rq) &&ttwu_runnable (p, wake_flags)) - goto unlock; +/* +* Ensure we load p->on_rq _after_ p->state, otherwise it would +* be possible to, falsely, observe p->on_rq == 0 and get stuck +* in smp_cond_load_acquire() below. +* +* sched_ttwu_pending() try_to_wake_up() +* STORE p->on_rq = 1 LOAD p->state +* UNLOCK rq->lock +* +* __schedule() (switch to task 'p') +* LOCK rq->lock smp_rmb(); +* smp_mb__after_spinlock(); +* UNLOCK rq->lock +* +* [task p] +* STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq +* +* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in +* __schedule(). See the comment for smp_mb__after_spinlock(). +* +* A similar smb_rmb() lives in try_invoke_on_locked_down_task(). +*/ +smp_rmb (); +if (READ_ONCE (p->on_rq) &&ttwu_runnable (p, wake_flags)) +break ;#ifdef CONFIG_SMP -/* -* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be -* possible to, falsely, observe p->on_cpu == 0. -* -* One must be running (->on_cpu == 1) in order to remove oneself -* from the runqueue. -* -* __schedule() (switch to task 'p') try_to_wake_up() -* STORE p->on_cpu = 1 LOAD p->on_rq -* UNLOCK rq->lock -* -* __schedule() (put 'p' to sleep) -* LOCK rq->lock smp_rmb(); -* smp_mb__after_spinlock(); -* STORE p->on_rq = 0 LOAD p->on_cpu -* -* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in -* __schedule(). See the comment for smp_mb__after_spinlock(). -* -* Form a control-dep-acquire with p->on_rq == 0 above, to ensure -* schedule()'s deactivate_task() has 'happened' and p will no longer -* care about it's own p->state. See the comment in __schedule(). -*/ -smp_acquire__after_ctrl_dep (); +/* +* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be +* possible to, falsely, observe p->on_cpu == 0. +* +* One must be running (->on_cpu == 1) in order to remove oneself +* from the runqueue. +* +* __schedule() (switch to task 'p') try_to_wake_up() +* STORE p->on_cpu = 1 LOAD p->on_rq +* UNLOCK rq->lock +* +* __schedule() (put 'p' to sleep) +* LOCK rq->lock smp_rmb(); +* smp_mb__after_spinlock(); +* STORE p->on_rq = 0 LOAD p->on_cpu +* +* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in +* __schedule(). See the comment for smp_mb__after_spinlock(). +* +* Form a control-dep-acquire with p->on_rq == 0 above, to ensure +* schedule()'s deactivate_task() has 'happened' and p will no longer +* care about it's own p->state. See the comment in __schedule(). +*/ +smp_acquire__after_ctrl_dep (); -/* -* We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq -* == 0), which means we need to do an enqueue, change p->state to -* TASK_WAKING such that we can unlock p->pi_lock before doing the -* enqueue, such as ttwu_queue_wakelist(). -*/ -WRITE_ONCE (p->__state,TASK_WAKING ); +/* +* We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq +* == 0), which means we need to do an enqueue, change p->state to +* TASK_WAKING such that we can unlock p->pi_lock before doing the +* enqueue, such as ttwu_queue_wakelist(). +*/ +WRITE_ONCE (p->__state,TASK_WAKING ); -/* -* If the owning (remote) CPU is still in the middle of schedule() with -* this task as prev, considering queueing p on the remote CPUs wake_list -* which potentially sends an IPI instead of spinning on p->on_cpu to -* let the waker make forward progress. This is safe because IRQs are -* disabled and the IPI will deliver after on_cpu is cleared. -* -* Ensure we load task_cpu(p) after p->on_cpu: -* -* set_task_cpu(p, cpu); -* STORE p->cpu = @cpu -* __schedule() (switch to task 'p') -* LOCK rq->lock -* smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) -* STORE p->on_cpu = 1 LOAD p->cpu -* -* to ensure we observe the correct CPU on which the task is currently -* scheduling. -*/ -if (smp_load_acquire (&p->on_cpu) && -ttwu_queue_wakelist (p,task_cpu (p), wake_flags)) - goto unlock; +/* +* If the owning (remote) CPU is still in the middle of schedule() with +* this task as prev, considering queueing p on the remote CPUs wake_list +* which potentially sends an IPI instead of spinning on p->on_cpu to +* let the waker make forward progress. This is safe because IRQs are +* disabled and the IPI will deliver after on_cpu is cleared. +* +* Ensure we load task_cpu(p) after p->on_cpu: +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) +* STORE p->on_cpu = 1 LOAD p->cpu +* +* to ensure we observe the correct CPU on which the task is currently +* scheduling. +*/ +if (smp_load_acquire (&p->on_cpu) && +ttwu_queue_wakelist (p,task_cpu (p), wake_flags)) +break ; -/* -* If the owning (remote) CPU is still in the middle of schedule() with -* this task as prev, wait until it's done referencing the task. -* -* Pairs with the smp_store_release() in finish_task(). -* -* This ensures that tasks getting woken will be fully ordered against -* their previous state and preserve Program Order. -*/ -smp_cond_load_acquire (&p->on_cpu, !VAL ); +/* +* If the owning (remote) CPU is still in the middle of schedule() with +* this task as prev, wait until it's done referencing the task. +* +* Pairs with the smp_store_release() in finish_task(). +* +* This ensures that tasks getting woken will be fully ordered against +* their previous state and preserve Program Order. +*/ +smp_cond_load_acquire (&p->on_cpu, !VAL ); - cpu =select_task_rq (p, p->wake_cpu, wake_flags |WF_TTWU ); -if (task_cpu (p) != cpu) { -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } + cpu =select_task_rq (p, p->wake_cpu, wake_flags |WF_TTWU ); +if (task_cpu (p) != cpu) { +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } - wake_flags |=WF_MIGRATED ; -psi_ttwu_dequeue (p); -set_task_cpu (p, cpu); - } + wake_flags |=WF_MIGRATED ; +psi_ttwu_dequeue (p); +set_task_cpu (p, cpu); + }#else - cpu =task_cpu (p); + cpu =task_cpu (p);#endif /* CONFIG_SMP */ -ttwu_queue (p, cpu, wake_flags); -unlock: -raw_spin_unlock_irqrestore (&p->pi_lock, flags); +ttwu_queue (p, cpu, wake_flags); + } out:if (success)ttwu_stat (p,task_cpu (p), wake_flags); -preempt_enable ();return success; }
Valentin Schneider
reviewerRe: [PATCH 5/9] sched: Simplify ttwu()
Quoted 10 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 221 +++++++++++++++++++++++++--------------------------- 1 file changed, 109 insertions(+), 112 deletions(-)
kernel/sched/core.c ttwu_stat(struct task_struct *p, int cpu · line 3710
Quoted 6 lines
kernel/sched/core.c +2
-1@@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpustruct sched_domain *sd;__schedstat_inc (p->stats.nr_wakeups_remote); -rcu_read_lock (); + +guard (rcu)();
This isn't strictly equivalent, right? AFAICT that pushes the rcu_read_unlock() further down than it currently is - not a big deal, but indentation aside scoped_guard() would preserve that.
Peter Zijlstra
maintainerRe: [PATCH 5/9] sched: Simplify ttwu()
Quoted 13 lines · Valentin Schneider
Quoted 12 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 221 +++++++++++++++++++++++++--------------------------- 1 file changed, 109 insertions(+), 112 deletions(-)kernel/sched/core.c +0
-0@@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpustruct sched_domain *sd;
Quoted 8 lines
kernel/sched/core.c ttwu_stat(struct task_struct *p, int cpu · line 3710
Quoted 4 lines
kernel/sched/core.c +2
-1__schedstat_inc (p->stats.nr_wakeups_remote); -rcu_read_lock (); + +guard (rcu)();This isn't strictly equivalent, right? AFAICT that pushes the
rcu_read_unlock()further down than it currently is - not a big deal, but indentation asidescoped_guard()would preserve that.
The full hunk:
| @@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpu
| struct sched_domain *sd;
|
| __schedstat_inc(p->stats.nr_wakeups_remote);
| - rcu_read_lock();
| +
| + guard(rcu)();
| for_each_domain(rq->cpu, sd) {
| if (cpumask_test_cpu(cpu, sched_domain_span(sd))) {
| __schedstat_inc(sd->ttwu_wake_remote);
| break;
| }
| }
| - rcu_read_unlock();
| }And you'll see the guard goes out of scope here ^
Which is the exact place rcu_read_unlock() was at, no?
|
| if (wake_flags & WF_MIGRATED)
Valentin Schneider
reviewerRe: [PATCH 5/9] sched: Simplify ttwu()
Quoted 41 lines · Peter Zijlstra
Quoted 13 lines · Valentin Schneider
Quoted 12 lines · Peter Zijlstra
Use guards to reduce gotos and simplify control flow.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- kernel/sched/core.c | 221 +++++++++++++++++++++++++--------------------------- 1 file changed, 109 insertions(+), 112 deletions(-)kernel/sched/core.c +0
-0@@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpustruct sched_domain *sd;Quoted 8 lines
kernel/sched/core.c ttwu_stat(struct task_struct *p, int cpu · line 3710
Quoted 4 lines
kernel/sched/core.c +2
-1__schedstat_inc (p->stats.nr_wakeups_remote); -rcu_read_lock (); + +guard (rcu)();This isn't strictly equivalent, right? AFAICT that pushes the
rcu_read_unlock()further down than it currently is - not a big deal, but indentation asidescoped_guard()would preserve that.The full hunk:
| @@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpu | struct sched_domain *sd; | | __schedstat_inc(p->stats.nr_wakeups_remote); | - rcu_read_lock(); | + | + guard(rcu)(); | for_each_domain(rq->cpu, sd) { | if (cpumask_test_cpu(cpu, sched_domain_span(sd))) { | __schedstat_inc(sd->ttwu_wake_remote); | break; | } | } | - rcu_read_unlock(); | }
Quoted 3 lines
And you'll see the guard goes out of scope here ^
Which is the exact place
rcu_read_unlock()was at, no?
Bleh, yes, lost track of the scope there...
[tip: sched/core] sched: Simplify ttwu()
In reply to Peter ZijlstraThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 857d315f1201cfcf60e5849c96d2b4dd20f90ebf Gitweb: https://git.kernel.org/tip/857d315f1201cfcf60e5849c96d2b4dd20f90ebf Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:26 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:25 +02:00
sched: Simplify ttwu()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 221 +++++++++++++++++++++---------------------- 1 file changed, 109 insertions(+), 112 deletions(-)
kernel/sched/core.c +109 -112
@@ -3733,14 +3733,14 @@ ttwu_stat(struct task_struct *p, int cpu, int wake_flags)struct sched_domain *sd;__schedstat_inc (p->stats.nr_wakeups_remote); -rcu_read_lock (); + +guard (rcu)();for_each_domain (rq->cpu, sd) {if (cpumask_test_cpu (cpu,sched_domain_span (sd))) {__schedstat_inc (sd->ttwu_wake_remote);break ; } } -rcu_read_unlock (); }if (wake_flags &WF_MIGRATED )
@@ -4199,10 +4199,9 @@ bool ttwu_state_match(struct task_struct *p, unsigned int state, int *success)static int try_to_wake_up (struct task_struct *p,unsigned int state,int wake_flags) { -unsigned long flags; +guard (preempt)();int cpu, success =0 ; -preempt_disable ();if (p == current) {/* * We're waking current, this means 'p->on_rq' and 'task_cpu(p)
@@ -4229,129 +4228,127 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)* reordered with p->state check below. This pairs with smp_store_mb() * in set_current_state() that the waiting thread does. */ -raw_spin_lock_irqsave (&p->pi_lock, flags); -smp_mb__after_spinlock (); -if (!ttwu_state_match (p, state, &success)) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { +smp_mb__after_spinlock (); +if (!ttwu_state_match (p, state, &success)) +break ; -trace_sched_waking (p); +trace_sched_waking (p); -/* -* Ensure we load p->on_rq _after_ p->state, otherwise it would -* be possible to, falsely, observe p->on_rq == 0 and get stuck -* in smp_cond_load_acquire() below. -* -* sched_ttwu_pending() try_to_wake_up() -* STORE p->on_rq = 1 LOAD p->state -* UNLOCK rq->lock -* -* __schedule() (switch to task 'p') -* LOCK rq->lock smp_rmb(); -* smp_mb__after_spinlock(); -* UNLOCK rq->lock -* -* [task p] -* STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq -* -* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in -* __schedule(). See the comment for smp_mb__after_spinlock(). -* -* A similar smb_rmb() lives in try_invoke_on_locked_down_task(). -*/ -smp_rmb (); -if (READ_ONCE (p->on_rq) &&ttwu_runnable (p, wake_flags)) - goto unlock; +/* +* Ensure we load p->on_rq _after_ p->state, otherwise it would +* be possible to, falsely, observe p->on_rq == 0 and get stuck +* in smp_cond_load_acquire() below. +* +* sched_ttwu_pending() try_to_wake_up() +* STORE p->on_rq = 1 LOAD p->state +* UNLOCK rq->lock +* +* __schedule() (switch to task 'p') +* LOCK rq->lock smp_rmb(); +* smp_mb__after_spinlock(); +* UNLOCK rq->lock +* +* [task p] +* STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq +* +* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in +* __schedule(). See the comment for smp_mb__after_spinlock(). +* +* A similar smb_rmb() lives in try_invoke_on_locked_down_task(). +*/ +smp_rmb (); +if (READ_ONCE (p->on_rq) &&ttwu_runnable (p, wake_flags)) +break ;#ifdef CONFIG_SMP -/* -* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be -* possible to, falsely, observe p->on_cpu == 0. -* -* One must be running (->on_cpu == 1) in order to remove oneself -* from the runqueue. -* -* __schedule() (switch to task 'p') try_to_wake_up() -* STORE p->on_cpu = 1 LOAD p->on_rq -* UNLOCK rq->lock -* -* __schedule() (put 'p' to sleep) -* LOCK rq->lock smp_rmb(); -* smp_mb__after_spinlock(); -* STORE p->on_rq = 0 LOAD p->on_cpu -* -* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in -* __schedule(). See the comment for smp_mb__after_spinlock(). -* -* Form a control-dep-acquire with p->on_rq == 0 above, to ensure -* schedule()'s deactivate_task() has 'happened' and p will no longer -* care about it's own p->state. See the comment in __schedule(). -*/ -smp_acquire__after_ctrl_dep (); +/* +* Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be +* possible to, falsely, observe p->on_cpu == 0. +* +* One must be running (->on_cpu == 1) in order to remove oneself +* from the runqueue. +* +* __schedule() (switch to task 'p') try_to_wake_up() +* STORE p->on_cpu = 1 LOAD p->on_rq +* UNLOCK rq->lock +* +* __schedule() (put 'p' to sleep) +* LOCK rq->lock smp_rmb(); +* smp_mb__after_spinlock(); +* STORE p->on_rq = 0 LOAD p->on_cpu +* +* Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in +* __schedule(). See the comment for smp_mb__after_spinlock(). +* +* Form a control-dep-acquire with p->on_rq == 0 above, to ensure +* schedule()'s deactivate_task() has 'happened' and p will no longer +* care about it's own p->state. See the comment in __schedule(). +*/ +smp_acquire__after_ctrl_dep (); -/* -* We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq -* == 0), which means we need to do an enqueue, change p->state to -* TASK_WAKING such that we can unlock p->pi_lock before doing the -* enqueue, such as ttwu_queue_wakelist(). -*/ -WRITE_ONCE (p->__state,TASK_WAKING ); +/* +* We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq +* == 0), which means we need to do an enqueue, change p->state to +* TASK_WAKING such that we can unlock p->pi_lock before doing the +* enqueue, such as ttwu_queue_wakelist(). +*/ +WRITE_ONCE (p->__state,TASK_WAKING ); -/* -* If the owning (remote) CPU is still in the middle of schedule() with -* this task as prev, considering queueing p on the remote CPUs wake_list -* which potentially sends an IPI instead of spinning on p->on_cpu to -* let the waker make forward progress. This is safe because IRQs are -* disabled and the IPI will deliver after on_cpu is cleared. -* -* Ensure we load task_cpu(p) after p->on_cpu: -* -* set_task_cpu(p, cpu); -* STORE p->cpu = @cpu -* __schedule() (switch to task 'p') -* LOCK rq->lock -* smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) -* STORE p->on_cpu = 1 LOAD p->cpu -* -* to ensure we observe the correct CPU on which the task is currently -* scheduling. -*/ -if (smp_load_acquire (&p->on_cpu) && -ttwu_queue_wakelist (p,task_cpu (p), wake_flags)) - goto unlock; +/* +* If the owning (remote) CPU is still in the middle of schedule() with +* this task as prev, considering queueing p on the remote CPUs wake_list +* which potentially sends an IPI instead of spinning on p->on_cpu to +* let the waker make forward progress. This is safe because IRQs are +* disabled and the IPI will deliver after on_cpu is cleared. +* +* Ensure we load task_cpu(p) after p->on_cpu: +* +* set_task_cpu(p, cpu); +* STORE p->cpu = @cpu +* __schedule() (switch to task 'p') +* LOCK rq->lock +* smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) +* STORE p->on_cpu = 1 LOAD p->cpu +* +* to ensure we observe the correct CPU on which the task is currently +* scheduling. +*/ +if (smp_load_acquire (&p->on_cpu) && +ttwu_queue_wakelist (p,task_cpu (p), wake_flags)) +break ; -/* -* If the owning (remote) CPU is still in the middle of schedule() with -* this task as prev, wait until it's done referencing the task. -* -* Pairs with the smp_store_release() in finish_task(). -* -* This ensures that tasks getting woken will be fully ordered against -* their previous state and preserve Program Order. -*/ -smp_cond_load_acquire (&p->on_cpu, !VAL ); +/* +* If the owning (remote) CPU is still in the middle of schedule() with +* this task as prev, wait until it's done referencing the task. +* +* Pairs with the smp_store_release() in finish_task(). +* +* This ensures that tasks getting woken will be fully ordered against +* their previous state and preserve Program Order. +*/ +smp_cond_load_acquire (&p->on_cpu, !VAL ); - cpu =select_task_rq (p, p->wake_cpu, wake_flags |WF_TTWU ); -if (task_cpu (p) != cpu) { -if (p->in_iowait) { -delayacct_blkio_end (p); -atomic_dec (&task_rq (p)->nr_iowait); - } + cpu =select_task_rq (p, p->wake_cpu, wake_flags |WF_TTWU ); +if (task_cpu (p) != cpu) { +if (p->in_iowait) { +delayacct_blkio_end (p); +atomic_dec (&task_rq (p)->nr_iowait); + } - wake_flags |=WF_MIGRATED ; -psi_ttwu_dequeue (p); -set_task_cpu (p, cpu); - } + wake_flags |=WF_MIGRATED ; +psi_ttwu_dequeue (p); +set_task_cpu (p, cpu); + }#else - cpu =task_cpu (p); + cpu =task_cpu (p);#endif /* CONFIG_SMP */ -ttwu_queue (p, cpu, wake_flags); -unlock: -raw_spin_unlock_irqrestore (&p->pi_lock, flags); +ttwu_queue (p, cpu, wake_flags); + } out:if (success)ttwu_stat (p,task_cpu (p), wake_flags); -preempt_enable ();return success; }
Peter Zijlstra
maintainerPATCH 6/9 sched: Simplify sched_exec()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
kernel/sched/core.c +9 -12
@@ -5431,23 +5431,20 @@ unsigned int nr_iowait(void)void sched_exec (void ) {struct task_struct *p = current; -unsigned long flags; +struct migration_arg arg;int dest_cpu; -raw_spin_lock_irqsave (&p->pi_lock, flags); - dest_cpu = p->sched_class->select_task_rq (p,task_cpu (p),WF_EXEC ); -if (dest_cpu ==smp_processor_id ()) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + dest_cpu = p->sched_class->select_task_rq (p,task_cpu (p),WF_EXEC ); +if (dest_cpu ==smp_processor_id ()) +return ; -if (likely (cpu_active (dest_cpu))) { -struct migration_arg arg = { p, dest_cpu }; +if (unlikely (!cpu_active (dest_cpu))) +return ; -raw_spin_unlock_irqrestore (&p->pi_lock, flags); -stop_one_cpu (task_cpu (p), migration_cpu_stop, &arg); -return ; + arg = (struct migration_arg ){ p, dest_cpu }; } -unlock: -raw_spin_unlock_irqrestore (&p->pi_lock, flags); +stop_one_cpu (task_cpu (p), migration_cpu_stop, &arg); }#endif
[tip: sched/core] sched: Simplify sched_exec()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 4bdada79f3464d85f6e187213c088e7c934e0554 Gitweb: https://git.kernel.org/tip/4bdada79f3464d85f6e187213c088e7c934e0554 Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:27 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:26 +02:00
sched: Simplify sched_exec()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
kernel/sched/core.c +9 -12
@@ -5498,23 +5498,20 @@ unsigned int nr_iowait(void)void sched_exec (void ) {struct task_struct *p = current; -unsigned long flags; +struct migration_arg arg;int dest_cpu; -raw_spin_lock_irqsave (&p->pi_lock, flags); - dest_cpu = p->sched_class->select_task_rq (p,task_cpu (p),WF_EXEC ); -if (dest_cpu ==smp_processor_id ()) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + dest_cpu = p->sched_class->select_task_rq (p,task_cpu (p),WF_EXEC ); +if (dest_cpu ==smp_processor_id ()) +return ; -if (likely (cpu_active (dest_cpu))) { -struct migration_arg arg = { p, dest_cpu }; +if (unlikely (!cpu_active (dest_cpu))) +return ; -raw_spin_unlock_irqrestore (&p->pi_lock, flags); -stop_one_cpu (task_cpu (p), migration_cpu_stop, &arg); -return ; + arg = (struct migration_arg ){ p, dest_cpu }; } -unlock: -raw_spin_unlock_irqrestore (&p->pi_lock, flags); +stop_one_cpu (task_cpu (p), migration_cpu_stop, &arg); }#endif
Peter Zijlstra
maintainerPATCH 7/9 sched: Simplify sched_tick_remote()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 43 ++++++++++++++++++------------------------- 1 file changed, 18 insertions(+), 25 deletions(-)
kernel/sched/core.c +18 -25
@@ -5651,9 +5651,6 @@ static void sched_tick_remote(struct wor struct tick_work *twork = container_of(dwork, struct tick_work, work);int cpu = twork->cpu;struct rq *rq =cpu_rq (cpu); -struct task_struct *curr; -struct rq_flags rf; -u64 delta;int os;/*
@@ -5663,30 +5660,26 @@ static void sched_tick_remote(struct wor* statistics and checks timeslices in a time-independent way, regardless * of when exactly it is running. */ -if (!tick_nohz_tick_stopped_cpu (cpu)) - goto out_requeue; +if (tick_nohz_tick_stopped_cpu (cpu)) { +guard (rq_lock_irq)(rq); +struct task_struct *curr = rq->curr; + +if (cpu_online (cpu)) { +update_rq_clock (rq); + +if (!is_idle_task (curr)) { +/* +* Make sure the next tick runs within a +* reasonable amount of time. +*/ +u64 delta =rq_clock_task (rq) - curr->se.exec_start; +WARN_ON_ONCE (delta > (u64 )NSEC_PER_SEC *3 ); + } + curr->sched_class->task_tick (rq, curr,0 ); -rq_lock_irq (rq, &rf); - curr = rq->curr; -if (cpu_is_offline (cpu)) - goto out_unlock; - -update_rq_clock (rq); - -if (!is_idle_task (curr)) { -/* -* Make sure the next tick runs within a reasonable -* amount of time. -*/ - delta =rq_clock_task (rq) - curr->se.exec_start; -WARN_ON_ONCE (delta > (u64 )NSEC_PER_SEC *3 ); +calc_load_nohz_remote (rq); + } } - curr->sched_class->task_tick (rq, curr,0 ); - -calc_load_nohz_remote (rq); -out_unlock: -rq_unlock_irq (rq, &rf); -out_requeue:/* * Run the remote tick once per second (1Hz). This arbitrary
[tip: sched/core] sched: Simplify sched_tick_remote()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 6dafc713e3b0d8ffbd696d200d8c9dd212ddcdfc Gitweb: https://git.kernel.org/tip/6dafc713e3b0d8ffbd696d200d8c9dd212ddcdfc Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:28 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:26 +02:00
sched: Simplify sched_tick_remote()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 39 ++++++++++++++++----------------------- 1 file changed, 16 insertions(+), 23 deletions(-)
kernel/sched/core.c +16 -23
@@ -5721,9 +5721,6 @@ static void sched_tick_remote(struct work_struct *work) struct tick_work *twork = container_of(dwork, struct tick_work, work);int cpu = twork->cpu;struct rq *rq =cpu_rq (cpu); -struct task_struct *curr; -struct rq_flags rf; -u64 delta;int os;/*
@@ -5733,30 +5730,26 @@ static void sched_tick_remote(struct work_struct *work)* statistics and checks timeslices in a time-independent way, regardless * of when exactly it is running. */ -if (!tick_nohz_tick_stopped_cpu (cpu)) - goto out_requeue; +if (tick_nohz_tick_stopped_cpu (cpu)) { +guard (rq_lock_irq)(rq); +struct task_struct *curr = rq->curr; -rq_lock_irq (rq, &rf); - curr = rq->curr; -if (cpu_is_offline (cpu)) - goto out_unlock; +if (cpu_online (cpu)) { +update_rq_clock (rq); -update_rq_clock (rq); +if (!is_idle_task (curr)) { +/* +* Make sure the next tick runs within a +* reasonable amount of time. +*/ +u64 delta =rq_clock_task (rq) - curr->se.exec_start; +WARN_ON_ONCE (delta > (u64 )NSEC_PER_SEC *3 ); + } + curr->sched_class->task_tick (rq, curr,0 ); -if (!is_idle_task (curr)) { -/* -* Make sure the next tick runs within a reasonable -* amount of time. -*/ - delta =rq_clock_task (rq) - curr->se.exec_start; -WARN_ON_ONCE (delta > (u64 )NSEC_PER_SEC *3 ); +calc_load_nohz_remote (rq); + } } - curr->sched_class->task_tick (rq, curr,0 ); - -calc_load_nohz_remote (rq); -out_unlock: -rq_unlock_irq (rq, &rf); -out_requeue:/* * Run the remote tick once per second (1Hz). This arbitrary
Peter Zijlstra
maintainerPATCH 8/9 sched: Simplify try_steal_cookie()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
kernel/sched/core.c +9 -12
@@ -6229,19 +6229,19 @@ static bool try_steal_cookie(int this, iunsigned long cookie;bool success = false; -local_irq_disable (); -double_rq_lock (dst, src); +guard (irq)(); +guard (double_rq_lock)(dst, src); cookie = dst->core->core_cookie;if (!cookie) - goto unlock; +return false;if (dst->curr != dst->idle) - goto unlock; +return false; p =sched_core_find (src, cookie);if (!p) - goto unlock; +return false; do { if (p == src->core_pick || p == src->curr)
@@ -6253,9 +6253,10 @@ static bool try_steal_cookie(int this, iif (p->core_occupation > dst->idle->core_occupation) goto next;/* -* sched_core_find() and sched_core_next() will ensure that task @p -* is not throttled now, we also need to check whether the runqueue -* of the destination CPU is being throttled. +* sched_core_find() and sched_core_next() will ensure +* that task @p is not throttled now, we also need to +* check whether the runqueue of the destination CPU is +* being throttled. */ if (sched_task_is_throttled (p, this)) goto next;
@@ -6273,10 +6274,6 @@ static bool try_steal_cookie(int this, i p =sched_core_next (p, cookie); }while (p); -unlock: -double_rq_unlock (dst, src); -local_irq_enable (); -return success; }
[tip: sched/core] sched: Simplify try_steal_cookie()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: b4e1fa1e14286f7a825b10d8ebb2e9c0f77c241b Gitweb: https://git.kernel.org/tip/b4e1fa1e14286f7a825b10d8ebb2e9c0f77c241b Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:29 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:27 +02:00
sched: Simplify try_steal_cookie()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
kernel/sched/core.c +9 -12
@@ -6298,19 +6298,19 @@ static bool try_steal_cookie(int this, int that)unsigned long cookie;bool success = false; -local_irq_disable (); -double_rq_lock (dst, src); +guard (irq)(); +guard (double_rq_lock)(dst, src); cookie = dst->core->core_cookie;if (!cookie) - goto unlock; +return false;if (dst->curr != dst->idle) - goto unlock; +return false; p =sched_core_find (src, cookie);if (!p) - goto unlock; +return false; do { if (p == src->core_pick || p == src->curr)
@@ -6322,9 +6322,10 @@ static bool try_steal_cookie(int this, int that)if (p->core_occupation > dst->idle->core_occupation) goto next;/* -* sched_core_find() and sched_core_next() will ensure that task @p -* is not throttled now, we also need to check whether the runqueue -* of the destination CPU is being throttled. +* sched_core_find() and sched_core_next() will ensure +* that task @p is not throttled now, we also need to +* check whether the runqueue of the destination CPU is +* being throttled. */ if (sched_task_is_throttled (p, this)) goto next;
@@ -6342,10 +6343,6 @@ next: p =sched_core_next (p, cookie); }while (p); -unlock: -double_rq_unlock (dst, src); -local_irq_enable (); -return success; }
Peter Zijlstra
maintainerPATCH 9/9 sched: Simplify sched_core_cpu_{starting,deactivate}()
In reply to Peter ZijlstraUse guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-)
kernel/sched/core.c +12 -15
@@ -6331,20 +6331,24 @@ static void queue_core_balance(struct rqqueue_balance_callback (rq, &per_cpu (core_balance_head, rq->cpu), sched_core_balance); } +DEFINE_LOCK_GUARD_1 (core_lock, int, +sched_core_lock (*_T->lock, &_T->flags), +sched_core_unlock (*_T->lock, &_T->flags), + unsigned long flags) +static void sched_core_cpu_starting (unsigned int cpu) {const struct cpumask *smt_mask =cpu_smt_mask (cpu);struct rq *rq =cpu_rq (cpu), *core_rq =NULL ; -unsigned long flags;int t; -sched_core_lock (cpu, &flags); +guard (core_lock)(&cpu);WARN_ON_ONCE (rq->core != rq);/* if we're the first, we'll be our own leader */ if (cpumask_weight (smt_mask) ==1 ) - goto unlock; +return ;/* find the leader */ for_each_cpu (t, smt_mask) {
@@ -6358,7 +6362,7 @@ static void sched_core_cpu_starting(unsi }if (WARN_ON_ONCE (!core_rq))/* whoopsie */ - goto unlock; +return ;/* install and validate core_rq */ for_each_cpu (t, smt_mask) {
@@ -6369,29 +6373,25 @@ static void sched_core_cpu_starting(unsiWARN_ON_ONCE (rq->core != core_rq); } - -unlock: -sched_core_unlock (cpu, &flags); }static void sched_core_cpu_deactivate (unsigned int cpu) {const struct cpumask *smt_mask =cpu_smt_mask (cpu);struct rq *rq =cpu_rq (cpu), *core_rq =NULL ; -unsigned long flags;int t; -sched_core_lock (cpu, &flags); +guard (core_lock)(&cpu);/* if we're the last man standing, nothing to do */ if (cpumask_weight (smt_mask) ==1 ) {WARN_ON_ONCE (rq->core != rq); - goto unlock; +return ; }/* if we're not the leader, nothing to do */ if (rq->core != rq) - goto unlock; +return ;/* find a new leader */ for_each_cpu (t, smt_mask) {
@@ -6402,7 +6402,7 @@ static void sched_core_cpu_deactivate(un }if (WARN_ON_ONCE (!core_rq))/* impossible */ - goto unlock; +return ;/* copy the shared state to the new leader */ core_rq->core_task_seq = rq->core_task_seq;
@@ -6424,9 +6424,6 @@ static void sched_core_cpu_deactivate(un rq =cpu_rq (t); rq->core = core_rq; } - -unlock: -sched_core_unlock (cpu, &flags); }static inline void sched_core_cpu_dying (unsigned int cpu)
[tip: sched/core] sched: Simplify sched_core_cpu_{starting,deactivate}()
The following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: 7170509cadbb76e5fa7d7b090d2cbdb93d56a2de Gitweb: https://git.kernel.org/tip/7170509cadbb76e5fa7d7b090d2cbdb93d56a2de Author: Peter Zijlstra <[email protected]> AuthorDate: Tue, 01 Aug 2023 22:41:30 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 14 Aug 2023 17:01:27 +02:00
sched: Simplify sched_core_cpu_{starting,deactivate}()
Use guards to reduce gotos and simplify control flow.
--- kernel/sched/core.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-)
kernel/sched/core.c +12 -15
@@ -6400,20 +6400,24 @@ static void queue_core_balance(struct rq *rq)queue_balance_callback (rq, &per_cpu (core_balance_head, rq->cpu), sched_core_balance); } +DEFINE_LOCK_GUARD_1 (core_lock, int, +sched_core_lock (*_T->lock, &_T->flags), +sched_core_unlock (*_T->lock, &_T->flags), + unsigned long flags) +static void sched_core_cpu_starting (unsigned int cpu) {const struct cpumask *smt_mask =cpu_smt_mask (cpu);struct rq *rq =cpu_rq (cpu), *core_rq =NULL ; -unsigned long flags;int t; -sched_core_lock (cpu, &flags); +guard (core_lock)(&cpu);WARN_ON_ONCE (rq->core != rq);/* if we're the first, we'll be our own leader */ if (cpumask_weight (smt_mask) ==1 ) - goto unlock; +return ;/* find the leader */ for_each_cpu (t, smt_mask) {
@@ -6427,7 +6431,7 @@ static void sched_core_cpu_starting(unsigned int cpu) }if (WARN_ON_ONCE (!core_rq))/* whoopsie */ - goto unlock; +return ;/* install and validate core_rq */ for_each_cpu (t, smt_mask) {
@@ -6438,29 +6442,25 @@ static void sched_core_cpu_starting(unsigned int cpu)WARN_ON_ONCE (rq->core != core_rq); } - -unlock: -sched_core_unlock (cpu, &flags); }static void sched_core_cpu_deactivate (unsigned int cpu) {const struct cpumask *smt_mask =cpu_smt_mask (cpu);struct rq *rq =cpu_rq (cpu), *core_rq =NULL ; -unsigned long flags;int t; -sched_core_lock (cpu, &flags); +guard (core_lock)(&cpu);/* if we're the last man standing, nothing to do */ if (cpumask_weight (smt_mask) ==1 ) {WARN_ON_ONCE (rq->core != rq); - goto unlock; +return ; }/* if we're not the leader, nothing to do */ if (rq->core != rq) - goto unlock; +return ;/* find a new leader */ for_each_cpu (t, smt_mask) {
@@ -6471,7 +6471,7 @@ static void sched_core_cpu_deactivate(unsigned int cpu) }if (WARN_ON_ONCE (!core_rq))/* impossible */ - goto unlock; +return ;/* copy the shared state to the new leader */ core_rq->core_task_seq = rq->core_task_seq;
@@ -6493,9 +6493,6 @@ static void sched_core_cpu_deactivate(unsigned int cpu) rq =cpu_rq (t); rq->core = core_rq; } - -unlock: -sched_core_unlock (cpu, &flags); }static inline void sched_core_cpu_dying (unsigned int cpu)
Quoted 3 lines · Peter Zijlstra
If there are no objections / comments, I'm aiming to post the second batch at the end of the week.
Other than the nitpick on 5/9, the conversions LGTM.