Skip to contentSkip to conversation

Fix the scheduler-IPI mess.

RFC PATCH Applied Peter Zijlstra 14 participants 66 messages May 26, 2020 – Jan 22, 2021

Subsystems (2)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover Fix the scheduler-IPI mess.

Hi!

So commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") is pretty buggered, and I'm afraid this is the 'prettiest' pile of puke I could come up with.

It boots and builds a kernel, but it already did that for me.

I do think there's definite room for further integration of smp_function_call and irq_work, but figured this was quite enough for now.

Please look hard, there be dragons here, I'm sure.

author maintainer
#1 lore raw

RFC PATCH 1/7 sched: Fix smp_call_function_single_async() usage for ILB

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

Fixes 90b5363acd47 ("sched: Clean up scheduler_ipi()")
---
 include/linux/sched.h      |    4 +
 include/linux/sched/idle.h |   55 ++++++++++++++++++
 kernel/sched/core.c        |  132 +++++++++------------------------------------
 kernel/sched/fair.c        |   18 ++----
 kernel/sched/idle.c        |    3 -
 kernel/sched/sched.h       |    2 
 kernel/smp.c               |    7 ++
 7 files changed, 102 insertions(+), 119 deletions(-)
kernel/sched/core.c +10 -26
@@ -637,41 +568,25 @@ void wake_up_nohz_cpu(int cpu)
 		wake_up_idle_cpu(cpu);
 }
 
-static inline bool got_nohz_idle_kick(void)
+static void nohz_csd_func(void *info)
 {
-	int cpu = smp_processor_id();
-
-	if (!(atomic_read(nohz_flags(cpu)) & NOHZ_KICK_MASK))
-		return false;
-
-	if (idle_cpu(cpu) && !need_resched())
-		return true;
+	struct rq *rq = info;
+	int cpu = cpu_of(rq);
+	unsigned int flags;
 
 	/*
-	 * We can't run Idle Load Balance on this CPU for this time so we
-	 * cancel it and clear NOHZ_BALANCE_KICK
+	 * Release the rq::nohz_csd.
 	 */
-	atomic_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));
-	return false;
-}
-
-static void nohz_csd_func(void *info)
-{
-	struct rq *rq = info;
+	flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));
+	WARN_ON(!(flags & NOHZ_KICK_MASK));
 
-	if (got_nohz_idle_kick()) {
-		rq->idle_balance = 1;
+	rq->idle_balance = idle_cpu(cpu);
+	if (rq->idle_balance && !need_resched()) {
+		rq->nohz_idle_balance = flags;
 		raise_softirq_irqoff(SCHED_SOFTIRQ);
 	}
 }
 
-#else /* CONFIG_NO_HZ_COMMON */
-
-static inline bool got_nohz_idle_kick(void)
-{
-	return false;
-}
-
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
kernel/sched/fair.c +8 -10
@@ -10024,6 +10024,10 @@ static void kick_ilb(unsigned int flags)
 	if (ilb_cpu >= nr_cpu_ids)
 		return;
 
+	/*
+	 * Access to rq::nohz_csd is serialized by NOHZ_KICK_MASK; he who sets
+	 * the first flag owns it; cleared by nohz_csd_func().
+	 */
 	flags = atomic_fetch_or(flags, nohz_flags(ilb_cpu));
 	if (flags & NOHZ_KICK_MASK)
 		return;
@@ -10371,20 +10375,14 @@ static bool _nohz_idle_balance(struct rq
  */
 static bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle)
 {
-	int this_cpu = this_rq->cpu;
-	unsigned int flags;
+	unsigned int flags = this_rq->nohz_idle_balance;
 
-	if (!(atomic_read(nohz_flags(this_cpu)) & NOHZ_KICK_MASK))
+	if (!flags)
 		return false;
 
-	if (idle != CPU_IDLE) {
-		atomic_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-		return false;
-	}
+	this_rq->nohz_idle_balance = 0;
 
-	/* could be _relaxed() */
-	flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-	if (!(flags & NOHZ_KICK_MASK))
+	if (idle != CPU_IDLE)
 		return false;
 
 	_nohz_idle_balance(this_rq, flags, idle);
kernel/sched/sched.h +1 -0
@@ -951,6 +951,7 @@ struct rq {
 
 	struct callback_head	*balance_callback;
 
+	unsigned char		nohz_idle_balance;
 	unsigned char		idle_balance;
 
 	unsigned long		misfit_task_load;
maintainer Reviewed-by on v1 1/7
#2 lore raw

Re: [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB

Quoted 11 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Quoted 7 lines

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

Fixes 90b5363acd47 ("sched: Clean up scheduler_ipi()")

Looks very good! Thanks!

maintainer Reviewed-by on v1 1/7
#3 lore raw

Re: [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB

In reply to Peter Zijlstra
Quoted 12 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Quoted 7 lines

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

Fixes 90b5363acd47 ("sched: Clean up scheduler_ipi()")
Quoted 33 lines
---
 include/linux/sched.h      |    4 +
 include/linux/sched/idle.h |   55 ++++++++++++++++++
 kernel/sched/core.c        |  132 +++++++++------------------------------------
 kernel/sched/fair.c        |   18 ++----
 kernel/sched/idle.c        |    3 -
 kernel/sched/sched.h       |    2
 kernel/smp.c               |    7 ++
 7 files changed, 102 insertions(+), 119 deletions(-)
kernel/sched/core.c +4 -10
@@ -637,41 +568,25 @@ void wake_up_nohz_cpu(int cpu)
                wake_up_idle_cpu(cpu);
 }
 
-static inline bool got_nohz_idle_kick(void)
+static void nohz_csd_func(void *info)
 {
-       int cpu = smp_processor_id();
-
-       if (!(atomic_read(nohz_flags(cpu)) & NOHZ_KICK_MASK))
-               return false;
-
-       if (idle_cpu(cpu) && !need_resched())
-               return true;
+       struct rq *rq = info;
+       int cpu = cpu_of(rq);
+       unsigned int flags;
 
        /*
-        * We can't run Idle Load Balance on this CPU for this time so we
-        * cancel it and clear NOHZ_BALANCE_KICK

kernel/sched/core.c void wake_up_nohz_cpu(int cpu) · line 580

Quoted 10 lines
kernel/sched/core.c +2 -7
+        * Release the rq::nohz_csd.
         */
-       atomic_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));
-       return false;
-}
-
-static void nohz_csd_func(void *info)
-{
-       struct rq *rq = info;
+       flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));

Why can't this be done in nohz_idle_balance() instead ?

you are not using flags in nohz_csd_func() and SCHED_SOFTIRQ which calls nohz_idle_balance(), happens after nohz_csd_func(), isn't it ?

In this case, you don't have to use the intermediate variable this_rq->nohz_idle_balance

Quoted 71 lines
kernel/sched/core.c +4 -9
+       WARN_ON(!(flags & NOHZ_KICK_MASK));
 
-       if (got_nohz_idle_kick()) {
-               rq->idle_balance = 1;
+       rq->idle_balance = idle_cpu(cpu);
+       if (rq->idle_balance && !need_resched()) {
+               rq->nohz_idle_balance = flags;
                raise_softirq_irqoff(SCHED_SOFTIRQ);
        }
 }
 
-#else /* CONFIG_NO_HZ_COMMON */
-
-static inline bool got_nohz_idle_kick(void)
-{
-       return false;
-}
-
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
kernel/sched/fair.c +8 -10
@@ -10024,6 +10024,10 @@ static void kick_ilb(unsigned int flags)
        if (ilb_cpu >= nr_cpu_ids)
                return;
 
+       /*
+        * Access to rq::nohz_csd is serialized by NOHZ_KICK_MASK; he who sets
+        * the first flag owns it; cleared by nohz_csd_func().
+        */
        flags = atomic_fetch_or(flags, nohz_flags(ilb_cpu));
        if (flags & NOHZ_KICK_MASK)
                return;
@@ -10371,20 +10375,14 @@ static bool _nohz_idle_balance(struct rq
  */
 static bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle)
 {
-       int this_cpu = this_rq->cpu;
-       unsigned int flags;
+       unsigned int flags = this_rq->nohz_idle_balance;
 
-       if (!(atomic_read(nohz_flags(this_cpu)) & NOHZ_KICK_MASK))
+       if (!flags)
                return false;
 
-       if (idle != CPU_IDLE) {
-               atomic_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-               return false;
-       }
+       this_rq->nohz_idle_balance = 0;
 
-       /* could be _relaxed() */
-       flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-       if (!(flags & NOHZ_KICK_MASK))
+       if (idle != CPU_IDLE)
                return false;
 
        _nohz_idle_balance(this_rq, flags, idle);
kernel/sched/sched.h +1 -0
@@ -951,6 +951,7 @@ struct rq {
 
        struct callback_head    *balance_callback;
 
+       unsigned char           nohz_idle_balance;
        unsigned char           idle_balance;
 
        unsigned long           misfit_task_load;
#4 lore raw

Re: [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB

Quoted 7 lines · Vincent Guittot

kernel/sched/core.c void wake_up_nohz_cpu(int cpu) · line 580

Quoted 4 lines
kernel/sched/core.c +1 -3
-static void nohz_csd_func(void *info)
-{
-       struct rq *rq = info;
+       flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));

Why can't this be done in nohz_idle_balance() instead ?

Quoted 5 lines

you are not using flags in nohz_csd_func() and SCHED_SOFTIRQ which calls nohz_idle_balance(), happens after nohz_csd_func(), isn't it ?

In this case, you don't have to use the intermediate variable this_rq->nohz_idle_balance

That's in fact to fix the original issue. The softirq was clearing
the nohz_flags but the softirq could be issued from two sources:
the tick and the IPI. And the tick source softirq could then clear
the flags set from the IPI sender before the IPI itself, resulting
in races such as described there: https://lore.kernel.org/lkml/20200521004035.GA15455@lenoir/

Thanks.

maintainer Reviewed-by
#5 lore raw

Re: [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB

Quoted 12 lines · Frederic Weisbecker
Quoted 10 lines · Vincent Guittot

kernel/sched/core.c void wake_up_nohz_cpu(int cpu) · line 580

Quoted 4 lines
kernel/sched/core.c +1 -3
-static void nohz_csd_func(void *info)
-{
-       struct rq *rq = info;
+       flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));

Why can't this be done in nohz_idle_balance() instead ?

you are not using flags in nohz_csd_func() and SCHED_SOFTIRQ which calls nohz_idle_balance(), happens after nohz_csd_func(), isn't it ?

Quoted 8 lines
Quoted 2 lines

In this case, you don't have to use the intermediate variable this_rq->nohz_idle_balance

That's in fact to fix the original issue. The softirq was clearing
the nohz_flags but the softirq could be issued from two sources:
the tick and the IPI. And the tick source softirq could then clear
the flags set from the IPI sender before the IPI itself, resulting
in races such as described there: https://lore.kernel.org/lkml/20200521004035.GA15455@lenoir/

ah yes, even if the cpu is idle, the tick can fire and clear it.

Quoted 2 lines

Thanks.

Reviewed-by on v1 1/7
#6 lore raw

Re: [RFC][PATCH 1/7] sched: Fix smp_call_function_single_async() usage for ILB

In reply to Peter Zijlstra
Quoted 11 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

I got confused the first time I read through that, and had the same thoughts as Vincent; reading Frederic's reply and the thread helped. Could we mention the tick softirq vs ilb kick race thing in the changelog?

Otherwise the change looks okay, I couldn't convince myself there were more gaps left to fill.

Quoted 7 lines

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

Fixes 90b5363acd47 ("sched: Clean up scheduler_ipi()")
merge bot Applied
#7 lore raw

[tip: sched/core] sched: Fix smp_call_function_single_async() usage for ILB

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     19a1f5ec699954d21be10f74ff71c2a7079e99ad
Gitweb:        https://git.kernel.org/tip/19a1f5ec699954d21be10f74ff71c2a7079e99ad
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:10:58 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

sched: Fix smp_call_function_single_async() usage for ILB

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

---
 kernel/sched/core.c  | 36 ++++++++++--------------------------
 kernel/sched/fair.c  | 18 ++++++++----------
 kernel/sched/sched.h |  1 +
 3 files changed, 19 insertions(+), 36 deletions(-)
kernel/sched/core.c +10 -26
index 95e457d..2cacc1e 100644
@@ -637,41 +637,25 @@ void wake_up_nohz_cpu(int cpu)
 		wake_up_idle_cpu(cpu);
 }
 
-static inline bool got_nohz_idle_kick(void)
+static void nohz_csd_func(void *info)
 {
-	int cpu = smp_processor_id();
-
-	if (!(atomic_read(nohz_flags(cpu)) & NOHZ_KICK_MASK))
-		return false;
-
-	if (idle_cpu(cpu) && !need_resched())
-		return true;
+	struct rq *rq = info;
+	int cpu = cpu_of(rq);
+	unsigned int flags;
 
 	/*
-	 * We can't run Idle Load Balance on this CPU for this time so we
-	 * cancel it and clear NOHZ_BALANCE_KICK
+	 * Release the rq::nohz_csd.
 	 */
-	atomic_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));
-	return false;
-}
+	flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(cpu));
+	WARN_ON(!(flags & NOHZ_KICK_MASK));
 
-static void nohz_csd_func(void *info)
-{
-	struct rq *rq = info;
-
-	if (got_nohz_idle_kick()) {
-		rq->idle_balance = 1;
+	rq->idle_balance = idle_cpu(cpu);
+	if (rq->idle_balance && !need_resched()) {
+		rq->nohz_idle_balance = flags;
 		raise_softirq_irqoff(SCHED_SOFTIRQ);
 	}
 }
 
-#else /* CONFIG_NO_HZ_COMMON */
-
-static inline bool got_nohz_idle_kick(void)
-{
-	return false;
-}
-
 #endif /* CONFIG_NO_HZ_COMMON */
 
 #ifdef CONFIG_NO_HZ_FULL
kernel/sched/fair.c +8 -10
index dda9b19..2890bd5 100644
@@ -10024,6 +10024,10 @@ static void kick_ilb(unsigned int flags)
 	if (ilb_cpu >= nr_cpu_ids)
 		return;
 
+	/*
+	 * Access to rq::nohz_csd is serialized by NOHZ_KICK_MASK; he who sets
+	 * the first flag owns it; cleared by nohz_csd_func().
+	 */
 	flags = atomic_fetch_or(flags, nohz_flags(ilb_cpu));
 	if (flags & NOHZ_KICK_MASK)
 		return;
@@ -10371,20 +10375,14 @@ abort:
  */
 static bool nohz_idle_balance(struct rq *this_rq, enum cpu_idle_type idle)
 {
-	int this_cpu = this_rq->cpu;
-	unsigned int flags;
+	unsigned int flags = this_rq->nohz_idle_balance;
 
-	if (!(atomic_read(nohz_flags(this_cpu)) & NOHZ_KICK_MASK))
+	if (!flags)
 		return false;
 
-	if (idle != CPU_IDLE) {
-		atomic_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-		return false;
-	}
+	this_rq->nohz_idle_balance = 0;
 
-	/* could be _relaxed() */
-	flags = atomic_fetch_andnot(NOHZ_KICK_MASK, nohz_flags(this_cpu));
-	if (!(flags & NOHZ_KICK_MASK))
+	if (idle != CPU_IDLE)
 		return false;
 
 	_nohz_idle_balance(this_rq, flags, idle);
kernel/sched/sched.h +1 -0
index 4b32cff..3c163cb 100644
@@ -951,6 +951,7 @@ struct rq {
 
 	struct callback_head	*balance_callback;
 
+	unsigned char		nohz_idle_balance;
 	unsigned char		idle_balance;
 
 	unsigned long		misfit_task_load;
#8 lore raw

Re: [tip: sched/core] sched: Fix smp_call_function_single_async() usage for ILB

Quoted 26 lines · On Mon, Jun 01, 2020 at 09:52:21AM -0000, tip-bot2 for Peter Zijlstra wrote:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     19a1f5ec699954d21be10f74ff71c2a7079e99ad
Gitweb:        https://git.kernel.org/tip/19a1f5ec699954d21be10f74ff71c2a7079e99ad
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:10:58 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

sched: Fix smp_call_function_single_async() usage for ILB

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in kick_ilb() got this wrong.

While on first reading kick_ilb() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Rework the nohz_idle_balance() trigger so that the release is in the IPI callback and thus guarantees the required serialization for the CSD.

Quoted 7 lines

Many patches in the series lack some Reviewed-by tags.

Thanks.

author maintainer
#9 lore raw

RFC PATCH 2/7 smp: Optimize flush_smp_call_function_queue()

In reply to Peter Zijlstra

The call_single_queue can contain (two) different callbacks, synchronous and asynchronous. The current interrupt handler runs them in-order, which means that remote CPUs that are waiting for their synchronous call can be delayed by running asynchronous callbacks.

Rework the interrupt handler to first run the synchonous callbacks.

---
 kernel/smp.c |   27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)
kernel/smp.c +23 -4
@@ -209,9 +209,9 @@ void generic_smp_call_function_single_in
  */
 static void flush_smp_call_function_queue(bool warn_cpu_offline)
 {
-	struct llist_head *head;
-	struct llist_node *entry;
 	call_single_data_t *csd, *csd_next;
+	struct llist_node *entry, *prev;
+	struct llist_head *head;
 	static bool warned;
 
 	lockdep_assert_irqs_disabled();
@@ -235,21 +235,40 @@ static void flush_smp_call_function_queu
 				csd->func);
 	}
 
+	/*
+	 * First; run all SYNC callbacks, people are waiting for us.
+	 */
+	prev = NULL;
 	llist_for_each_entry_safe(csd, csd_next, entry, llist) {
 		smp_call_func_t func = csd->func;
 		void *info = csd->info;
 
 		/* Do we wait until *after* callback? */
 		if (csd->flags & CSD_FLAG_SYNCHRONOUS) {
+			if (prev) {
+				prev->next = &csd_next->llist;
+			} else {
+				entry = &csd_next->llist;
+			}
 			func(info);
 			csd_unlock(csd);
 		} else {
-			csd_unlock(csd);
-			func(info);
+			prev = &csd->llist;
 		}
 	}
 
 	/*
+	 * Second; run all !SYNC callbacks.
+	 */
+	llist_for_each_entry_safe(csd, csd_next, entry, llist) {
+		smp_call_func_t func = csd->func;
+		void *info = csd->info;
+
+		csd_unlock(csd);
+		func(info);
+	}
+
+	/*
 	 * Handle irq works queued remotely by irq_work_queue_on().
 	 * Smp functions above are typically synchronous so they
 	 * better run first since some other CPUs may be busy waiting
maintainer Reviewed-by on v1 2/7
#10 lore raw

Re: [RFC][PATCH 2/7] smp: Optimize flush_smp_call_function_queue()

Quoted 8 lines · Peter Zijlstra

The call_single_queue can contain (two) different callbacks, synchronous and asynchronous. The current interrupt handler runs them in-order, which means that remote CPUs that are waiting for their synchronous call can be delayed by running asynchronous callbacks.

Rework the interrupt handler to first run the synchonous callbacks.

I hope we are not unlucky enough to have dependencies between some async and sync callbacks that require the whole execution in order.

(Sweet llist dance, I think I need fresh air and coffee now).

merge bot Applied
#11 lore raw

[tip: sched/core] smp: Optimize flush_smp_call_function_queue()

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     52103be07d8b08311955f8c30e535c2dda290cf4
Gitweb:        https://git.kernel.org/tip/52103be07d8b08311955f8c30e535c2dda290cf4
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:10:59 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

smp: Optimize flush_smp_call_function_queue()

The call_single_queue can contain (two) different callbacks, synchronous and asynchronous. The current interrupt handler runs them in-order, which means that remote CPUs that are waiting for their synchronous call can be delayed by running asynchronous callbacks.

Rework the interrupt handler to first run the synchonous callbacks.

---
 kernel/smp.c | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)
kernel/smp.c +23 -4
index 786092a..db2f738 100644
@@ -209,9 +209,9 @@ void generic_smp_call_function_single_interrupt(void)
  */
 static void flush_smp_call_function_queue(bool warn_cpu_offline)
 {
-	struct llist_head *head;
-	struct llist_node *entry;
 	call_single_data_t *csd, *csd_next;
+	struct llist_node *entry, *prev;
+	struct llist_head *head;
 	static bool warned;
 
 	lockdep_assert_irqs_disabled();
@@ -235,21 +235,40 @@ static void flush_smp_call_function_queue(bool warn_cpu_offline)
 				csd->func);
 	}
 
+	/*
+	 * First; run all SYNC callbacks, people are waiting for us.
+	 */
+	prev = NULL;
 	llist_for_each_entry_safe(csd, csd_next, entry, llist) {
 		smp_call_func_t func = csd->func;
 		void *info = csd->info;
 
 		/* Do we wait until *after* callback? */
 		if (csd->flags & CSD_FLAG_SYNCHRONOUS) {
+			if (prev) {
+				prev->next = &csd_next->llist;
+			} else {
+				entry = &csd_next->llist;
+			}
 			func(info);
 			csd_unlock(csd);
 		} else {
-			csd_unlock(csd);
-			func(info);
+			prev = &csd->llist;
 		}
 	}
 
 	/*
+	 * Second; run all !SYNC callbacks.
+	 */
+	llist_for_each_entry_safe(csd, csd_next, entry, llist) {
+		smp_call_func_t func = csd->func;
+		void *info = csd->info;
+
+		csd_unlock(csd);
+		func(info);
+	}
+
+	/*
 	 * Handle irq works queued remotely by irq_work_queue_on().
 	 * Smp functions above are typically synchronous so they
 	 * better run first since some other CPUs may be busy waiting
author maintainer
#12 lore raw

RFC PATCH 3/7 smp: Move irq_work_run() out of flush_smp_call_function_queue()

In reply to Peter Zijlstra

This ensures flush_smp_call_function_queue() is strictly about call_single_queue.

---
 kernel/smp.c |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
kernel/smp.c +9 -8
@@ -84,6 +84,7 @@ int smpcfd_dying_cpu(unsigned int cpu)
 	 * still pending.
 	 */
 	flush_smp_call_function_queue(false);
+	irq_work_run();
 	return 0;
 }
 
@@ -191,6 +192,14 @@ static int generic_exec_single(int cpu,
 void generic_smp_call_function_single_interrupt(void)
 {
 	flush_smp_call_function_queue(true);
+
+	/*
+	 * Handle irq works queued remotely by irq_work_queue_on().
+	 * Smp functions above are typically synchronous so they
+	 * better run first since some other CPUs may be busy waiting
+	 * for them.
+	 */
+	irq_work_run();
 }
 
 /**
@@ -267,14 +276,6 @@ static void flush_smp_call_function_queu
 		csd_unlock(csd);
 		func(info);
 	}
-
-	/*
-	 * Handle irq works queued remotely by irq_work_queue_on().
-	 * Smp functions above are typically synchronous so they
-	 * better run first since some other CPUs may be busy waiting
-	 * for them.
-	 */
-	irq_work_run();
 }
 
 /*
#13 lore raw

Re: [RFC][PATCH 3/7] smp: Move irq_work_run() out of flush_smp_call_function_queue()

Quoted 19 lines · Peter Zijlstra

This ensures flush_smp_call_function_queue() is strictly about call_single_queue.

---
 kernel/smp.c |   17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
kernel/smp.c +1 -0
@@ -84,6 +84,7 @@ int smpcfd_dying_cpu(unsigned int cpu)
 	 * still pending.
 	 */
 	flush_smp_call_function_queue(false);
+	irq_work_run();
 	return 0;
 }
 

kernel/smp.c static int generic_exec_single(int cpu, · line 201

Quoted 10 lines
kernel/smp.c +7 -0
@@ -191,6 +192,14 @@ static int generic_exec_single(int cpu,
 void generic_smp_call_function_single_interrupt(void)
 {
 	flush_smp_call_function_queue(true);
+
+	/*
+	 * Handle irq works queued remotely by irq_work_queue_on().
+	 * Smp functions above are typically synchronous so they
+	 * better run first since some other CPUs may be busy waiting
+	 * for them.
+	 */

You may want to update that comment once you merge remote irq_work and csd.

Thanks.

Quoted 21 lines
kernel/smp.c +1 -8
+	irq_work_run();
 }
 
 /**
@@ -267,14 +276,6 @@ static void flush_smp_call_function_queu
 		csd_unlock(csd);
 		func(info);
 	}
-
-	/*
-	 * Handle irq works queued remotely by irq_work_queue_on().
-	 * Smp functions above are typically synchronous so they
-	 * better run first since some other CPUs may be busy waiting
-	 * for them.
-	 */
-	irq_work_run();
 }
 
 /*
merge bot Applied
#14 lore raw

[tip: sched/core] smp: Move irq_work_run() out of flush_smp_call_function_queue()

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     afaa653c564da38c0b34c4baba31e88c46a8764c
Gitweb:        https://git.kernel.org/tip/afaa653c564da38c0b34c4baba31e88c46a8764c
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:00 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

smp: Move irq_work_run() out of flush_smp_call_function_queue()

This ensures flush_smp_call_function_queue() is strictly about call_single_queue.

---
 kernel/smp.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
kernel/smp.c +9 -8
index db2f738..f720e38 100644
@@ -84,6 +84,7 @@ int smpcfd_dying_cpu(unsigned int cpu)
 	 * still pending.
 	 */
 	flush_smp_call_function_queue(false);
+	irq_work_run();
 	return 0;
 }
 
@@ -191,6 +192,14 @@ static int generic_exec_single(int cpu, call_single_data_t *csd,
 void generic_smp_call_function_single_interrupt(void)
 {
 	flush_smp_call_function_queue(true);
+
+	/*
+	 * Handle irq works queued remotely by irq_work_queue_on().
+	 * Smp functions above are typically synchronous so they
+	 * better run first since some other CPUs may be busy waiting
+	 * for them.
+	 */
+	irq_work_run();
 }
 
 /**
@@ -267,14 +276,6 @@ static void flush_smp_call_function_queue(bool warn_cpu_offline)
 		csd_unlock(csd);
 		func(info);
 	}
-
-	/*
-	 * Handle irq works queued remotely by irq_work_queue_on().
-	 * Smp functions above are typically synchronous so they
-	 * better run first since some other CPUs may be busy waiting
-	 * for them.
-	 */
-	irq_work_run();
 }
 
 /*
author maintainer
#15 lore raw

RFC PATCH 4/7 smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra

Just like the ttwu_queue_remote() IPI, make use of _TIF_POLLING_NRFLAG to avoid sending IPIs to idle CPUs.

---
 kernel/sched/core.c  |   10 ++++++++++
 kernel/sched/idle.c  |    1 +
 kernel/sched/sched.h |    2 ++
 kernel/smp.c         |   16 +++++++++++++++-
 4 files changed, 28 insertions(+), 1 deletion(-)
kernel/sched/core.c +10 -0
@@ -2296,6 +2296,16 @@ static void wake_csd_func(void *info)
 	sched_ttwu_pending();
 }
 
+void send_call_function_single_ipi(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if (!set_nr_if_polling(rq->idle))
+		arch_send_call_function_single_ipi(cpu);
+	else
+		trace_sched_wake_idle_without_ipi(cpu);
+}
+
 /*
  * Queue a task on the target CPUs wake_list and wake the CPU via IPI if
  * necessary. The wakee CPU on receipt of the IPI will queue the task
kernel/sched/idle.c +1 -0
@@ -289,6 +289,7 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	flush_smp_call_function_from_idle();
 	sched_ttwu_pending();
 	schedule_idle();
 
kernel/sched/sched.h +2 -0
@@ -1506,6 +1506,8 @@ static inline void unregister_sched_doma
 }
 #endif
 
+extern void flush_smp_call_function_from_idle(void);
+
 #else
 
 static inline void sched_ttwu_pending(void) { }
kernel/smp.c +15 -1
@@ -135,6 +135,8 @@ static __always_inline void csd_unlock(c
 
 static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data);
 
+extern void send_call_function_single_ipi(int cpu);
+
 /*
  * Insert a previously allocated call_single_data_t element
  * for execution on the given CPU. data must already have
@@ -178,7 +180,7 @@ static int generic_exec_single(int cpu,
 	 * equipped to do the right thing...
 	 */
 	if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu)))
-		arch_send_call_function_single_ipi(cpu);
+		send_call_function_single_ipi(cpu);
 
 	return 0;
 }
@@ -278,6 +280,18 @@ static void flush_smp_call_function_queu
 	}
 }
 
+void flush_smp_call_function_from_idle(void)
+{
+	unsigned long flags;
+
+	if (llist_empty(this_cpu_ptr(&call_single_queue)))
+		return;
+
+	local_irq_save(flags);
+	flush_smp_call_function_queue(true);
+	local_irq_restore(flags);
+}
+
 /*
  * smp_call_function_single - Run a function on a specific CPU
  * @func: The function to run. This must be fast and non-blocking.
#16 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Peter,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/sched/core]
[also build test ERROR on tip/auto-latest next-20200526]
[cannot apply to linus/master linux/master v5.7-rc7]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Fix-the-scheduler-IPI-mess/20200527-010828
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 2ebb17717550607bcd85fb8cf7d24ac870e9d762
config: i386-tinyconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-13) 9.3.0
reproduce (this is a W=1 build):
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

kernel/sched/idle.c: In function 'do_idle':

>> kernel/sched/idle.c:292:2: error: implicit declaration of function 'flush_smp_call_function_from_idle' [-Werror=implicit-function-declaration]

292 | flush_smp_call_function_from_idle();
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors

vim +/flush_smp_call_function_from_idle +292 kernel/sched/idle.c

225
226 /*
227 * Generic idle loop implementation
228 *
229 * Called with polling cleared.
230 */
231 static void do_idle(void)
232 {
233 int cpu = smp_processor_id();
234 /*
235 * If the arch has a polling bit, we maintain an invariant:
236 *
237 * Our polling bit is clear if we're not scheduled (i.e. if rq->curr !=
238 * rq->idle). This means that, if rq->idle has the polling bit set,
239 * then setting need_resched is guaranteed to cause the CPU to
240 * reschedule.
241 */
242
243 __current_set_polling();
244 tick_nohz_idle_enter();
245
246 while (!need_resched()) {
247 rmb();
248
249 local_irq_disable();
250
251 if (cpu_is_offline(cpu)) {
252 tick_nohz_idle_stop_tick();
253 cpuhp_report_idle_dead();
254 arch_cpu_idle_dead();
255 }
256
257 arch_cpu_idle_enter();
258
259 /*
260 * In poll mode we reenable interrupts and spin. Also if we
261 * detected in the wakeup from idle path that the tick
262 * broadcast device expired for us, we don't want to go deep
263 * idle as we know that the IPI is going to arrive right away.
264 */
265 if (cpu_idle_force_poll || tick_check_broadcast_expired()) {
266 tick_nohz_idle_restart_tick();
267 cpu_idle_poll();
268 } else {
269 cpuidle_idle_call();
270 }
271 arch_cpu_idle_exit();
272 }
273
274 /*
275 * Since we fell out of the loop above, we know TIF_NEED_RESCHED must
276 * be set, propagate it into PREEMPT_NEED_RESCHED.
277 *
278 * This is required because for polling idle loops we will not have had
279 * an IPI to fold the state for us.
280 */
281 preempt_set_need_resched();
282 tick_nohz_idle_exit();
283 __current_clr_polling();
284
285 /*
286 * We promise to call sched_ttwu_pending() and reschedule if
287 * need_resched() is set while polling is set. That means that clearing
288 * polling needs to be visible before doing these things.
289 */
290 smp_mb__after_atomic();
291

 > 292		flush_smp_call_function_from_idle();

293 sched_ttwu_pending();
294 schedule_idle();
295
296 if (unlikely(klp_patch_pending(current)))
297 klp_update_patch_state(current);
298 }
299

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author maintainer
#17 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra
Quoted 33 lines · Peter Zijlstra

Just like the ttwu_queue_remote() IPI, make use of _TIF_POLLING_NRFLAG to avoid sending IPIs to idle CPUs.

---
 kernel/sched/core.c  |   10 ++++++++++
 kernel/sched/idle.c  |    1 +
 kernel/sched/sched.h |    2 ++
 kernel/smp.c         |   16 +++++++++++++++-
 4 files changed, 28 insertions(+), 1 deletion(-)
kernel/sched/core.c +10 -0
@@ -2296,6 +2296,16 @@ static void wake_csd_func(void *info)
 	sched_ttwu_pending();
 }
 
+void send_call_function_single_ipi(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if (!set_nr_if_polling(rq->idle))
+		arch_send_call_function_single_ipi(cpu);
+	else
+		trace_sched_wake_idle_without_ipi(cpu);
+}
+
 /*
  * Queue a task on the target CPUs wake_list and wake the CPU via IPI if
  * necessary. The wakee CPU on receipt of the IPI will queue the task

kernel/sched/idle.c static void do_idle(void) · line 295

Quoted 7 lines
kernel/sched/idle.c +1 -0
@@ -289,6 +289,7 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	flush_smp_call_function_from_idle();
 	sched_ttwu_pending();
 	schedule_idle();
 

Paul; the above patch basically allows smp_call_function_single() to run from the idle context (with IRQs disabled, obviously) instead of from an actual IRQ context.

This makes RCU unhappy (as reported by mingo):

[ ] ------------[ cut here ]------------
[ ] Not in hardirq as expected
[ ] WARNING: CPU: 4 PID: 0 at kernel/rcu/tree.c:430 rcu_is_cpu_rrupt_from_idle+0xed/0x110 [ ] Modules linked in:
[ ] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.7.0-rc7-00840-ga61d572-dirty #1
[ ] Hardware name: Supermicro H8DG6/H8DGi/H8DG6/H8DGi, BIOS 2.0b 03/01/2012
[ ] RIP: 0010:rcu_is_cpu_rrupt_from_idle+0xed/0x110
[ ] Call Trace:

 [ ]  rcu_exp_handler+0x38/0x90
 [ ]  flush_smp_call_function_queue+0xce/0x230
 [ ]  flush_smp_call_function_from_idle+0x2f/0x60
 [ ]  do_idle+0x163/0x260
 [ ]  cpu_startup_entry+0x19/0x20
 [ ]  start_secondary+0x14f/0x1a0

[ ] irq event stamp: 189300
[ ] hardirqs last enabled at (189299): [<ffffffff811d3e25>] tick_nohz_idle_exit+0x55/0xb0 [ ] hardirqs last disabled at (189300): [<ffffffff811da5f5>] flush_smp_call_function_from_idle+0x25/0x60 [ ] softirqs last enabled at (189284): [<ffffffff811280a0>] irq_enter_rcu+0x70/0x80
[ ] softirqs last disabled at (189283): [<ffffffff81128085>] irq_enter_rcu+0x55/0x80

This is rcu_is_cpu_rrupt_from_idle()'s lockdep_assert_in_irq() tripping up (it's comment is obviously a bit antiquated).

Now, if I read that code correctly, it actually relies on rcu_irq_enter() and thus really wants to be in an interrupt. Is there any way this code can be made to work from the new context too?

After all, all that really is different is not having gone throught he bother of setting up the IRQ context, nothing else changed -- it just so happens you actually relied on that ;/

author maintainer
#18 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 10 lines · Peter Zijlstra

This is rcu_is_cpu_rrupt_from_idle()'s lockdep_assert_in_irq() tripping up (it's comment is obviously a bit antiquated).

Now, if I read that code correctly, it actually relies on rcu_irq_enter() and thus really wants to be in an interrupt. Is there any way this code can be made to work from the new context too?

After all, all that really is different is not having gone throught he bother of setting up the IRQ context, nothing else changed -- it just so happens you actually relied on that ;/

At first glance, something like the below could work. But obviously I might have overlooked something more subtle than a brick :-)

---

kernel/rcu/tree.c +10 -4
index 90c8be22d57a..0792c032a972 100644
@@ -426,8 +426,11 @@ EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	/*
+	 * Usually called from the tick; but also used from smp_call_function()
+	 * for expedited grace periods.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -435,8 +438,11 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <= 0,
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
-	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	/*
+	 * Are we at first interrupt nesting level? -- or below, when running
+	 * directly from the idle loop itself.
+	 */
+	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) > 1)
 		return false;
 
 	/* Does CPU appear to be idle from an RCU standpoint? */
maintainer
#19 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 9 lines · Peter Zijlstra
Quoted 7 lines · Peter Zijlstra

This is rcu_is_cpu_rrupt_from_idle()'s lockdep_assert_in_irq() tripping up (it's comment is obviously a bit antiquated).

Now, if I read that code correctly, it actually relies on rcu_irq_enter() and thus really wants to be in an interrupt. Is there any way this code can be made to work from the new context too?

Quoted 6 lines
Quoted 3 lines

After all, all that really is different is not having gone throught he bother of setting up the IRQ context, nothing else changed -- it just so happens you actually relied on that ;/

At first glance, something like the below could work. But obviously I might have overlooked something more subtle than a brick :-)

This can work, but only if the call from the idle loop is a place where either RCU isn't watching on the one hand or that cannot be in an RCU read-side critical section on the other. Because rcu_exp_handler() assumes that if this function returns true, we are not in an RCU read-side critical section. (I would expect this to be the case, but I figured that I should make it explicit.)

kernel/rcu/tree.c EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle); · line 426

Quoted 8 lines

---

kernel/rcu/tree.c +0 -0
index 90c8be22d57a..0792c032a972 100644
@@ -426,8 +426,11 @@ EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
  */

Could we please have a comment noting the change in semantics and the reason?

Quoted 13 lines
kernel/rcu/tree.c +5 -2
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	/*
+	 * Usually called from the tick; but also used from smp_call_function()
+	 * for expedited grace periods.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 445

Quoted 10 lines
kernel/rcu/tree.c +5 -2
@@ -435,8 +438,11 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <= 0,
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
-	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	/*
+	 * Are we at first interrupt nesting level? -- or below, when running
+	 * directly from the idle loop itself.
+	 */
+	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) > 1)

Wouldn't it also be a good idea to check that we are in the context of an idle thread? Just in case some idiot like me drops a call to this function in the wrong place, for example, if I were to mistakenly remember the old semantics where it would return false from process context?

Maybe something like this?

	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting;
	if (nesting > 1)
		return false;
	WARN_ON_ONCE(!nesting && !is_idle_task(current));

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 448

Quoted 3 lines
kernel/rcu/tree.c +0 -0
 		return false;
 
 	/* Does CPU appear to be idle from an RCU standpoint? */

And let's check the other callers:

rcu_sched_clock_irq(): This will always be called from IRQ (right?), so
no problem.

rcu_pending(): Only called from rcu_sched_clock_irq(), so still no problem.

rcu_flavor_sched_clock_irq(): Ditto for both definitions.

Thanx, Paul

author maintainer
#20 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 8 lines · Paul E. McKenney
Quoted 2 lines · Peter Zijlstra

At first glance, something like the below could work. But obviously I might have overlooked something more subtle than a brick :-)

This can work, but only if the call from the idle loop is a place where either RCU isn't watching on the one hand or that cannot be in an RCU read-side critical section on the other.

Guaranteed no RCU read side, although the call is in a place where RCU is active again, is that a problem? I think with a bit of work I can move it to where RCU is still idle.

Quoted 3 lines

Because rcu_exp_handler() assumes that if this function returns true, we are not in an RCU read-side critical section. (I would expect this to be the case, but I figured that I should make it explicit.)

Indeed, I shall put a comment in the idle look to make sure it stays that way.

Quoted 11 lines

kernel/rcu/tree.c EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle); · line 426

Quoted 8 lines

---

kernel/rcu/tree.c +0 -0
index 90c8be22d57a..0792c032a972 100644
@@ -426,8 +426,11 @@ EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
  */

Could we please have a comment noting the change in semantics and the reason?

A Changelog you mean? Sure, I can do, but I wasn't nowhere confident enough in the change to even bother trying to write one.

Quoted 29 lines
Quoted 13 lines
kernel/rcu/tree.c +5 -2
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	/*
+	 * Usually called from the tick; but also used from smp_call_function()
+	 * for expedited grace periods.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 445

Quoted 10 lines
kernel/rcu/tree.c +5 -2
@@ -435,8 +438,11 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nmi_nesting) <= 0,
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
-	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	/*
+	 * Are we at first interrupt nesting level? -- or below, when running
+	 * directly from the idle loop itself.
+	 */
+	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) > 1)

Wouldn't it also be a good idea to check that we are in the context of an idle thread? Just in case some idiot like me drops a call to this function in the wrong place, for example, if I were to mistakenly remember the old semantics where it would return false from process context?

Quoted 6 lines

Maybe something like this?

	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting;
	if (nesting > 1)
		return false;
	WARN_ON_ONCE(!nesting && !is_idle_task(current));

Yep, that should do.

Quoted 4 lines

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 448

Quoted 3 lines
kernel/rcu/tree.c +0 -0
 		return false;
 
 	/* Does CPU appear to be idle from an RCU standpoint? */
Quoted 8 lines

And let's check the other callers:

rcu_sched_clock_irq(): This will always be called from IRQ (right?), so
no problem.

rcu_pending(): Only called from rcu_sched_clock_irq(), so still no problem.

rcu_flavor_sched_clock_irq(): Ditto for both definitions.

Right, I went though them, didn't find anything obvious amiss. OK, let me do a nicer patch.

author maintainer
#21 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 2 lines · Peter Zijlstra

Right, I went though them, didn't find anything obvious amiss. OK, let me do a nicer patch.

something like so then?

---
Subject: rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

---
 kernel/rcu/tree.c   | 25 +++++++++++++++++++------
 kernel/sched/idle.c |  4 ++++
 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +19 -6
index d8e9dbbefcfa..c716eadc7617 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }
kernel/sched/idle.c +4 -0
index e9cef84c2b70..05deb81bb3e3 100644
@@ -289,6 +289,10 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	/*
+	 * RCU relies on this call to be done outside of an RCU read-side
+	 * critical section.
+	 */
 	flush_smp_call_function_from_idle();
 	schedule_idle();
 
maintainer Reviewed-by
#22 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 15 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

Right, I went though them, didn't find anything obvious amiss. OK, let me do a nicer patch.

something like so then?

---
Subject: rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Quoted 4 lines

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

Looks good to me!

Quoted 70 lines
---
 kernel/rcu/tree.c   | 25 +++++++++++++++++++------
 kernel/sched/idle.c |  4 ++++
 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +19 -6
index d8e9dbbefcfa..c716eadc7617 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }
kernel/sched/idle.c +4 -0
index e9cef84c2b70..05deb81bb3e3 100644
@@ -289,6 +289,10 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	/*
+	 * RCU relies on this call to be done outside of an RCU read-side
+	 * critical section.
+	 */
 	flush_smp_call_function_from_idle();
 	schedule_idle();
 
Reviewed-by
#23 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 16 lines · Paul E. McKenney
Quoted 15 lines · Peter Zijlstra
Quoted 2 lines · Peter Zijlstra

Right, I went though them, didn't find anything obvious amiss. OK, let me do a nicer patch.

something like so then?

---
Subject: rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Quoted 8 lines
Quoted 4 lines

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

Looks good to me!

thanks,

- Joel

Quoted 71 lines
Quoted 71 lines
---
 kernel/rcu/tree.c   | 25 +++++++++++++++++++------
 kernel/sched/idle.c |  4 ++++
 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +19 -6
index d8e9dbbefcfa..c716eadc7617 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }
kernel/sched/idle.c +4 -0
index e9cef84c2b70..05deb81bb3e3 100644
@@ -289,6 +289,10 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	/*
+	 * RCU relies on this call to be done outside of an RCU read-side
+	 * critical section.
+	 */
 	flush_smp_call_function_from_idle();
 	schedule_idle();
 
merge bot Applied
#24 lore raw

[tip: core/rcu] rcu: Allow for smp_call_function() running callbacks from idle

In reply to Peter Zijlstra
The following commit has been merged into the core/rcu branch of tip:

The following commit has been merged into the core/rcu branch of tip:

Commit-ID:     806f04e9fd2c6ad1e39bc2dba77155be0e4becde
Gitweb:        https://git.kernel.org/tip/806f04e9fd2c6ad1e39bc2dba77155be0e4becde
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Wed, 27 May 2020 19:12:36 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:50:12 +02:00

rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

---
 kernel/rcu/tree.c | 25 +++++++++++++++++++------
 1 file changed, 19 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +19 -6
index 90c8be2..f51385b 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }
author maintainer
#25 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra
Quoted 57 lines · Peter Zijlstra

Subject: rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

---
 kernel/rcu/tree.c   | 25 +++++++++++++++++++------
 kernel/sched/idle.c |  4 ++++
 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +14 -6
index d8e9dbbefcfa..c716eadc7617 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 457

Quoted 10 lines
kernel/rcu/tree.c +5 -0
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }

Let me revive this thread after yesterdays IRC conversation.

As said; it might be _extremely_ unlikely, but somewhat possible for us to send the IPI concurrent with hot-unplug, not yet observing rcutree_offline_cpu() or thereabout.

Then have the IPI 'delayed' enough to not happen until smpcfd_dying() and getting ran there.

This would then run the function from the stopper thread instead of the idle thread and trigger the warning, even though we're not holding rcu_read_lock() (which, IIRC, was the only constraint).

So would something like the below be acceptable?

---

kernel/rcu/tree.c +8 -3
index 368749008ae8..2c8d4c3e341e 100644
@@ -445,7 +445,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	/*
 	 * Usually called from the tick; but also used from smp_function_call()
 	 * for expedited grace periods. This latter can result in running from
-	 * the idle task, instead of an actual IPI.
+	 * a (usually the idle) task, instead of an actual IPI.
 	 */
 	lockdep_assert_irqs_disabled();
 
@@ -461,9 +461,14 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 		return false;
 
 	/*
-	 * If we're not in an interrupt, we must be in the idle task!
+	 * If we're not in an interrupt, we must be in task context.
+	 *
+	 * This will typically be the idle task through:
+	 *   flush_smp_call_function_from_idle(),
+	 *
+	 * but can also be in CPU HotPlug through smpcfd_dying().
 	 */
-	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+	WARN_ON_ONCE(!nesting && !in_task(current));
 
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
maintainer
#26 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 90 lines · Peter Zijlstra
Quoted 57 lines · Peter Zijlstra

Subject: rcu: Allow for smp_call_function() running callbacks from idle

Current RCU hard relies on smp_call_function() callbacks running from interrupt context. A pending optimization is going to break that, it will allow idle CPUs to run the callbacks from the idle loop. This avoids raising the IPI on the requesting CPU and avoids handling an exception on the receiving CPU.

Change rcu_is_cpu_rrupt_from_idle() to also accept task context, provided it is the idle task.

---
 kernel/rcu/tree.c   | 25 +++++++++++++++++++------
 kernel/sched/idle.c |  4 ++++
 2 files changed, 23 insertions(+), 6 deletions(-)
kernel/rcu/tree.c +14 -6
index d8e9dbbefcfa..c716eadc7617 100644
@@ -418,16 +418,23 @@ void rcu_momentary_dyntick_idle(void)
 EXPORT_SYMBOL_GPL(rcu_momentary_dyntick_idle);
 
 /**
- * rcu_is_cpu_rrupt_from_idle - see if interrupted from idle
+ * rcu_is_cpu_rrupt_from_idle - see if 'interrupted' from idle
  *
  * If the current CPU is idle and running at a first-level (not nested)
- * interrupt from idle, return true.  The caller must have at least
- * disabled preemption.
+ * interrupt, or directly, from idle, return true.
+ *
+ * The caller must have at least disabled IRQs.
  */
 static int rcu_is_cpu_rrupt_from_idle(void)
 {
-	/* Called only from within the scheduling-clock interrupt */
-	lockdep_assert_in_irq();
+	long nesting;
+
+	/*
+	 * Usually called from the tick; but also used from smp_function_call()
+	 * for expedited grace periods. This latter can result in running from
+	 * the idle task, instead of an actual IPI.
+	 */
+	lockdep_assert_irqs_disabled();
 
 	/* Check for counter underflows */
 	RCU_LOCKDEP_WARN(__this_cpu_read(rcu_data.dynticks_nesting) < 0,
@@ -436,9 +443,15 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 			 "RCU dynticks_nmi_nesting counter underflow/zero!");
 
 	/* Are we at first interrupt nesting level? */
-	if (__this_cpu_read(rcu_data.dynticks_nmi_nesting) != 1)
+	nesting = __this_cpu_read(rcu_data.dynticks_nmi_nesting);
+	if (nesting > 1)

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 457

Quoted 10 lines
kernel/rcu/tree.c +5 -0
 		return false;
 
+	/*
+	 * If we're not in an interrupt, we must be in the idle task!
+	 */
+	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
 }

Let me revive this thread after yesterdays IRC conversation.

As said; it might be _extremely_ unlikely, but somewhat possible for us to send the IPI concurrent with hot-unplug, not yet observing rcutree_offline_cpu() or thereabout.

Then have the IPI 'delayed' enough to not happen until smpcfd_dying() and getting ran there.

This would then run the function from the stopper thread instead of the idle thread and trigger the warning, even though we're not holding rcu_read_lock() (which, IIRC, was the only constraint).

So would something like the below be acceptable?

---

kernel/rcu/tree.c +0 -0
index 368749008ae8..2c8d4c3e341e 100644

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 448

Quoted 5 lines
kernel/rcu/tree.c +1 -1
@@ -445,7 +445,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	/*
 	 * Usually called from the tick; but also used from smp_function_call()
 	 * for expedited grace periods. This latter can result in running from
-	 * the idle task, instead of an actual IPI.
+	 * a (usually the idle) task, instead of an actual IPI.

The story is growing enough hair that we should tell it only once. So here just where it is called from:

	/*
	 * Usually called from the tick; but also used from smp_function_call()
	 * for expedited grace periods.
	 */
Quoted 4 lines
kernel/rcu/tree.c +0 -0
 	lockdep_assert_irqs_disabled();
 
@@ -461,9 +461,14 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 		return false;

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 470

Quoted 10 lines
kernel/rcu/tree.c +6 -1
 
 	/*
-	 * If we're not in an interrupt, we must be in the idle task!
+	 * If we're not in an interrupt, we must be in task context.
+	 *
+	 * This will typically be the idle task through:
+	 *   flush_smp_call_function_from_idle(),
+	 *
+	 * but can also be in CPU HotPlug through smpcfd_dying().
 	 */

Good, but how about like this?

/*
* If we are not in an interrupt handler, we must be in
* smp_call_function() handler.
*
* Normally, smp_call_function() handlers are invoked from
* the idle task via flush_smp_call_function_from_idle().
* However, they can also be invoked from CPU hotplug
* operations via smpcfd_dying().
*/

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 471

Quoted 2 lines
kernel/rcu/tree.c +1 -1
-	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+	WARN_ON_ONCE(!nesting && !in_task(current));

This is used in time-critical contexts, so why not RCU_LOCKDEP_WARN()? That should also allow checking more closely. Would something like the following work?

	RCU_LOCKDEP_WARN(!nesting && !is_idle_task(current) && (!in_task(current) || !lockdep_cpus_write_held()));

Where lockdep_cpus_write_held is defined in kernel/cpu.c:

void lockdep_cpus_write_held(void)
{

#ifdef CONFIG_PROVE_LOCKING
	if (system_state < SYSTEM_RUNNING)
		return false;
	return lockdep_is_held_type(&cpu_hotplug_lock, 0);
#else
	return false;

#endif
}

Seem reasonable?

Thanx, Paul

Quoted 2 lines
kernel/rcu/tree.c +0 -0
 	/* Does CPU appear to be idle from an RCU standpoint? */
 	return __this_cpu_read(rcu_data.dynticks_nesting) == 0;
author maintainer
#27 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 54 lines · Paul E. McKenney

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 448

Quoted 11 lines

---

kernel/rcu/tree.c +1 -1
index 368749008ae8..2c8d4c3e341e 100644
@@ -445,7 +445,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	/*
 	 * Usually called from the tick; but also used from smp_function_call()
 	 * for expedited grace periods. This latter can result in running from
-	 * the idle task, instead of an actual IPI.
+	 * a (usually the idle) task, instead of an actual IPI.

The story is growing enough hair that we should tell it only once. So here just where it is called from:

	/*
	 * Usually called from the tick; but also used from smp_function_call()
	 * for expedited grace periods.
	 */
Quoted 4 lines
kernel/rcu/tree.c +0 -0
 	lockdep_assert_irqs_disabled();
 
@@ -461,9 +461,14 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 		return false;

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 470

Quoted 10 lines
kernel/rcu/tree.c +6 -1
 
 	/*
-	 * If we're not in an interrupt, we must be in the idle task!
+	 * If we're not in an interrupt, we must be in task context.
+	 *
+	 * This will typically be the idle task through:
+	 *   flush_smp_call_function_from_idle(),
+	 *
+	 * but can also be in CPU HotPlug through smpcfd_dying().
 	 */

Good, but how about like this?

/*
* If we are not in an interrupt handler, we must be in
* smp_call_function() handler.
*
* Normally, smp_call_function() handlers are invoked from
* the idle task via flush_smp_call_function_from_idle().
* However, they can also be invoked from CPU hotplug
* operations via smpcfd_dying().
*/

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 471

Quoted 2 lines
kernel/rcu/tree.c +1 -1
-	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+	WARN_ON_ONCE(!nesting && !in_task(current));

This is used in time-critical contexts, so why not RCU_LOCKDEP_WARN()? That should also allow checking more closely. Would something like the following work?

Quoted 3 lines
	RCU_LOCKDEP_WARN(!nesting && !is_idle_task(current) && (!in_task(current) || !lockdep_cpus_write_held()));

Where lockdep_cpus_write_held is defined in kernel/cpu.c:

Works for me, except s/in_task(current)/in_task()/ compiles a lot better.

maintainer
#28 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 56 lines · Peter Zijlstra
Quoted 54 lines · Paul E. McKenney

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 448

Quoted 11 lines

---

kernel/rcu/tree.c +1 -1
index 368749008ae8..2c8d4c3e341e 100644
@@ -445,7 +445,7 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 	/*
 	 * Usually called from the tick; but also used from smp_function_call()
 	 * for expedited grace periods. This latter can result in running from
-	 * the idle task, instead of an actual IPI.
+	 * a (usually the idle) task, instead of an actual IPI.

The story is growing enough hair that we should tell it only once. So here just where it is called from:

	/*
	 * Usually called from the tick; but also used from smp_function_call()
	 * for expedited grace periods.
	 */
Quoted 4 lines
kernel/rcu/tree.c +0 -0
 	lockdep_assert_irqs_disabled();
 
@@ -461,9 +461,14 @@ static int rcu_is_cpu_rrupt_from_idle(void)
 		return false;

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 470

Quoted 10 lines
kernel/rcu/tree.c +6 -1
 
 	/*
-	 * If we're not in an interrupt, we must be in the idle task!
+	 * If we're not in an interrupt, we must be in task context.
+	 *
+	 * This will typically be the idle task through:
+	 *   flush_smp_call_function_from_idle(),
+	 *
+	 * but can also be in CPU HotPlug through smpcfd_dying().
 	 */

Good, but how about like this?

/*
* If we are not in an interrupt handler, we must be in
* smp_call_function() handler.
*
* Normally, smp_call_function() handlers are invoked from
* the idle task via flush_smp_call_function_from_idle().
* However, they can also be invoked from CPU hotplug
* operations via smpcfd_dying().
*/

kernel/rcu/tree.c static int rcu_is_cpu_rrupt_from_idle(void) · line 471

Quoted 2 lines
kernel/rcu/tree.c +1 -1
-	WARN_ON_ONCE(!nesting && !is_idle_task(current));
+	WARN_ON_ONCE(!nesting && !in_task(current));

This is used in time-critical contexts, so why not RCU_LOCKDEP_WARN()? That should also allow checking more closely. Would something like the following work?

Quoted 6 lines
Quoted 3 lines
	RCU_LOCKDEP_WARN(!nesting && !is_idle_task(current) && (!in_task(current) || !lockdep_cpus_write_held()));

Where lockdep_cpus_write_held is defined in kernel/cpu.c:

Works for me, except s/in_task(current)/in_task()/ compiles a lot better.

These compilers sure constrain my creativity! ;-)

Might be a good thing, though...

Thanx, Paul

#29 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Peter,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on next-20200528]
[cannot apply to tip/auto-latest linus/master linux/master v5.7-rc7] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Peter-Zijlstra/Fix-the-scheduler-IPI-mess/20200527-010828
base:   https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 2ebb17717550607bcd85fb8cf7d24ac870e9d762
config: arm64-allyesconfig (attached as .config)
compiler: aarch64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=arm64

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>, old ones prefixed by <<):

kernel//sched/core.c:2116:6: warning: no previous prototype for 'sched_set_stop_task' [-Wmissing-prototypes] 2116 | void sched_set_stop_task(int cpu, struct task_struct *stop)
| ^~~~~~~~~~~~~~~~~~~

>> kernel//sched/core.c:2299:6: warning: no previous prototype for 'send_call_function_single_ipi' [-Wmissing-prototypes]

2299 | void send_call_function_single_ipi(int cpu)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

vim +/send_call_function_single_ipi +2299 kernel//sched/core.c

2298

> 2299	void send_call_function_single_ipi(int cpu)

2300 {
2301 struct rq *rq = cpu_rq(cpu);
2302
2303 if (!set_nr_if_polling(rq->idle))
2304 arch_send_call_function_single_ipi(cpu);
2305 else
2306 trace_sched_wake_idle_without_ipi(cpu);
2307 }
2308

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

author maintainer
#30 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 9 lines · kbuild test robot

Hi Peter,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on next-20200528]
[cannot apply to tip/auto-latest linus/master linux/master v5.7-rc7] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

Can you please just not build scraped patches from me? If I want 0day coverage of them I'll stick them in my k.org git tree.

#31 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

Quoted 11 lines · Peter Zijlstra
Quoted 9 lines · kbuild test robot

Hi Peter,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on tip/sched/core]
[also build test WARNING on next-20200528]
[cannot apply to tip/auto-latest linus/master linux/master v5.7-rc7] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

Quoted 2 lines

Can you please just not build scraped patches from me? If I want 0day coverage of them I'll stick them in my k.org git tree.

Got it, Peter, we will not pick up the patches from you to avoid noise.

Thanks

Quoted 3 lines

_______________________________________________
kbuild-all mailing list -- kbuild-all(a)lists.01.org To unsubscribe send an email to kbuild-all-leave(a)lists.01.org

maintainer Reviewed-by on v1 4/7
#32 lore raw

Re: [RFC][PATCH 4/7] smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra

kernel/smp.c static void flush_smp_call_function_queu · line 288

Quoted 6 lines · Peter Zijlstra
kernel/smp.c +6 -0
+void flush_smp_call_function_from_idle(void)
+{
+	unsigned long flags;
+
+	if (llist_empty(this_cpu_ptr(&call_single_queue)))
+		return;

Now it seems weird that sched_ttwu_pending() didn't have that llist_empty() optimization. The ordering should allow it.

Anyway,

merge bot Applied
#33 lore raw

[tip: sched/core] smp: Optimize send_call_function_single_ipi()

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     b2a02fc43a1f40ef4eb2fb2b06357382608d4d84
Gitweb:        https://git.kernel.org/tip/b2a02fc43a1f40ef4eb2fb2b06357382608d4d84
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:01 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

smp: Optimize send_call_function_single_ipi()

Just like the ttwu_queue_remote() IPI, make use of _TIF_POLLING_NRFLAG to avoid sending IPIs to idle CPUs.

[ mingo: Fix UP build bug. ]

---
 kernel/sched/core.c  | 10 ++++++++++
 kernel/sched/idle.c  |  5 +++++
 kernel/sched/sched.h |  7 ++++---
 kernel/smp.c         | 16 +++++++++++++++-
 4 files changed, 34 insertions(+), 4 deletions(-)
kernel/sched/core.c +10 -0
index 2cacc1e..fa0d499 100644
@@ -2296,6 +2296,16 @@ static void wake_csd_func(void *info)
 	sched_ttwu_pending();
 }
 
+void send_call_function_single_ipi(int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if (!set_nr_if_polling(rq->idle))
+		arch_send_call_function_single_ipi(cpu);
+	else
+		trace_sched_wake_idle_without_ipi(cpu);
+}
+
 /*
  * Queue a task on the target CPUs wake_list and wake the CPU via IPI if
  * necessary. The wakee CPU on receipt of the IPI will queue the task
kernel/sched/idle.c +5 -0
index b743bf3..387fd75 100644
@@ -289,6 +289,11 @@ static void do_idle(void)
 	 */
 	smp_mb__after_atomic();
 
+	/*
+	 * RCU relies on this call to be done outside of an RCU read-side
+	 * critical section.
+	 */
+	flush_smp_call_function_from_idle();
 	sched_ttwu_pending();
 	schedule_idle();
 
kernel/sched/sched.h +4 -3
index 3c163cb..75b0629 100644
@@ -1506,11 +1506,12 @@ static inline void unregister_sched_domain_sysctl(void)
 }
 #endif
 
-#else
+extern void flush_smp_call_function_from_idle(void);
 
+#else /* !CONFIG_SMP: */
+static inline void flush_smp_call_function_from_idle(void) { }
 static inline void sched_ttwu_pending(void) { }
-
-#endif /* CONFIG_SMP */
+#endif
 
 #include "stats.h"
 #include "autogroup.h"
kernel/smp.c +15 -1
index f720e38..9f11813 100644
@@ -135,6 +135,8 @@ static __always_inline void csd_unlock(call_single_data_t *csd)
 
 static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data);
 
+extern void send_call_function_single_ipi(int cpu);
+
 /*
  * Insert a previously allocated call_single_data_t element
  * for execution on the given CPU. data must already have
@@ -178,7 +180,7 @@ static int generic_exec_single(int cpu, call_single_data_t *csd,
 	 * equipped to do the right thing...
 	 */
 	if (llist_add(&csd->llist, &per_cpu(call_single_queue, cpu)))
-		arch_send_call_function_single_ipi(cpu);
+		send_call_function_single_ipi(cpu);
 
 	return 0;
 }
@@ -278,6 +280,18 @@ static void flush_smp_call_function_queue(bool warn_cpu_offline)
 	}
 }
 
+void flush_smp_call_function_from_idle(void)
+{
+	unsigned long flags;
+
+	if (llist_empty(this_cpu_ptr(&call_single_queue)))
+		return;
+
+	local_irq_save(flags);
+	flush_smp_call_function_queue(true);
+	local_irq_restore(flags);
+}
+
 /*
  * smp_call_function_single - Run a function on a specific CPU
  * @func: The function to run. This must be fast and non-blocking.
author maintainer
#34 lore raw

RFC PATCH 5/7 irq_work, smp: Allow irq_work on call_single_queue

In reply to Peter Zijlstra

Currently irq_work_queue_on() will issue an unconditional arch_send_call_function_single_ipi() and has the handler do irq_work_run().

This is unfortunate in that it makes the IPI handler look at a second cacheline and it misses the opportunity to avoid the IPI. Instead note that struct irq_work and struct __call_single_data are very similar in layout, so use a few bits in the flags word to encode a type and stick the irq_work on the call_single_queue list.

---
 include/linux/irq_work.h |    7 ++
 include/linux/smp.h      |   23 ++++++++-
 kernel/irq_work.c        |   53 +++++++++++---------
 kernel/smp.c             |  119 ++++++++++++++++++++++++++++-------------------
 4 files changed, 130 insertions(+), 72 deletions(-)
include/linux/irq_work.h +6 -1
@@ -13,6 +13,8 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
+/* flags share CSD_FLAG_ space */
+
 #define IRQ_WORK_PENDING	BIT(0)
 #define IRQ_WORK_BUSY		BIT(1)
 
@@ -23,9 +25,12 @@
 
 #define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
 
+/*
+ * structure shares layout with single_call_data_t.
+ */
 struct irq_work {
-	atomic_t flags;
 	struct llist_node llnode;
+	atomic_t flags;
 	void (*func)(struct irq_work *);
 };
 
include/linux/smp.h +22 -1
@@ -16,17 +16,38 @@
 
 typedef void (*smp_call_func_t)(void *info);
 typedef bool (*smp_cond_func_t)(int cpu, void *info);
+
+enum {
+	CSD_FLAG_LOCK		= 0x01,
+
+	/* IRQ_WORK_flags */
+
+	CSD_TYPE_ASYNC		= 0x00,
+	CSD_TYPE_SYNC		= 0x10,
+	CSD_TYPE_IRQ_WORK	= 0x20,
+	CSD_FLAG_TYPE_MASK	= 0xF0,
+};
+
+/*
+ * structure shares (partial) layout with struct irq_work
+ */
 struct __call_single_data {
 	struct llist_node llist;
+	unsigned int flags;
 	smp_call_func_t func;
 	void *info;
-	unsigned int flags;
 };
 
 /* Use __aligned() to avoid to use 2 cache lines for 1 csd */
 typedef struct __call_single_data call_single_data_t
 	__aligned(sizeof(struct __call_single_data));
 
+/*
+ * Enqueue a llist_node on the call_single_queue; be very careful, read
+ * flush_smp_call_function_queue() in detail.
+ */
+extern void __smp_call_single_queue(int cpu, struct llist_node *node);
+
 /* total number of cpus in this system (may exceed NR_CPUS) */
 extern unsigned int total_cpus;
 
kernel/irq_work.c +29 -24

77 lines not on this page · raw message

kernel/smp.c +73 -46

212 lines not on this page · raw message

#35 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

Quoted 30 lines · Peter Zijlstra

Currently irq_work_queue_on() will issue an unconditional arch_send_call_function_single_ipi() and has the handler do irq_work_run().

This is unfortunate in that it makes the IPI handler look at a second cacheline and it misses the opportunity to avoid the IPI. Instead note that struct irq_work and struct __call_single_data are very similar in layout, so use a few bits in the flags word to encode a type and stick the irq_work on the call_single_queue list.

---
 include/linux/irq_work.h |    7 ++
 include/linux/smp.h      |   23 ++++++++-
 kernel/irq_work.c        |   53 +++++++++++---------
 kernel/smp.c             |  119 ++++++++++++++++++++++++++++-------------------
 4 files changed, 130 insertions(+), 72 deletions(-)
include/linux/irq_work.h +2 -0
@@ -13,6 +13,8 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
+/* flags share CSD_FLAG_ space */
+
 #define IRQ_WORK_PENDING	BIT(0)
 #define IRQ_WORK_BUSY		BIT(1)
 

include/linux/irq_work.h line 33

Quoted 10 lines
include/linux/irq_work.h +4 -1
@@ -23,9 +25,12 @@
 
 #define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
 
+/*
+ * structure shares layout with single_call_data_t.
+ */
 struct irq_work {
-	atomic_t flags;
 	struct llist_node llnode;
+	atomic_t flags;

We should probably have:

struct csd_node {
       atomic_t flags;
       struct llist_node;
}

embed inside struct irq_work and struct __call_single_data. Relying on structure layout for things to work doesn't really clarify things :-)

Thanks.

author maintainer
#36 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

Quoted 13 lines · Frederic Weisbecker

include/linux/irq_work.h line 33

Quoted 7 lines · Peter Zijlstra
include/linux/irq_work.h +4 -1
+/*
+ * structure shares layout with single_call_data_t.
+ */
 struct irq_work {
-	atomic_t flags;
 	struct llist_node llnode;
+	atomic_t flags;

We should probably have:

Quoted 7 lines
struct csd_node {
       atomic_t flags;
       struct llist_node;
}

embed inside struct irq_work and struct __call_single_data. Relying on structure layout for things to work doesn't really clarify things :-)

Yes I know, but changing those structures is going to cause an aweful lot of churn, and I didn't want to do that just now.. :-(

Also, there's more fun..

CSD_TYPE_SYNC/ASYNC:

	struct {
		struct llist_node node;
		unsigned int flags;
		smp_call_func_t func;
		void *info;
	};

CSD_TYPE_IRQ_WORK:

	struct {
		struct llist_node node;
		atomic_t flags;
		void (*func)(struct irq_work *);
	};

CSD_TYPE_TTWU:

	struct {
		struct llist_node node;
		unsigned int flags;
	};

So while they all have a 'u32' sized @flags, irq_work wants it atomic. Also, if we were to actually have the struct csd_node {}, you get a 4 byte hole when you embed it in task_struct.

This is all entirely fugly. No doubt about it.

But I failed to find a 'sane' way to express it and needed to get these patches out because things were broken.

Maybe I can anonymous-union my way around it, dunno. I'll think about it. I'm certainly not proud of this. But at least the BUILD_BUG_ON()s should catch the more blatant breakage here.

author maintainer
#37 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

Quoted 3 lines · Peter Zijlstra

Maybe I can anonymous-union my way around it, dunno. I'll think about it. I'm certainly not proud of this. But at least the BUILD_BUG_ON()s should catch the more blatant breakage here.

How's this then? Differently ugly, but at least it compiles with that horrible struct randomization junk enabled.

---
 include/linux/irq_work.h  |   28 ++++++-------------
 include/linux/sched.h     |    4 +-
 include/linux/smp.h       |   25 ++++++-----------
 include/linux/smp_types.h |   66 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/core.c       |    6 ++--
 kernel/smp.c              |   18 ------------
 6 files changed, 89 insertions(+), 58 deletions(-)
include/linux/irq_work.h +9 -19
@@ -2,7 +2,7 @@
 #ifndef _LINUX_IRQ_WORK_H
 #define _LINUX_IRQ_WORK_H
 
-#include <linux/llist.h>
+#include <linux/smp_types.h>
 
 /*
  * An entry can be in one of four states:
@@ -13,26 +13,16 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
-/* flags share CSD_FLAG_ space */
-
-#define IRQ_WORK_PENDING	BIT(0)
-#define IRQ_WORK_BUSY		BIT(1)
-
-/* Doesn't want IPI, wait for tick: */
-#define IRQ_WORK_LAZY		BIT(2)
-/* Run hard IRQ context, even on RT */
-#define IRQ_WORK_HARD_IRQ	BIT(3)
-
-#define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
-
-/*
- * structure shares layout with single_call_data_t.
- */
 struct irq_work {
-	struct llist_node llnode;
-	atomic_t flags;
+	union {
+		struct __call_single_node node;
+		struct {
+			struct llist_node llnode;
+			atomic_t flags;
+		};
+	};
 	void (*func)(struct irq_work *);
-};
+} __no_randomize_layout;
 
 static inline
 void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *))
include/linux/sched.h +2 -2
@@ -32,6 +32,7 @@
 #include <linux/posix-timers.h>
 #include <linux/rseq.h>
 #include <linux/kcsan.h>
+#include <linux/smp_types.h>
 
 /* task_struct member predeclarations (sorted alphabetically): */
 struct audit_context;
@@ -654,9 +655,8 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
 	int				on_cpu;
+	struct __call_single_node	wake_entry;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
 	unsigned int			cpu;
include/linux/smp.h +9 -16
@@ -12,32 +12,25 @@
 #include <linux/list.h>
 #include <linux/cpumask.h>
 #include <linux/init.h>
-#include <linux/llist.h>
+#include <linux/smp_types.h>
 
 typedef void (*smp_call_func_t)(void *info);
 typedef bool (*smp_cond_func_t)(int cpu, void *info);
 
-enum {
-	CSD_FLAG_LOCK		= 0x01,
-
-	/* IRQ_WORK_flags */
-
-	CSD_TYPE_ASYNC		= 0x00,
-	CSD_TYPE_SYNC		= 0x10,
-	CSD_TYPE_IRQ_WORK	= 0x20,
-	CSD_TYPE_TTWU		= 0x30,
-	CSD_FLAG_TYPE_MASK	= 0xF0,
-};
-
 /*
  * structure shares (partial) layout with struct irq_work
  */
 struct __call_single_data {
-	struct llist_node llist;
-	unsigned int flags;
+	union {
+		struct __call_single_node node;
+		struct {
+			struct llist_node llist;
+			unsigned int flags;
+		};
+	};
 	smp_call_func_t func;
 	void *info;
-};
+} __no_randomize_layout;
 
 /* Use __aligned() to avoid to use 2 cache lines for 1 csd */
 typedef struct __call_single_data call_single_data_t
include/linux/smp_types.h new file +66 -0

66 lines not on this page · raw message

kernel/sched/core.c +3 -3
@@ -2293,7 +2293,7 @@ void sched_ttwu_pending(void *arg)
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
-	llist_for_each_entry_safe(p, t, llist, wake_entry)
+	llist_for_each_entry_safe(p, t, llist, wake_entry.llist)
 		ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
 
 	rq_unlock_irqrestore(rq, &rf);
@@ -2322,7 +2322,7 @@ static void __ttwu_queue_wakelist(struct
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	WRITE_ONCE(rq->ttwu_pending, 1);
-	__smp_call_single_queue(cpu, &p->wake_entry);
+	__smp_call_single_queue(cpu, &p->wake_entry.llist);
 }
 
 void wake_up_if_idle(int cpu)
@@ -2763,7 +2763,7 @@ static void __sched_fork(unsigned long c
 #endif
 	init_numa_balancing(clone_flags, p);
 #ifdef CONFIG_SMP
-	p->wake_entry_type = CSD_TYPE_TTWU;
+	p->wake_entry.u_flags = CSD_TYPE_TTWU;
 #endif
 }
 
kernel/smp.c +0 -18
@@ -669,24 +669,6 @@ void __init smp_init(void)
 {
 	int num_nodes, num_cpus;
 
-	/*
-	 * Ensure struct irq_work layout matches so that
-	 * flush_smp_call_function_queue() can do horrible things.
-	 */
-	BUILD_BUG_ON(offsetof(struct irq_work, llnode) !=
-		     offsetof(struct __call_single_data, llist));
-	BUILD_BUG_ON(offsetof(struct irq_work, func) !=
-		     offsetof(struct __call_single_data, func));
-	BUILD_BUG_ON(offsetof(struct irq_work, flags) !=
-		     offsetof(struct __call_single_data, flags));
-
-	/*
-	 * Assert the CSD_TYPE_TTWU layout is similar enough
-	 * for task_struct to be on the @call_single_queue.
-	 */
-	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
-		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
-
 	idle_threads_init();
 	cpuhp_threads_init();
 
#38 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

Quoted 47 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

Maybe I can anonymous-union my way around it, dunno. I'll think about it. I'm certainly not proud of this. But at least the BUILD_BUG_ON()s should catch the more blatant breakage here.

How's this then? Differently ugly, but at least it compiles with that horrible struct randomization junk enabled.

---
 include/linux/irq_work.h  |   28 ++++++-------------
 include/linux/sched.h     |    4 +-
 include/linux/smp.h       |   25 ++++++-----------
 include/linux/smp_types.h |   66 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/core.c       |    6 ++--
 kernel/smp.c              |   18 ------------
 6 files changed, 89 insertions(+), 58 deletions(-)
include/linux/irq_work.h +1 -16
@@ -2,7 +2,7 @@
 #ifndef _LINUX_IRQ_WORK_H
 #define _LINUX_IRQ_WORK_H
 
-#include <linux/llist.h>
+#include <linux/smp_types.h>
 
 /*
  * An entry can be in one of four states:
@@ -13,26 +13,16 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
-/* flags share CSD_FLAG_ space */
-
-#define IRQ_WORK_PENDING	BIT(0)
-#define IRQ_WORK_BUSY		BIT(1)
-
-/* Doesn't want IPI, wait for tick: */
-#define IRQ_WORK_LAZY		BIT(2)
-/* Run hard IRQ context, even on RT */
-#define IRQ_WORK_HARD_IRQ	BIT(3)
-
-#define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
-
-/*
- * structure shares layout with single_call_data_t.
- */

include/linux/irq_work.h line 23

Quoted 10 lines
include/linux/irq_work.h +7 -2
 struct irq_work {
-	struct llist_node llnode;
-	atomic_t flags;
+	union {
+		struct __call_single_node node;
+		struct {
+			struct llist_node llnode;
+			atomic_t flags;
+		};
+	};

So why not just embed struct __call_single_node in struct irq_work and struct __call_single_data ?

Is the point of that anonymous second union layer to shorten the lines while accessing members?

Thanks.

author maintainer
#39 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

Quoted 15 lines · Frederic Weisbecker

include/linux/irq_work.h line 23

Quoted 10 lines · Peter Zijlstra

struct irq_work {

include/linux/irq_work.h +7 -2
-	struct llist_node llnode;
-	atomic_t flags;
+	union {
+		struct __call_single_node node;
+		struct {
+			struct llist_node llnode;
+			atomic_t flags;
+		};
+	};

So why not just embed struct __call_single_node in struct irq_work and struct __call_single_data ?

Quoted 2 lines

Is the point of that anonymous second union layer to shorten the lines while accessing members?

To cut down on all the churn. irq_work is small and could easily be fixed up, but csd is used in quite a few places.

#40 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

In reply to Peter Zijlstra
Quoted 50 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

Maybe I can anonymous-union my way around it, dunno. I'll think about it. I'm certainly not proud of this. But at least the BUILD_BUG_ON()s should catch the more blatant breakage here.

How's this then? Differently ugly, but at least it compiles with that horrible struct randomization junk enabled.

---
 include/linux/irq_work.h  |   28 ++++++-------------
 include/linux/sched.h     |    4 +-
 include/linux/smp.h       |   25 ++++++-----------
 include/linux/smp_types.h |   66 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/core.c       |    6 ++--
 kernel/smp.c              |   18 ------------
 6 files changed, 89 insertions(+), 58 deletions(-)
include/linux/irq_work.h +1 -18
@@ -2,7 +2,7 @@
 #ifndef _LINUX_IRQ_WORK_H
 #define _LINUX_IRQ_WORK_H
 
-#include <linux/llist.h>
+#include <linux/smp_types.h>
 
 /*
  * An entry can be in one of four states:
@@ -13,26 +13,16 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
-/* flags share CSD_FLAG_ space */
-
-#define IRQ_WORK_PENDING	BIT(0)
-#define IRQ_WORK_BUSY		BIT(1)
-
-/* Doesn't want IPI, wait for tick: */
-#define IRQ_WORK_LAZY		BIT(2)
-/* Run hard IRQ context, even on RT */
-#define IRQ_WORK_HARD_IRQ	BIT(3)
-
-#define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
-
-/*
- * structure shares layout with single_call_data_t.
- */
 struct irq_work {
-	struct llist_node llnode;
-	atomic_t flags;

include/linux/irq_work.h line 25

Quoted 10 lines
include/linux/irq_work.h +8 -1
+	union {
+		struct __call_single_node node;
+		struct {
+			struct llist_node llnode;
+			atomic_t flags;
+		};
+	};
 	void (*func)(struct irq_work *);
-};
+} __no_randomize_layout;

The "__no_randomize_layout" isn't needed here. The only automatically randomized structs are those entirely consisting of function pointers.

Quoted 54 lines
include/linux/irq_work.h +0 -0
 static inline
 void init_irq_work(struct irq_work *work, void (*func)(struct irq_work *))
include/linux/sched.h +2 -2
@@ -32,6 +32,7 @@
 #include <linux/posix-timers.h>
 #include <linux/rseq.h>
 #include <linux/kcsan.h>
+#include <linux/smp_types.h>
 
 /* task_struct member predeclarations (sorted alphabetically): */
 struct audit_context;
@@ -654,9 +655,8 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
 	int				on_cpu;
+	struct __call_single_node	wake_entry;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
 	unsigned int			cpu;
include/linux/smp.h +2 -15
@@ -12,32 +12,25 @@
 #include <linux/list.h>
 #include <linux/cpumask.h>
 #include <linux/init.h>
-#include <linux/llist.h>
+#include <linux/smp_types.h>
 
 typedef void (*smp_call_func_t)(void *info);
 typedef bool (*smp_cond_func_t)(int cpu, void *info);
 
-enum {
-	CSD_FLAG_LOCK		= 0x01,
-
-	/* IRQ_WORK_flags */
-
-	CSD_TYPE_ASYNC		= 0x00,
-	CSD_TYPE_SYNC		= 0x10,
-	CSD_TYPE_IRQ_WORK	= 0x20,
-	CSD_TYPE_TTWU		= 0x30,
-	CSD_FLAG_TYPE_MASK	= 0xF0,
-};
-
 /*
  * structure shares (partial) layout with struct irq_work
  */
 struct __call_single_data {
-	struct llist_node llist;
-	unsigned int flags;
+	union {

include/linux/smp.h line 33

Quoted 10 lines
include/linux/smp.h +7 -1
+		struct __call_single_node node;
+		struct {
+			struct llist_node llist;
+			unsigned int flags;
+		};
+	};
 	smp_call_func_t func;
 	void *info;
-};
+} __no_randomize_layout;

Same here.

Quoted 60 lines
include/linux/smp.h +0 -0
 /* Use __aligned() to avoid to use 2 cache lines for 1 csd */
 typedef struct __call_single_data call_single_data_t
include/linux/smp_types.h new file +54 -0
@@ -0,0 +1,66 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_SMP_TYPES_H
+#define __LINUX_SMP_TYPES_H
+
+#include <linux/llist.h>
+
+enum {
+	CSD_FLAG_LOCK		= 0x01,
+
+	IRQ_WORK_PENDING	= 0x01,
+	IRQ_WORK_BUSY		= 0x02,
+	IRQ_WORK_LAZY		= 0x04, /* No IPI, wait for tick */
+	IRQ_WORK_HARD_IRQ	= 0x08, /* IRQ context on PREEMPT_RT */
+
+	IRQ_WORK_CLAIMED	= (IRQ_WORK_PENDING | IRQ_WORK_BUSY),
+
+	CSD_TYPE_ASYNC		= 0x00,
+	CSD_TYPE_SYNC		= 0x10,
+	CSD_TYPE_IRQ_WORK	= 0x20,
+	CSD_TYPE_TTWU		= 0x30,
+
+	CSD_FLAG_TYPE_MASK	= 0xF0,
+};
+
+/*
+ * struct __call_single_node is the primary type on
+ * smp.c:call_single_queue.
+ *
+ * flush_smp_call_function_queue() only reads the type from
+ * __call_single_node::u_flags as a regular load, the above
+ * (anonymous) enum defines all the bits of this word.
+ *
+ * Other bits are not modified until the type is known.
+ *
+ * CSD_TYPE_SYNC/ASYNC:
+ *	struct {
+ *		struct llist_node node;
+ *		unsigned int flags;
+ *		smp_call_func_t func;
+ *		void *info;
+ *	};
+ *
+ * CSD_TYPE_IRQ_WORK:
+ *	struct {
+ *		struct llist_node node;
+ *		atomic_t flags;
+ *		void (*func)(struct irq_work *);
+ *	};
+ *
+ * CSD_TYPE_TTWU:
+ *	struct {
+ *		struct llist_node node;
+ *		unsigned int flags;
+ *	};

include/linux/smp_types.h line 64

Quoted 10 lines
include/linux/smp_types.h new file +10 -0
+ *
+ */
+
+struct __call_single_node {
+	struct llist_node	llist;
+	union {
+		unsigned int	u_flags;
+		atomic_t	a_flags;
+	};
+} __no_randomize_layout;

Same.

Quoted 45 lines
include/linux/smp_types.h new file +2 -0
+
+#endif /* __LINUX_SMP_TYPES_H */
kernel/sched/core.c +3 -3
@@ -2293,7 +2293,7 @@ void sched_ttwu_pending(void *arg)
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
-	llist_for_each_entry_safe(p, t, llist, wake_entry)
+	llist_for_each_entry_safe(p, t, llist, wake_entry.llist)
 		ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf);
 
 	rq_unlock_irqrestore(rq, &rf);
@@ -2322,7 +2322,7 @@ static void __ttwu_queue_wakelist(struct
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	WRITE_ONCE(rq->ttwu_pending, 1);
-	__smp_call_single_queue(cpu, &p->wake_entry);
+	__smp_call_single_queue(cpu, &p->wake_entry.llist);
 }
 
 void wake_up_if_idle(int cpu)
@@ -2763,7 +2763,7 @@ static void __sched_fork(unsigned long c
 #endif
 	init_numa_balancing(clone_flags, p);
 #ifdef CONFIG_SMP
-	p->wake_entry_type = CSD_TYPE_TTWU;
+	p->wake_entry.u_flags = CSD_TYPE_TTWU;
 #endif
 }
 
kernel/smp.c +0 -8
@@ -669,24 +669,6 @@ void __init smp_init(void)
 {
 	int num_nodes, num_cpus;
 
-	/*
-	 * Ensure struct irq_work layout matches so that
-	 * flush_smp_call_function_queue() can do horrible things.
-	 */
-	BUILD_BUG_ON(offsetof(struct irq_work, llnode) !=
-		     offsetof(struct __call_single_data, llist));
-	BUILD_BUG_ON(offsetof(struct irq_work, func) !=
-		     offsetof(struct __call_single_data, func));

kernel/smp.c void __init smp_init(void) · line 689

Quoted 10 lines
kernel/smp.c +0 -10
-	BUILD_BUG_ON(offsetof(struct irq_work, flags) !=
-		     offsetof(struct __call_single_data, flags));
-
-	/*
-	 * Assert the CSD_TYPE_TTWU layout is similar enough
-	 * for task_struct to be on the @call_single_queue.
-	 */
-	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
-		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
-

Do you want to validate that the individual members of the union struct still have their fields lining up with __call_single_node's members? Or better yet, I have the same question as Frederic about the need for the union. Why not just switch callers from "flags" to "node.u_flags" and "node.a_flags"? (Or could that be cleaned up in a later patch to avoid putting too much churn in one patch?)

Signature · 1 line
Kees Cook
maintainer Reviewed-by
#41 lore raw

Re: [RFC][PATCH 5/7] irq_work, smp: Allow irq_work on call_single_queue

In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

Maybe I can anonymous-union my way around it, dunno. I'll think about it. I'm certainly not proud of this. But at least the BUILD_BUG_ON()s should catch the more blatant breakage here.

How's this then? Differently ugly, but at least it compiles with that horrible struct randomization junk enabled.

Quoted 8 lines
---
 include/linux/irq_work.h  |   28 ++++++-------------
 include/linux/sched.h     |    4 +-
 include/linux/smp.h       |   25 ++++++-----------
 include/linux/smp_types.h |   66 ++++++++++++++++++++++++++++++++++++++++++++++
 kernel/sched/core.c       |    6 ++--
 kernel/smp.c              |   18 ------------
 6 files changed, 89 insertions(+), 58 deletions(-)

Looks good. I don't have a better idea.

Thanks!

merge bot Applied
#42 lore raw

[tip: sched/core] irq_work, smp: Allow irq_work on call_single_queue

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     4b44a21dd640b692d4e9b12d3e37c24825f90baa
Gitweb:        https://git.kernel.org/tip/4b44a21dd640b692d4e9b12d3e37c24825f90baa
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:02 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:15 +02:00

irq_work, smp: Allow irq_work on call_single_queue

Currently irq_work_queue_on() will issue an unconditional arch_send_call_function_single_ipi() and has the handler do irq_work_run().

This is unfortunate in that it makes the IPI handler look at a second cacheline and it misses the opportunity to avoid the IPI. Instead note that struct irq_work and struct __call_single_data are very similar in layout, so use a few bits in the flags word to encode a type and stick the irq_work on the call_single_queue list.

---
 include/linux/irq_work.h |   7 +-
 include/linux/smp.h      |  23 ++++++-
 kernel/irq_work.c        |  53 +++++++++--------
 kernel/smp.c             | 119 +++++++++++++++++++++++---------------
 4 files changed, 130 insertions(+), 72 deletions(-)
include/linux/irq_work.h +6 -1
index 3b752e8..f23a359 100644
@@ -13,6 +13,8 @@
  * busy      NULL, 2 -> {free, claimed} : callback in progress, can be claimed
  */
 
+/* flags share CSD_FLAG_ space */
+
 #define IRQ_WORK_PENDING	BIT(0)
 #define IRQ_WORK_BUSY		BIT(1)
 
@@ -23,9 +25,12 @@
 
 #define IRQ_WORK_CLAIMED	(IRQ_WORK_PENDING | IRQ_WORK_BUSY)
 
+/*
+ * structure shares layout with single_call_data_t.
+ */
 struct irq_work {
-	atomic_t flags;
 	struct llist_node llnode;
+	atomic_t flags;
 	void (*func)(struct irq_work *);
 };
 
include/linux/smp.h +22 -1
index cbc9162..45ad6e3 100644
@@ -16,17 +16,38 @@
 
 typedef void (*smp_call_func_t)(void *info);
 typedef bool (*smp_cond_func_t)(int cpu, void *info);
+
+enum {
+	CSD_FLAG_LOCK		= 0x01,
+
+	/* IRQ_WORK_flags */
+
+	CSD_TYPE_ASYNC		= 0x00,
+	CSD_TYPE_SYNC		= 0x10,
+	CSD_TYPE_IRQ_WORK	= 0x20,
+	CSD_FLAG_TYPE_MASK	= 0xF0,
+};
+
+/*
+ * structure shares (partial) layout with struct irq_work
+ */
 struct __call_single_data {
 	struct llist_node llist;
+	unsigned int flags;
 	smp_call_func_t func;
 	void *info;
-	unsigned int flags;
 };
 
 /* Use __aligned() to avoid to use 2 cache lines for 1 csd */
 typedef struct __call_single_data call_single_data_t
 	__aligned(sizeof(struct __call_single_data));
 
+/*
+ * Enqueue a llist_node on the call_single_queue; be very careful, read
+ * flush_smp_call_function_queue() in detail.
+ */
+extern void __smp_call_single_queue(int cpu, struct llist_node *node);
+
 /* total number of cpus in this system (may exceed NR_CPUS) */
 extern unsigned int total_cpus;
 
kernel/irq_work.c +29 -24

77 lines not on this page · raw message

kernel/smp.c +73 -46

212 lines not on this page · raw message

author maintainer
#43 lore raw

RFC PATCH 6/7 sched: Add rq::ttwu_pending

In reply to Peter Zijlstra

In preparation of removing rq->wake_list, replace the !list_empty(rq->wake_list) with rq->ttwu_pending. This is not fully equivalent as this new variable is racy.

---
 kernel/sched/core.c  |   13 +++++++++++--
 kernel/sched/debug.c |    1 -
 kernel/sched/fair.c  |    2 +-
 kernel/sched/sched.h |    4 +++-
 4 files changed, 15 insertions(+), 5 deletions(-)
kernel/sched/core.c +11 -2
@@ -2275,13 +2275,21 @@ static int ttwu_remote(struct task_struc
 void sched_ttwu_pending(void)
 {
 	struct rq *rq = this_rq();
-	struct llist_node *llist = llist_del_all(&rq->wake_list);
+	struct llist_node *llist;
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
+	llist = llist_del_all(&rq->wake_list);
 	if (!llist)
 		return;
 
+	/*
+	 * rq::ttwu_pending racy indication of out-standing wakeups.
+	 * Races such that false-negatives are possible, since they
+	 * are shorter lived that false-positives would be.
+	 */
+	WRITE_ONCE(rq->ttwu_pending, 0);
+
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
@@ -2312,6 +2320,7 @@ static void ttwu_queue_remote(struct tas
 
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
+	WRITE_ONCE(rq->ttwu_pending, 1);
 	if (llist_add(&p->wake_entry, &rq->wake_list)) {
 		if (!set_nr_if_polling(rq->idle))
 			smp_call_function_single_async(cpu, &rq->wake_csd);
@@ -4668,7 +4677,7 @@ int idle_cpu(int cpu)
 		return 0;
 
 #ifdef CONFIG_SMP
-	if (!llist_empty(&rq->wake_list))
+	if (rq->ttwu_pending)
 		return 0;
 #endif
 
kernel/sched/debug.c +0 -1
@@ -638,7 +638,6 @@ do {									\
 
 	P(nr_running);
 	P(nr_switches);
-	P(nr_load_updates);
 	P(nr_uninterruptible);
 	PN(next_balance);
 	SEQ_printf(m, "  .%-30s: %ld\n", "curr->pid", (long)(task_pid_nr(rq->curr)));
kernel/sched/fair.c +1 -1
@@ -8590,7 +8590,7 @@ static int idle_cpu_without(int cpu, str
 	 */
 
 #ifdef CONFIG_SMP
-	if (!llist_empty(&rq->wake_list))
+	if (rq->ttwu_pending)
 		return 0;
 #endif
 
kernel/sched/sched.h +3 -1
@@ -895,7 +895,9 @@ struct rq {
 	atomic_t		nohz_flags;
 #endif /* CONFIG_NO_HZ_COMMON */
 
-	unsigned long		nr_load_updates;
+#ifdef CONFIG_SMP
+	unsigned int		ttwu_pending;
+#endif
 	u64			nr_switches;
 
 #ifdef CONFIG_UCLAMP_TASK
merge bot Applied
#44 lore raw

[tip: sched/core] sched: Add rq::ttwu_pending

The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     126c2092e5c8b28623cb890cd2930aa292410676
Gitweb:        https://git.kernel.org/tip/126c2092e5c8b28623cb890cd2930aa292410676
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:03 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:16 +02:00

sched: Add rq::ttwu_pending

In preparation of removing rq->wake_list, replace the !list_empty(rq->wake_list) with rq->ttwu_pending. This is not fully equivalent as this new variable is racy.

---
 kernel/sched/core.c  | 13 +++++++++++--
 kernel/sched/debug.c |  1 -
 kernel/sched/fair.c  |  2 +-
 kernel/sched/sched.h |  4 +++-
 4 files changed, 15 insertions(+), 5 deletions(-)
kernel/sched/core.c +11 -2
index fa0d499..b71ed5e 100644
@@ -2275,13 +2275,21 @@ static int ttwu_remote(struct task_struct *p, int wake_flags)
 void sched_ttwu_pending(void)
 {
 	struct rq *rq = this_rq();
-	struct llist_node *llist = llist_del_all(&rq->wake_list);
+	struct llist_node *llist;
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
+	llist = llist_del_all(&rq->wake_list);
 	if (!llist)
 		return;
 
+	/*
+	 * rq::ttwu_pending racy indication of out-standing wakeups.
+	 * Races such that false-negatives are possible, since they
+	 * are shorter lived that false-positives would be.
+	 */
+	WRITE_ONCE(rq->ttwu_pending, 0);
+
 	rq_lock_irqsave(rq, &rf);
 	update_rq_clock(rq);
 
@@ -2318,6 +2326,7 @@ static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags
 
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
+	WRITE_ONCE(rq->ttwu_pending, 1);
 	if (llist_add(&p->wake_entry, &rq->wake_list)) {
 		if (!set_nr_if_polling(rq->idle))
 			smp_call_function_single_async(cpu, &rq->wake_csd);
@@ -4705,7 +4714,7 @@ int idle_cpu(int cpu)
 		return 0;
 
 #ifdef CONFIG_SMP
-	if (!llist_empty(&rq->wake_list))
+	if (rq->ttwu_pending)
 		return 0;
 #endif
 
kernel/sched/debug.c +0 -1
index 1c24a6b..36c5426 100644
@@ -638,7 +638,6 @@ do {									\
 
 	P(nr_running);
 	P(nr_switches);
-	P(nr_load_updates);
 	P(nr_uninterruptible);
 	PN(next_balance);
 	SEQ_printf(m, "  .%-30s: %ld\n", "curr->pid", (long)(task_pid_nr(rq->curr)));
kernel/sched/fair.c +1 -1
index 2890bd5..0ed04d2 100644
@@ -8590,7 +8590,7 @@ static int idle_cpu_without(int cpu, struct task_struct *p)
 	 */
 
 #ifdef CONFIG_SMP
-	if (!llist_empty(&rq->wake_list))
+	if (rq->ttwu_pending)
 		return 0;
 #endif
 
kernel/sched/sched.h +3 -1
index 75b0629..c86fc94 100644
@@ -895,7 +895,9 @@ struct rq {
 	atomic_t		nohz_flags;
 #endif /* CONFIG_NO_HZ_COMMON */
 
-	unsigned long		nr_load_updates;
+#ifdef CONFIG_SMP
+	unsigned int		ttwu_pending;
+#endif
 	u64			nr_switches;
 
 #ifdef CONFIG_UCLAMP_TASK
author maintainer
#45 lore raw

RFC PATCH 7/7 sched: Replace rq::wake_list

In reply to Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---
 include/linux/sched.h |    1 +
 include/linux/smp.h   |    1 +
 kernel/sched/core.c   |   25 +++++++------------------
 kernel/sched/idle.c   |    1 -
 kernel/sched/sched.h  |   11 -----------
 kernel/smp.c          |   47 ++++++++++++++++++++++++++++++++++++++++-------
 6 files changed, 49 insertions(+), 37 deletions(-)
include/linux/sched.h +1 -0
@@ -655,6 +655,7 @@ struct task_struct {
 
 #ifdef CONFIG_SMP
 	struct llist_node		wake_entry;
+	unsigned int			wake_entry_type;
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
include/linux/smp.h +1 -0
@@ -25,6 +25,7 @@ enum {
 	CSD_TYPE_ASYNC		= 0x00,
 	CSD_TYPE_SYNC		= 0x10,
 	CSD_TYPE_IRQ_WORK	= 0x20,
+	CSD_TYPE_TTWU		= 0x30,
 	CSD_FLAG_TYPE_MASK	= 0xF0,
 };
 
kernel/sched/core.c +7 -18
@@ -1538,7 +1538,7 @@ static int migration_cpu_stop(void *data
 	 * __migrate_task() such that we will not miss enforcing cpus_ptr
 	 * during wakeups, see set_cpus_allowed_ptr()'s TASK_WAKING test.
 	 */
-	sched_ttwu_pending();
+	flush_smp_call_function_from_idle();
 
 	raw_spin_lock(&p->pi_lock);
 	rq_lock(rq, &rf);
@@ -2272,14 +2272,13 @@ static int ttwu_remote(struct task_struc
 }
 
 #ifdef CONFIG_SMP
-void sched_ttwu_pending(void)
+void sched_ttwu_pending(void *arg)
 {
+	struct llist_node *llist = arg;
 	struct rq *rq = this_rq();
-	struct llist_node *llist;
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
-	llist = llist_del_all(&rq->wake_list);
 	if (!llist)
 		return;
 
@@ -2299,11 +2298,6 @@ void sched_ttwu_pending(void)
 	rq_unlock_irqrestore(rq, &rf);
 }
 
-static void wake_csd_func(void *info)
-{
-	sched_ttwu_pending();
-}
-
 void send_call_function_single_ipi(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
@@ -2321,12 +2315,7 @@ static void ttwu_queue_remote(struct tas
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	WRITE_ONCE(rq->ttwu_pending, 1);
-	if (llist_add(&p->wake_entry, &rq->wake_list)) {
-		if (!set_nr_if_polling(rq->idle))
-			smp_call_function_single_async(cpu, &rq->wake_csd);
-		else
-			trace_sched_wake_idle_without_ipi(cpu);
-	}
+	__smp_call_single_queue(cpu, &p->wake_entry);
 }
 
 void wake_up_if_idle(int cpu)
@@ -2735,6 +2724,9 @@ static void __sched_fork(unsigned long c
 	p->capture_control = NULL;
 #endif
 	init_numa_balancing(clone_flags, p);
+#ifdef CONFIG_SMP
+	p->wake_entry_type = CSD_TYPE_TTWU;
+#endif
 }
 
 DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);
@@ -6527,7 +6519,6 @@ int sched_cpu_dying(unsigned int cpu)
 	struct rq_flags rf;
 
 	/* Handle pending wakeups and then migrate everything off */
-	sched_ttwu_pending();
 	sched_tick_stop(cpu);
 
 	rq_lock_irqsave(rq, &rf);
@@ -6726,8 +6717,6 @@ void __init sched_init(void)
 		rq->avg_idle = 2*sysctl_sched_migration_cost;
 		rq->max_idle_balance_cost = sysctl_sched_migration_cost;
 
-		rq_csd_init(rq, &rq->wake_csd, wake_csd_func);
-
 		INIT_LIST_HEAD(&rq->cfs_tasks);
 
 		rq_attach_root(rq, &def_root_domain);
kernel/sched/idle.c +0 -1
@@ -290,7 +290,6 @@ static void do_idle(void)
 	smp_mb__after_atomic();
 
 	flush_smp_call_function_from_idle();
-	sched_ttwu_pending();
 	schedule_idle();
 
 	if (unlikely(klp_patch_pending(current)))
kernel/sched/sched.h +0 -11
@@ -1023,11 +1023,6 @@ struct rq {
 	unsigned int		ttwu_local;
 #endif
 
-#ifdef CONFIG_SMP
-	call_single_data_t	wake_csd;
-	struct llist_head	wake_list;
-#endif
-
 #ifdef CONFIG_CPU_IDLE
 	/* Must be inspected within a rcu lock section */
 	struct cpuidle_state	*idle_state;
@@ -1371,8 +1366,6 @@ queue_balance_callback(struct rq *rq,
 	rq->balance_callback = head;
 }
 
-extern void sched_ttwu_pending(void);
-
 #define rcu_dereference_check_sched_domain(p) \
 	rcu_dereference_check((p), \
 			      lockdep_is_held(&sched_domains_mutex))
@@ -1510,10 +1503,6 @@ static inline void unregister_sched_doma
 
 extern void flush_smp_call_function_from_idle(void);
 
-#else
-
-static inline void sched_ttwu_pending(void) { }
-
 #endif /* CONFIG_SMP */
 
 #include "stats.h"
kernel/smp.c +40 -7

80 lines not on this page · raw message

#46 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 13 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

 kernel/smp.c          |   47 ++++++++++++++++++++++++++++++++++++++++-------
kernel/smp.c +0 -0
@@ -659,6 +685,13 @@ void __init smp_init(void)
 	BUILD_BUG_ON(offsetof(struct irq_work, flags) !=
 		     offsetof(struct __call_single_data, flags));

kernel/smp.c void __init smp_init(void) · line 696

Quoted 10 lines
kernel/smp.c +7 -0
 
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+
 	idle_threads_init();
 	cpuhp_threads_init();

This blows up on a 32-bit ARM allmodconfig:

CC kernel/smp.o
kernel/smp.c:319:6: warning: no previous prototype for 'flush_smp_call_function_from_idle' [-Wmissing-prototypes]
void flush_smp_call_function_from_idle(void)

      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from ./arch/arm/include/asm/atomic.h:11,
                 from ./include/linux/atomic.h:7,
                 from ./include/linux/llist.h:51,
                 from ./include/linux/irq_work.h:5,
                 from kernel/smp.c:10:
kernel/smp.c: In function 'smp_init':
./include/linux/compiler.h:392:38: error: call to '__compiletime_assert_152' declared with attribute error: BUILD_BUG_ON failed: offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) != offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist)
  _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
                                      ^
./include/linux/compiler.h:373:4: note: in definition of macro '__compiletime_assert'
    prefix ## suffix();    \
    ^~~~~~

./include/linux/compiler.h:392:2: note: in expansion of macro '_compiletime_assert'
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__) ^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'

 #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                     ^~~~~~~~~~~~~~~~~~

./include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
^~~~~~~~~~~~~~~~
kernel/smp.c:689:2: note: in expansion of macro 'BUILD_BUG_ON'
BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) != ^~~~~~~~~~~~

merge bot Applied
#47 lore raw

[tip: sched/core] sched: Replace rq::wake_list

In reply to Peter Zijlstra
The following commit has been merged into the sched/core branch of tip:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     a148866489fbe243c936fe43e4525d8dbfa0318f
Gitweb:        https://git.kernel.org/tip/a148866489fbe243c936fe43e4525d8dbfa0318f
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:04 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:16 +02:00

sched: Replace rq::wake_list

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---
 include/linux/sched.h |  1 +-
 include/linux/smp.h   |  1 +-
 kernel/sched/core.c   | 25 ++++++----------------
 kernel/sched/idle.c   |  1 +-
 kernel/sched/sched.h  |  8 +-------
 kernel/smp.c          | 47 +++++++++++++++++++++++++++++++++++-------
 6 files changed, 49 insertions(+), 34 deletions(-)
include/linux/sched.h +1 -0
index ebc6870..e0f5f41 100644
@@ -654,6 +654,7 @@ struct task_struct {
 
 #ifdef CONFIG_SMP
 	struct llist_node		wake_entry;
+	unsigned int			wake_entry_type;
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
include/linux/smp.h +1 -0
index 45ad6e3..84f90e2 100644
@@ -25,6 +25,7 @@ enum {
 	CSD_TYPE_ASYNC		= 0x00,
 	CSD_TYPE_SYNC		= 0x10,
 	CSD_TYPE_IRQ_WORK	= 0x20,
+	CSD_TYPE_TTWU		= 0x30,
 	CSD_FLAG_TYPE_MASK	= 0xF0,
 };
 
kernel/sched/core.c +7 -18
index b71ed5e..b3c64c6 100644
@@ -1538,7 +1538,7 @@ static int migration_cpu_stop(void *data)
 	 * __migrate_task() such that we will not miss enforcing cpus_ptr
 	 * during wakeups, see set_cpus_allowed_ptr()'s TASK_WAKING test.
 	 */
-	sched_ttwu_pending();
+	flush_smp_call_function_from_idle();
 
 	raw_spin_lock(&p->pi_lock);
 	rq_lock(rq, &rf);
@@ -2272,14 +2272,13 @@ static int ttwu_remote(struct task_struct *p, int wake_flags)
 }
 
 #ifdef CONFIG_SMP
-void sched_ttwu_pending(void)
+void sched_ttwu_pending(void *arg)
 {
+	struct llist_node *llist = arg;
 	struct rq *rq = this_rq();
-	struct llist_node *llist;
 	struct task_struct *p, *t;
 	struct rq_flags rf;
 
-	llist = llist_del_all(&rq->wake_list);
 	if (!llist)
 		return;
 
@@ -2299,11 +2298,6 @@ void sched_ttwu_pending(void)
 	rq_unlock_irqrestore(rq, &rf);
 }
 
-static void wake_csd_func(void *info)
-{
-	sched_ttwu_pending();
-}
-
 void send_call_function_single_ipi(int cpu)
 {
 	struct rq *rq = cpu_rq(cpu);
@@ -2327,12 +2321,7 @@ static void __ttwu_queue_wakelist(struct task_struct *p, int cpu, int wake_flags
 	p->sched_remote_wakeup = !!(wake_flags & WF_MIGRATED);
 
 	WRITE_ONCE(rq->ttwu_pending, 1);
-	if (llist_add(&p->wake_entry, &rq->wake_list)) {
-		if (!set_nr_if_polling(rq->idle))
-			smp_call_function_single_async(cpu, &rq->wake_csd);
-		else
-			trace_sched_wake_idle_without_ipi(cpu);
-	}
+	__smp_call_single_queue(cpu, &p->wake_entry);
 }
 
 void wake_up_if_idle(int cpu)
@@ -2772,6 +2761,9 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
 	p->capture_control = NULL;
 #endif
 	init_numa_balancing(clone_flags, p);
+#ifdef CONFIG_SMP
+	p->wake_entry_type = CSD_TYPE_TTWU;
+#endif
 }
 
 DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);
@@ -6564,7 +6556,6 @@ int sched_cpu_dying(unsigned int cpu)
 	struct rq_flags rf;
 
 	/* Handle pending wakeups and then migrate everything off */
-	sched_ttwu_pending();
 	sched_tick_stop(cpu);
 
 	rq_lock_irqsave(rq, &rf);
@@ -6763,8 +6754,6 @@ void __init sched_init(void)
 		rq->avg_idle = 2*sysctl_sched_migration_cost;
 		rq->max_idle_balance_cost = sysctl_sched_migration_cost;
 
-		rq_csd_init(rq, &rq->wake_csd, wake_csd_func);
-
 		INIT_LIST_HEAD(&rq->cfs_tasks);
 
 		rq_attach_root(rq, &def_root_domain);
kernel/sched/idle.c +0 -1
index 387fd75..05deb81 100644
@@ -294,7 +294,6 @@ static void do_idle(void)
 	 * critical section.
 	 */
 	flush_smp_call_function_from_idle();
-	sched_ttwu_pending();
 	schedule_idle();
 
 	if (unlikely(klp_patch_pending(current)))
kernel/sched/sched.h +0 -8
index c86fc94..1d4e94c 100644
@@ -1023,11 +1023,6 @@ struct rq {
 	unsigned int		ttwu_local;
 #endif
 
-#ifdef CONFIG_SMP
-	call_single_data_t	wake_csd;
-	struct llist_head	wake_list;
-#endif
-
 #ifdef CONFIG_CPU_IDLE
 	/* Must be inspected within a rcu lock section */
 	struct cpuidle_state	*idle_state;
@@ -1371,8 +1366,6 @@ queue_balance_callback(struct rq *rq,
 	rq->balance_callback = head;
 }
 
-extern void sched_ttwu_pending(void);
-
 #define rcu_dereference_check_sched_domain(p) \
 	rcu_dereference_check((p), \
 			      lockdep_is_held(&sched_domains_mutex))
@@ -1512,7 +1505,6 @@ extern void flush_smp_call_function_from_idle(void);
 
 #else /* !CONFIG_SMP: */
 static inline void flush_smp_call_function_from_idle(void) { }
-static inline void sched_ttwu_pending(void) { }
 #endif
 
 #include "stats.h"
kernel/smp.c +40 -7

80 lines not on this page · raw message

#48 lore raw

Re: [tip: sched/core] sched: Replace rq::wake_list

Quoted 22 lines · On Mon, Jun 01, 2020 at 09:52:18AM -0000, tip-bot2 for Peter Zijlstra wrote:

The following commit has been merged into the sched/core branch of tip:

Commit-ID:     a148866489fbe243c936fe43e4525d8dbfa0318f
Gitweb:        https://git.kernel.org/tip/a148866489fbe243c936fe43e4525d8dbfa0318f
Author:        Peter Zijlstra <[email protected]>
AuthorDate:    Tue, 26 May 2020 18:11:04 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Thu, 28 May 2020 10:54:16 +02:00

sched: Replace rq::wake_list

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

Quoted 8 lines

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

Looks good, thanks :)

#49 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

In reply to Peter Zijlstra
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

#50 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 33 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;
Quoted 8 lines

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

CC kernel/smp.o
In file included from ./arch/x86/include/asm/atomic.h:5,

                 from ./include/linux/atomic.h:7,
                 from ./include/linux/llist.h:51,
                 from ./include/linux/irq_work.h:5,
                 from kernel/smp.c:10:

kernel/smp.c: In function ‘smp_init’:
./include/linux/compiler.h:403:38: error: call to ‘__compiletime_assert_68’ declared with attribute error: BUILD_BUG_ON failed: offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) != offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist)
403 | _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
| ^
./include/linux/compiler.h:384:4: note: in definition of macro ‘__compiletime_assert’

  384 |    prefix ## suffix();    \
      |    ^~~~~~
./include/linux/compiler.h:403:2: note: in expansion of macro ‘_compiletime_assert’
  403 |  _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
      |  ^~~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:39:37: note: in expansion of macro ‘compiletime_assert’
   39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
      |                                     ^~~~~~~~~~~~~~~~~~
./include/linux/build_bug.h:50:2: note: in expansion of macro ‘BUILD_BUG_ON_MSG’
   50 |  BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
      |  ^~~~~~~~~~~~~~~~
kernel/smp.c:687:2: note: in expansion of macro ‘BUILD_BUG_ON’
  687 |  BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
      |  ^~~~~~~~~~~~

make[1]: *** [scripts/Makefile.build:267: kernel/smp.o] Error 1
make: *** [Makefile:1735: kernel] Error 2

author maintainer
#51 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 38 lines · Eric Biggers
Quoted 37 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Quoted 6 lines
Quoted 4 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

-ENOCONFIG

#52 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 39 lines · Peter Zijlstra
Quoted 38 lines · Eric Biggers
Quoted 37 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

-ENOCONFIG

For me, the problem seems to be randstruct. To reproduce, you can use (on x86_64):

make defconfig
echo CONFIG_GCC_PLUGIN_RANDSTRUCT=y >> .config make olddefconfig
make kernel/smp.o

#53 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 49 lines · Eric Biggers
Quoted 39 lines · Peter Zijlstra
Quoted 38 lines · Eric Biggers
Quoted 37 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

-ENOCONFIG

Quoted 8 lines

For me, the problem seems to be randstruct. To reproduce, you can use (on x86_64):

make defconfig
echo CONFIG_GCC_PLUGIN_RANDSTRUCT=y >> .config make olddefconfig
make kernel/smp.o

I confirmed that disabling CONFIG_GCC_PLUGIN_RANDSTRUCT "fixes" the problem in my test builds. Maybe it would make sense to mark that configuration option for the time being as BROKEN.

Guenter

#54 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 59 lines · Guenter Roeck
Quoted 49 lines · Eric Biggers
Quoted 39 lines · Peter Zijlstra
Quoted 38 lines · Eric Biggers
Quoted 37 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

kernel/smp.c void __init smp_init(void) · line 694

Quoted 7 lines
kernel/smp.c +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

-ENOCONFIG

Quoted 8 lines

For me, the problem seems to be randstruct. To reproduce, you can use (on x86_64):

make defconfig
echo CONFIG_GCC_PLUGIN_RANDSTRUCT=y >> .config make olddefconfig
make kernel/smp.o

Quoted 4 lines

I confirmed that disabling CONFIG_GCC_PLUGIN_RANDSTRUCT "fixes" the problem in my test builds. Maybe it would make sense to mark that configuration option for the time being as BROKEN.

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

Shouldn't the kbuild test robot have caught this?

- Eric

#55 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 60 lines · Eric Biggers
Quoted 59 lines · Guenter Roeck
Quoted 57 lines · Eric Biggers
Quoted 39 lines · Peter Zijlstra
Quoted 38 lines · Eric Biggers
Quoted 37 lines · Guenter Roeck
Quoted 18 lines · Peter Zijlstra

The recent commit: 90b5363acd47 ("sched: Clean up scheduler_ipi()") got smp_call_function_single_async() subtly wrong. Even though it will return -EBUSY when trying to re-use a csd, that condition is not atomic and still requires external serialization.

The change in ttwu_queue_remote() got this wrong.

While on first reading ttwu_queue_remote() has an atomic test-and-set that appears to serialize the use, the matching 'release' is not in the right place to actually guarantee this serialization.

The actual race is vs the sched_ttwu_pending() call in the idle loop; that can run the wakeup-list without consuming the CSD.

Instead of trying to chain the lists, merge them.

---

...

Review

Quoted 7 lines
(quoted diff) +7 -0
+	/*
+	 * Assert the CSD_TYPE_TTWU layout is similar enough
+	 * for task_struct to be on the @call_single_queue.
+	 */
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

Guenter

Yep, this breaks the build for me.

-ENOCONFIG

For me, the problem seems to be randstruct. To reproduce, you can use (on x86_64):

make defconfig
echo CONFIG_GCC_PLUGIN_RANDSTRUCT=y >> .config make olddefconfig
make kernel/smp.o

Quoted 8 lines
Quoted 4 lines

I confirmed that disabling CONFIG_GCC_PLUGIN_RANDSTRUCT "fixes" the problem in my test builds. Maybe it would make sense to mark that configuration option for the time being as BROKEN.

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

How about the hack below ?

Guenter

---

include/linux/sched.h +12 -2
index c5d96e3e7fff..df1cbb04f9b3 100644
@@ -629,6 +629,15 @@ struct wake_q_node {
 	struct wake_q_node *next;
 };
 
+/*
+ * Hack around assumption that wake_entry_type follows wake_entry even with
+ * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+ */
+struct _wake_entry {
+	struct llist_node	wake_entry;
+	unsigned int		wake_entry_type;
+};
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -653,8 +662,9 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	struct _wake_entry		_we;
+#define wake_entry		_we.wake_entry
+#define wake_entry_type		_we.wake_entry_type
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
#56 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 32 lines · Guenter Roeck
Quoted 4 lines

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

How about the hack below ?

Guenter

---

include/linux/sched.h +9 -0
index c5d96e3e7fff..df1cbb04f9b3 100644
@@ -629,6 +629,15 @@ struct wake_q_node {
 	struct wake_q_node *next;
 };
 
+/*
+ * Hack around assumption that wake_entry_type follows wake_entry even with
+ * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+ */
+struct _wake_entry {
+	struct llist_node	wake_entry;
+	unsigned int		wake_entry_type;
+};
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -653,8 +662,9 @@ struct task_struct {
 	unsigned int			ptrace;

include/linux/sched.h struct task_struct { · line 670

Quoted 10 lines
include/linux/sched.h +3 -2
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	struct _wake_entry		_we;
+#define wake_entry		_we.wake_entry
+#define wake_entry_type		_we.wake_entry_type
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */

Does the struct actually have to be named? How about:

include/linux/sched.h +8 -2
index c5d96e3e7fff42..14ca25cda19150 100644
@@ -653,8 +653,14 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	/*
+	 * wake_entry_type must follow wake_entry, even when
+	 * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+	 */
+	struct {
+		struct llist_node	wake_entry;
+		unsigned int		wake_entry_type;
+	};
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */

However, it would be preferable to not rely on different structs sharing the same field order, but rather write proper C code that uses the same struct everywhere to encapsulate these 2 fields...

- Eric

author maintainer
#57 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 25 lines · Eric Biggers

Does the struct actually have to be named? How about:

include/linux/sched.h +8 -2
index c5d96e3e7fff42..14ca25cda19150 100644
@@ -653,8 +653,14 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	/*
+	 * wake_entry_type must follow wake_entry, even when
+	 * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+	 */
+	struct {
+		struct llist_node	wake_entry;
+		unsigned int		wake_entry_type;
+	};
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
Quoted 3 lines

However, it would be preferable to not rely on different structs sharing the same field order, but rather write proper C code that uses the same struct everywhere to encapsulate these 2 fields...

https://lkml.kernel.org/r/[email protected]

And I have more patches on top to clean up some of the anonymous union stuff, that that's quite a lot of frobbing.

#58 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Eric Biggers

Does the struct actually have to be named? How about:

include/linux/sched.h +8 -2
index c5d96e3e7fff42..14ca25cda19150 100644
@@ -653,8 +653,14 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	/*
+	 * wake_entry_type must follow wake_entry, even when
+	 * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+	 */
+	struct {
+		struct llist_node	wake_entry;
+		unsigned int		wake_entry_type;
+	};
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
Quoted 9 lines
Quoted 3 lines

However, it would be preferable to not rely on different structs sharing the same field order, but rather write proper C code that uses the same struct everywhere to encapsulate these 2 fields...

https://lkml.kernel.org/r/[email protected]

And I have more patches on top to clean up some of the anonymous union stuff, that that's quite a lot of frobbing.

That is why I tried to keep it simple as hackish fixup patch.

Guenter

author maintainer
#59 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 33 lines · Guenter Roeck
Quoted 32 lines · Peter Zijlstra
Quoted 25 lines · Eric Biggers

Does the struct actually have to be named? How about:

include/linux/sched.h +8 -2
index c5d96e3e7fff42..14ca25cda19150 100644
@@ -653,8 +653,14 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	/*
+	 * wake_entry_type must follow wake_entry, even when
+	 * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+	 */
+	struct {
+		struct llist_node	wake_entry;
+		unsigned int		wake_entry_type;
+	};
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
Quoted 3 lines

However, it would be preferable to not rely on different structs sharing the same field order, but rather write proper C code that uses the same struct everywhere to encapsulate these 2 fields...

https://lkml.kernel.org/r/[email protected]

Quoted 5 lines
Quoted 3 lines

And I have more patches on top to clean up some of the anonymous union stuff, that that's quite a lot of frobbing.

That is why I tried to keep it simple as hackish fixup patch.

Fair enough; I'll try and get the above variant merged to address the build fail. Then I can chase down Paul's bug and finisht the cleanup.

#60 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

In reply to Guenter Roeck
Quoted 6 lines · Guenter Roeck
Quoted 4 lines

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

How about the hack below ?

My test suite failed due to this bug (on my allmodconfig test).

Your hack appears to fix it. I've applied it to my "fixes" patches applied to my test suite before building my kernels.

Thanks!

-- Steve

Quoted 36 lines

Guenter

---

include/linux/sched.h +12 -2
index c5d96e3e7fff..df1cbb04f9b3 100644
@@ -629,6 +629,15 @@ struct wake_q_node {
 	struct wake_q_node *next;
 };
 
+/*
+ * Hack around assumption that wake_entry_type follows wake_entry even with
+ * CONFIG_GCC_PLUGIN_RANDSTRUCT=y.
+ */
+struct _wake_entry {
+	struct llist_node	wake_entry;
+	unsigned int		wake_entry_type;
+};
+
 struct task_struct {
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/*
@@ -653,8 +662,9 @@ struct task_struct {
 	unsigned int			ptrace;
 
 #ifdef CONFIG_SMP
-	struct llist_node		wake_entry;
-	unsigned int			wake_entry_type;
+	struct _wake_entry		_we;
+#define wake_entry		_we.wake_entry
+#define wake_entry_type		_we.wake_entry_type
 	int				on_cpu;
 #ifdef CONFIG_THREAD_INFO_IN_TASK
 	/* Current CPU: */
#61 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 6 lines · Steven Rostedt
Quoted 5 lines · Guenter Roeck
Quoted 4 lines

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

Quoted 7 lines
Quoted 1 line

How about the hack below ?

My test suite failed due to this bug (on my allmodconfig test).

Your hack appears to fix it. I've applied it to my "fixes" patches applied to my test suite before building my kernels.

Ah, I would have hoped that this was by now fixed upstream. Apparently that is not the case. Guess Linus is using an old version of gcc (or maybe clang) for his test builds, and everyone doing test builds is now using this or a similar hack. For my part, I disabled CONFIG_GCC_PLUGIN_RANDSTRUCT.

Guenter

author maintainer
#62 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

In reply to Eric Biggers
Quoted 4 lines · Eric Biggers

Still occurring on Linus' tree. This needs to be fixed. (And not by removing support for randstruct; that's not a "fix"...)

Shouldn't the kbuild test robot have caught this?

Should probably, but the thing has been rather spotty of late.

author maintainer
#63 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

In reply to Guenter Roeck
Quoted 8 lines · Guenter Roeck

kernel/smp.c void __init smp_init(void) · line 694

Quoted 3 lines · Peter Zijlstra
kernel/smp.c +3 -0
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

Quoted 6 lines
	type1 a;
	type2 b;

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Do tell more; the alignment requirements and size of the types remains the same, this resulting in different layout is unlikely.

I found this excellent quote on Hacker News this morning:

"I think the attitude of compiler writers is a good reason to fix the
spec so they can't keep ratfucking developers trying to get work done."

Quoted 2 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

The only thing I can think of that's actually a problem is that retarded struct randomization stuff.

Anyway, I'll move cleaning it up a little higher on the todo list.

#64 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Guenter Roeck

kernel/smp.c void __init smp_init(void) · line 694

Quoted 3 lines · Peter Zijlstra
kernel/smp.c +3 -0
+	BUILD_BUG_ON(offsetof(struct task_struct, wake_entry_type) - offsetof(struct task_struct, wake_entry) !=
+		     offsetof(struct __call_single_data, flags) - offsetof(struct __call_single_data, llist));
+

There is no guarantee in C that

	type1 a;
	type2 b;
Quoted 7 lines
Quoted 3 lines

in two different data structures means that offsetof(b) - offsetof(a) is the same in both data structures unless attributes such as __attribute__((__packed__)) are used.

Do tell more; the alignment requirements and size of the types remains the same, this resulting in different layout is unlikely.

I have not made the C standard. You point out yourself a possible explicit
culprit: struct randomization. That by itself shows that you can not rely on two elements of different structures having the same alignment, which is pretty much exactly what I said (and may explain why observing the problem seemed to at least somewhat depend on the weather).

Quoted 5 lines

I found this excellent quote on Hacker News this morning:

"I think the attitude of compiler writers is a good reason to fix the
spec so they can't keep ratfucking developers trying to get work done."

Qed.

Guenter

Quoted 8 lines
Quoted 2 lines

As result, this does and will cause a variety of build errors depending on the compiler version and compile flags.

The only thing I can think of that's actually a problem is that retarded struct randomization stuff.

Anyway, I'll move cleaning it up a little higher on the todo list.

author maintainer
#65 lore raw

Re: [RFC][PATCH 7/7] sched: Replace rq::wake_list

Quoted 2 lines · Guenter Roeck

I have not made the C standard. You point out yourself a possible explicit
culprit: struct randomization.

The randomization crud is very much outside the C spec.

Quoted 2 lines

That by itself shows that you can not rely
on two elements of different structures having the same alignment,

Randomization does not change the alignment, if it were to do that it would break all sorts. All it does is change the order of elements.

Quoted 2 lines

which is pretty much exactly what I said (and may explain why observing the problem seemed to at least somewhat depend on the weather).

A normal C compiler will have deterministic layout, otherwise unions and union based type punning would not be a thing.