Skip to contentSkip to conversation

rcu-tasks: Remove smp_mb() in rcu_spawn_tasks_kthread_generic()

PATCH Superseded Zqiang 2 participants 6 messages Jul 10 – Jul 11 RCU

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Zqiang

author reviewer
#0 lore raw

PATCH 1/1 rcu-tasks: Remove smp_mb() in rcu_spawn_tasks_kthread_generic()

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic() from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...);
->smp_mb(); /* Ensure others see full kthread. */
->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

							call_rcu_tasks()
					      		->if (READ_ONCE(rcu_tasks_kthread_ptr))
					         	  ->wake_up(&rcu_tasks_cbs_wq)
							    ->try_to_wake_up()
							      lock pi_lock
							      ->smp_mb__after_spinlock()
								//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

---
 kernel/rcu/tasks.h | 2 --
 1 file changed, 2 deletions(-)
kernel/rcu/tasks.h +0 -2
index 5a8db89135eb..18ab335665e3 100644
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 	}
 	rcu_read_unlock();
-	/* We can't create the thread unless interrupts are enabled. */
 	if (needwake && READ_ONCE(rtp->kthread_ptr))
 		irq_work_queue(&rtpcp->rtp_irq_work);
 }
@@ -681,7 +680,6 @@ static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
 	t = kthread_run(rcu_tasks_kthread, rtp, "%s_kthread", rtp->kname);
 	if (WARN_ONCE(IS_ERR(t), "%s: Could not start %s grace-period kthread, OOM is now expected behavior\n", __func__, rtp->name))
 		return;
-	smp_mb(); /* Ensure others see full kthread. */
 }
 
 #ifndef CONFIG_TINY_RCU
Signature · 1 line
2.17.1
maintainer
#1 lore raw
Quoted 33 lines · Zqiang

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic() from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...);
->smp_mb(); /* Ensure others see full kthread. */
->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

							call_rcu_tasks()
					      		->if (READ_ONCE(rcu_tasks_kthread_ptr))
					         	  ->wake_up(&rcu_tasks_cbs_wq)
							    ->try_to_wake_up()
							      lock pi_lock
							      ->smp_mb__after_spinlock()
								//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

Quoted 3 lines

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

Very good, a removal of an smp_mb() that also simplifies analysis!

One question...

kernel/rcu/tasks.h static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, · line 398

Quoted 13 lines
---
 kernel/rcu/tasks.h | 2 --
 1 file changed, 2 deletions(-)
kernel/rcu/tasks.h +0 -1
index 5a8db89135eb..18ab335665e3 100644
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 		raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 	}
 	rcu_read_unlock();
-	/* We can't create the thread unless interrupts are enabled. */

Why remove this comment? People reading this code might wonder why we are not just invoking rcu_spawn_tasks_kthread_generic() to create the task here.

Now, if you say that the current comment is confusing because if that kthread already exists, we also defer the wakeup, fair enough. But in that case, could you please propose a replacement comment, perhaps as a separate patch?

Thanx, Paul

Quoted 14 lines
kernel/rcu/tasks.h +0 -1
 	if (needwake && READ_ONCE(rtp->kthread_ptr))
 		irq_work_queue(&rtpcp->rtp_irq_work);
 }
@@ -681,7 +680,6 @@ static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
 	t = kthread_run(rcu_tasks_kthread, rtp, "%s_kthread", rtp->kname);
 	if (WARN_ONCE(IS_ERR(t), "%s: Could not start %s grace-period kthread, OOM is now expected behavior\n", __func__, rtp->name))
 		return;
-	smp_mb(); /* Ensure others see full kthread. */
 }
 
 #ifndef CONFIG_TINY_RCU
Signature · 1 line
2.17.1
author reviewer
#2 lore raw
Quoted 60 lines · Zqiang
Quoted 35 lines

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic()
from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...); ->smp_mb(); /* Ensure others see full kthread. */ ->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

call_rcu_tasks()
->if (READ_ONCE(rcu_tasks_kthread_ptr))
->wake_up(&rcu_tasks_cbs_wq)
->try_to_wake_up()
lock pi_lock
->smp_mb__after_spinlock()
//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

Quoted 4 lines

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

Very good, a removal of an smp_mb() that also simplifies analysis!

One question...

Quoted 5 lines
---
 kernel/rcu/tasks.h | 2 --
 1 file changed, 2 deletions(-)
Quoted 10 lines

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 5a8db89135eb..18ab335665e3 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
}
rcu_read_unlock();
- /* We can't create the thread unless interrupts are enabled. */

Quoted 8 lines

Why remove this comment? People reading this code might wonder why we are not just invoking rcu_spawn_tasks_kthread_generic() to create the task here.

Now, if you say that the current comment is confusing because if that kthread already exists, we also defer the wakeup, fair enough. But in that case, could you please propose a replacement comment, perhaps as a separate patch?

How about the following comment:

/* Use irq_work do deferred wakeup to avoid potential lockdep splat */

If directly use rcuwait_wake_up() to wakup, the lockdep will possible spalt, (for example we hook trace_sched_wakeup() tracepoint, and in our hook func inovke call_rcu_tasks())

Thanks
Zqiang

Quoted 19 lines

Thanx, Paul

Quoted 15 lines

if (needwake && READ_ONCE(rtp->kthread_ptr))
irq_work_queue(&rtpcp->rtp_irq_work);
}
@@ -681,7 +680,6 @@ static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
t = kthread_run(rcu_tasks_kthread, rtp, "%s_kthread", rtp->kname);
if (WARN_ONCE(IS_ERR(t), "%s: Could not start %s grace-period kthread, OOM is now expected behavior\n", __func__, rtp->name)) return;
- smp_mb(); /* Ensure others see full kthread. */
}

#ifndef CONFIG_TINY_RCU
--
2.17.1

maintainer
#3 lore raw
Quoted 69 lines · Zqiang
Quoted 60 lines · Zqiang
Quoted 35 lines

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic()
from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...); ->smp_mb(); /* Ensure others see full kthread. */ ->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

call_rcu_tasks()
->if (READ_ONCE(rcu_tasks_kthread_ptr))
->wake_up(&rcu_tasks_cbs_wq)
->try_to_wake_up()
lock pi_lock
->smp_mb__after_spinlock()
//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

Quoted 4 lines

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

Very good, a removal of an smp_mb() that also simplifies analysis!

One question...

Quoted 5 lines
---
 kernel/rcu/tasks.h | 2 --
 1 file changed, 2 deletions(-)
Quoted 10 lines

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 5a8db89135eb..18ab335665e3 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
}
rcu_read_unlock();
- /* We can't create the thread unless interrupts are enabled. */

Quoted 8 lines

Why remove this comment? People reading this code might wonder why we are not just invoking rcu_spawn_tasks_kthread_generic() to create the task here.

Now, if you say that the current comment is confusing because if that kthread already exists, we also defer the wakeup, fair enough. But in that case, could you please propose a replacement comment, perhaps as a separate patch?

Quoted 7 lines

How about the following comment:

/* Use irq_work do deferred wakeup to avoid potential lockdep splat */

If directly use rcuwait_wake_up() to wakup, the lockdep will possible spalt, (for example we hook trace_sched_wakeup() tracepoint, and in our hook func inovke call_rcu_tasks())

Fair enough. But the reason for the READ_ONCE(rtp->kthread_ptr) part of the condition of that "if" statement is that we cannot create a kthread with interrupts disabled, either because it is too early in boot or because one of the scheduler locks might be held.

Maybe that is obvious? But we did add that comment at some point.

Thoughts?

Thanx, Paul

Quoted 23 lines

Thanks
Zqiang

Quoted 19 lines

Thanx, Paul

Quoted 15 lines

if (needwake && READ_ONCE(rtp->kthread_ptr))
irq_work_queue(&rtpcp->rtp_irq_work);
}
@@ -681,7 +680,6 @@ static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
t = kthread_run(rcu_tasks_kthread, rtp, "%s_kthread", rtp->kname);
if (WARN_ONCE(IS_ERR(t), "%s: Could not start %s grace-period kthread, OOM is now expected behavior\n", __func__, rtp->name)) return;
- smp_mb(); /* Ensure others see full kthread. */
}

#ifndef CONFIG_TINY_RCU
--
2.17.1

author reviewer
#4 lore raw
Quoted 80 lines · Zqiang
Quoted 70 lines · Zqiang
Quoted 35 lines

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic() from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...);
->smp_mb(); /* Ensure others see full kthread. */
->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

call_rcu_tasks()
->if (READ_ONCE(rcu_tasks_kthread_ptr))
->wake_up(&rcu_tasks_cbs_wq)
->try_to_wake_up()
lock pi_lock
->smp_mb__after_spinlock()
//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

Quoted 4 lines

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

Very good, a removal of an smp_mb() that also simplifies analysis!

One question...

Quoted 10 lines

---
kernel/rcu/tasks.h | 2 --

1 file changed, 2 deletions(-)
kernel/rcu/tasks.h +0 -0
index 5a8db89135eb..18ab335665e3 100644

kernel/rcu/tasks.h static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, · line 398

Quoted 5 lines
kernel/rcu/tasks.h +0 -1
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 }
 rcu_read_unlock();
- /* We can't create the thread unless interrupts are enabled. */
 

Why remove this comment? People reading this code might wonder why we are not just invoking rcu_spawn_tasks_kthread_generic() to create the task here.

Now, if you say that the current comment is confusing because if that kthread already exists, we also defer the wakeup, fair enough. But in that case, could you please propose a replacement comment, perhaps as a separate patch?

Quoted 8 lines

How about the following comment:

/* Use irq_work do deferred wakeup to avoid potential lockdep splat */

If directly use rcuwait_wake_up() to wakup, the lockdep will possible spalt, (for example we hook trace_sched_wakeup() tracepoint, and in our hook func inovke call_rcu_tasks())

Quoted 8 lines

Fair enough. But the reason for the READ_ONCE(rtp->kthread_ptr) part of the condition of that "if" statement is that we cannot create a kthread with interrupts disabled, either because it is too early in boot or because one of the scheduler locks might be held.

Maybe that is obvious? But we did add that comment at some point.

Thoughts?

Ok, I find that:

c63eb17ff06 ("rcu: Create call_rcu_tasks() kthread at boot time") 4929c913bda ("rcu: Make call_rcu_tasks() tolerate first call with irqs disabled")

As a separate patch send, the update comments are as follows:

"We can't create the kthread with interrupts disabled, or a sinlock might be held, create the kthread moved at core_initcall() time, at the same time, use the irq_work does deferred wakeup to avoid potential lockdep splat"

Thanks
Zqiang

Quoted 28 lines

Thanx, Paul

Quoted 24 lines

Thanks
Zqiang

Thanx, Paul

Quoted 15 lines
kernel/rcu/tasks.h +0 -0
 if (needwake && READ_ONCE(rtp->kthread_ptr))
 irq_work_queue(&rtpcp->rtp_irq_work);

}

kernel/rcu/tasks.h +0 -1
@@ -681,7 +680,6 @@ static void __init rcu_spawn_tasks_kthread_generic(struct rcu_tasks *rtp)
 t = kthread_run(rcu_tasks_kthread, rtp, "%s_kthread", rtp->kname);
 if (WARN_ONCE(IS_ERR(t), "%s: Could not start %s grace-period kthread, OOM is now expected behavior\n", __func__, rtp->name))
 return;
- smp_mb(); /* Ensure others see full kthread. */
 }
 
 #ifndef CONFIG_TINY_RCU
Signature · 1 line
2.17.1
maintainer
#5 lore raw
Quoted 94 lines · Zqiang
Quoted 80 lines · Zqiang
Quoted 70 lines · Zqiang
Quoted 35 lines

The smp_mb() after kthread_run() in rcu_spawn_tasks_kthread_generic() from early 'commit eacd6f04a133 ("rcu-tasks: Move Tasks RCU to its own file")' and 'commit 84a8f446ffd7 ("rcu: Defer rcu_tasks_kthread() creation till first call_rcu_tasks()")', the pairing as follows:

rcu_spawn_tasks_kthread()
->t = kthread_run(rcu_tasks_kthread, ...);
->smp_mb(); /* Ensure others see full kthread. */
->WRITE_ONCE(rcu_tasks_kthread_ptr, t);

call_rcu_tasks()
->if (READ_ONCE(rcu_tasks_kthread_ptr))
->wake_up(&rcu_tasks_cbs_wq)
->try_to_wake_up()
lock pi_lock
->smp_mb__after_spinlock()
//see full kthread

Currently, the 'commit d119357d0743 ("rcu-tasks: Treat only synchronous grace periods urgently")' moved kthread_ptr assignment into the rcu_tasks_kthread(), the following pairings are sufficient:

The rq->lock/unlock from wake_up_process() in kthread_run() and __schedule() provides a memory barrier when the kthread is first scheduled, this ensures the kthread itself observes all of the kthread's initialization.

The kthread's smp_store_release(&rtp->kthread_ptr, ...) in rcu_tasks_kthread() and smp_load_acquire(&rtp->kthread_ptr) in call_rcu_tasks_generic() compose release/acquire pairing, the cumulativity of smp_store_release() propagates visibility of the kthread's initialization through the scheduler chain.

Quoted 4 lines

This commit therefore remove smp_mb() in rcu_spawn_tasks_kthread_generic().

Very good, a removal of an smp_mb() that also simplifies analysis!

One question...

Quoted 10 lines

---
kernel/rcu/tasks.h | 2 --

1 file changed, 2 deletions(-)
kernel/rcu/tasks.h +0 -0
index 5a8db89135eb..18ab335665e3 100644

kernel/rcu/tasks.h static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func, · line 398

Quoted 5 lines
kernel/rcu/tasks.h +0 -1
@@ -395,7 +395,6 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
 raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
 }
 rcu_read_unlock();
- /* We can't create the thread unless interrupts are enabled. */
 

Why remove this comment? People reading this code might wonder why we are not just invoking rcu_spawn_tasks_kthread_generic() to create the task here.

Now, if you say that the current comment is confusing because if that kthread already exists, we also defer the wakeup, fair enough. But in that case, could you please propose a replacement comment, perhaps as a separate patch?

Quoted 8 lines

How about the following comment:

/* Use irq_work do deferred wakeup to avoid potential lockdep splat */

If directly use rcuwait_wake_up() to wakup, the lockdep will possible spalt, (for example we hook trace_sched_wakeup() tracepoint, and in our hook func inovke call_rcu_tasks())

Quoted 8 lines

Fair enough. But the reason for the READ_ONCE(rtp->kthread_ptr) part of the condition of that "if" statement is that we cannot create a kthread with interrupts disabled, either because it is too early in boot or because one of the scheduler locks might be held.

Maybe that is obvious? But we did add that comment at some point.

Thoughts?

Ok, I find that:

c63eb17ff06 ("rcu: Create call_rcu_tasks() kthread at boot time") 4929c913bda ("rcu: Make call_rcu_tasks() tolerate first call with irqs disabled")

Quoted 6 lines

As a separate patch send, the update comments are as follows:

"We can't create the kthread with interrupts disabled, or a sinlock might be held, create the kthread moved at core_initcall() time, at the same time, use the irq_work does deferred wakeup to avoid potential lockdep splat"

Very good, looking forward to the updates!

Thanx, Paul