Skip to contentSkip to conversation

rcu: When rcuog kthreads is in polling mode, wakeup waitqueue is not requried

PATCH In review Zqiang 2 participants 5 messages Jan 28, 2022 – Jan 30, 2022 RCU

lore
Share & tools
lore

Zqiang

author
#0 lore raw

PATCH 1/1 rcu: When rcuog kthreads is in polling mode, wakeup waitqueue is not requried

When grace period cleanup, the rcuog kthreads that waiting in sq waitqueue will be awakened, however if the 'rcu_nocb_poll' is set, the sq waitqueue always empty, so if 'rcu_nocb_poll' is set, return directly.

---
 kernel/rcu/tree_nocb.h | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/tree_nocb.h +2 -0
index 636d0546a4e9..9e106c590e56 100644
@@ -201,6 +201,8 @@ static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
  */
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq)
 {
+	if (rcu_nocb_poll)
+		return;
 	swake_up_all(sq);
 }
 
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 4 lines · Zqiang

When grace period cleanup, the rcuog kthreads that waiting in sq waitqueue will be awakened, however if the 'rcu_nocb_poll' is set, the sq waitqueue always empty, so if 'rcu_nocb_poll' is set, return directly.

This does decrease grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y and booted with the rcu_nocb_poll kernel boot parameter set. On the other hand, it increases grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y but booted without the rcu_nocb_poll kernel boot parameter set.

Last I checked, more kernels were booted without the rcu_nocb_poll kernel boot parameter set. If this is still the case, this patch would slow things down for most systems.

Or are there now lots of systems booted with rcu_nocb_poll?

Thanx, Paul

Quoted 21 lines
---
 kernel/rcu/tree_nocb.h | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/tree_nocb.h +2 -0
index 636d0546a4e9..9e106c590e56 100644
@@ -201,6 +201,8 @@ static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
  */
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq)
 {
+	if (rcu_nocb_poll)
+		return;
 	swake_up_all(sq);
 }
 
Signature · 1 line
2.25.1
#2 lore raw
Quoted 10 lines · Zqiang

When grace period cleanup, the rcuog kthreads that waiting in sq waitqueue will be awakened, however if the 'rcu_nocb_poll' is set, the sq waitqueue always empty, so if 'rcu_nocb_poll' is set, return directly.

This does decrease grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y and booted with the rcu_nocb_poll kernel boot parameter set. On the other hand, it increases grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y but booted without the rcu_nocb_poll kernel boot parameter set.

Last I checked, more kernels were booted without the rcu_nocb_poll kernel boot parameter set. If this is still the case, this patch would slow things down for most systems.

Or are there now lots of systems booted with rcu_nocb_poll?

Hi Paul
I found that some of our customers configured rcu_nocb_poll startup parameters under Preempt-RT kernel. at each grace period cleanup, we'll all wakeup sq waitqueue, however when rcuog kthreads is in polling mode, the wakeup operation doesn't required, because the sq waitqueue always empty.

Thanks,
Zqiang

Quoted 23 lines

Thanx, Paul

---
 kernel/rcu/tree_nocb.h | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/tree_nocb.h index renamed from kernel/rcu/tree_nocb.h +0 -0

636d0546a4e9..9e106c590e56 100644

kernel/rcu/tree_nocb.h +2 -1
@@ -201,6 +201,8 @@ static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
  */
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq)  {
+	if (rcu_nocb_poll)
+		return;
 	swake_up_all(sq);
 }
 
--

2.25.1

maintainer
#3 lore raw
Quoted 9 lines · On Sat, Jan 29, 2022 at 05:55:34AM +0000, Zhang, Qiang1 wrote:
Quoted 7 lines · Zqiang

When grace period cleanup, the rcuog kthreads that waiting in sq waitqueue will be awakened, however if the 'rcu_nocb_poll' is set, the sq waitqueue always empty, so if 'rcu_nocb_poll' is set, return directly.

This does decrease grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y and booted with the rcu_nocb_poll kernel boot parameter set. On the other hand, it increases grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y but booted without the rcu_nocb_poll kernel boot parameter set.

Quoted 8 lines
Quoted 3 lines

Last I checked, more kernels were booted without the rcu_nocb_poll kernel boot parameter set. If this is still the case, this patch would slow things down for most systems.

Or are there now lots of systems booted with rcu_nocb_poll?

Hi Paul
I found that some of our customers configured rcu_nocb_poll startup parameters under Preempt-RT kernel. at each grace period cleanup, we'll all wakeup sq waitqueue, however when rcuog kthreads is in polling mode, the wakeup operation doesn't required, because the sq waitqueue always empty.

OK, fair enough. But was there any difference in performance measurable at the system level? Let's take a look at swake_up_all():

	void swake_up_all(struct swait_queue_head *q)
	{
		struct swait_queue *curr;
		LIST_HEAD(tmp);

		raw_spin_lock_irq(&q->lock);
		list_splice_init(&q->task_list, &tmp);
		while (!list_empty(&tmp)) {
			curr = list_first_entry(&tmp, typeof(*curr), task_list);

			wake_up_state(curr->task, TASK_NORMAL);
			list_del_init(&curr->task_list);

			if (list_empty(&tmp))
				break;

			raw_spin_unlock_irq(&q->lock);
			raw_spin_lock_irq(&q->lock);
		}
		raw_spin_unlock_irq(&q->lock);
	}

If the list is empty, we acquire an uncontended lock, splice an empty list, check a pair of pointers for equality, and release that lock. We do this once per 16 CPUs per grace period, which normally will be every few milliseconds or less frequently.

What is the system-level performance difference and how did you measure it?

Please don't get me wrong. If this really is causing your users trouble, we clearly do need to fix it. But if so, let's do so in a way that doesn't risk penalizing the many users who do not set rcu_nocb_poll.

Thanx, Paul

Quoted 26 lines

Thanks,
Zqiang

Quoted 23 lines

Thanx, Paul

---
 kernel/rcu/tree_nocb.h | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/tree_nocb.h index renamed from kernel/rcu/tree_nocb.h +0 -0

636d0546a4e9..9e106c590e56 100644

kernel/rcu/tree_nocb.h +2 -1
@@ -201,6 +201,8 @@ static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
  */
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq)  {
+	if (rcu_nocb_poll)
+		return;
 	swake_up_all(sq);
 }
 
--

2.25.1

#4 lore raw
Quoted 9 lines · On Sat, Jan 29, 2022 at 05:55:34AM +0000, Zhang, Qiang1 wrote:
Quoted 7 lines · Zqiang

When grace period cleanup, the rcuog kthreads that waiting in sq waitqueue will be awakened, however if the 'rcu_nocb_poll' is set, the sq waitqueue always empty, so if 'rcu_nocb_poll' is set, return directly.

This does decrease grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y and booted with the rcu_nocb_poll kernel boot parameter set. On the other hand, it increases grace-period-cleanup overhead in kernels built with CONFIG_RCU_NOCB_CPU=y but booted without the rcu_nocb_poll kernel boot parameter set.

Quoted 37 lines
Quoted 3 lines

Last I checked, more kernels were booted without the rcu_nocb_poll kernel boot parameter set. If this is still the case, this patch would slow things down for most systems.

Or are there now lots of systems booted with rcu_nocb_poll?

Hi Paul
I found that some of our customers configured rcu_nocb_poll startup parameters under Preempt-RT kernel. at each grace period cleanup, we'll all wakeup sq waitqueue, however
when rcuog kthreads is in polling mode, the wakeup operation doesn't required, because the sq waitqueue always empty.

Quoted 28 lines
Quoted 28 lines

OK, fair enough. But was there any difference in performance measurable at the system level? Let's take a look at swake_up_all():>>>

	void swake_up_all(struct swait_queue_head *q)
	{
		struct swait_queue *curr;
		LIST_HEAD(tmp);

		raw_spin_lock_irq(&q->lock);
		list_splice_init(&q->task_list, &tmp);
		while (!list_empty(&tmp)) {
			curr = list_first_entry(&tmp, typeof(*curr), task_list);

			wake_up_state(curr->task, TASK_NORMAL);
			list_del_init(&curr->task_list);

			if (list_empty(&tmp))
				break;

			raw_spin_unlock_irq(&q->lock);
			raw_spin_lock_irq(&q->lock);
		}
		raw_spin_unlock_irq(&q->lock);
	}

If the list is empty, we acquire an uncontended lock, splice an empty list, check a pair of pointers for equality, and release that lock. We do this once per 16 CPUs per grace period, which normally will be every few milliseconds or less frequently.

What is the system-level performance difference and how did you measure it?

Sorry I ignored that. I don't measure performance differences at the system level,

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Please don't get me wrong. If this really is causing your users trouble, we clearly do need to fix it. But if so, let's do so in a way that doesn't risk penalizing the many users who do not set rcu_nocb_poll.

Thank you for detailed analysis . the original intention of my modification is avoid unnecessary wake-up operations,
when rcu_nocb_poll is set. In polling mode, the rcuog kthreads don't use sq waitqueue, however, every time the grace period cleanup, all rnp nodes must be traversed for wake-up operation. after that, I will do the test.

Thanks
Zqiang

Quoted 29 lines
Quoted 3 lines
Quoted 3 lines

Thanx, Paul

Thanks,
Zqiang

Quoted 23 lines

Thanx, Paul

---
 kernel/rcu/tree_nocb.h | 2 ++
 1 file changed, 2 insertions(+)
kernel/rcu/tree_nocb.h index renamed from kernel/rcu/tree_nocb.h +0 -0

636d0546a4e9..9e106c590e56 100644

kernel/rcu/tree_nocb.h +2 -1
@@ -201,6 +201,8 @@ static void rcu_lockdep_assert_cblist_protected(struct rcu_data *rdp)
  */
 static void rcu_nocb_gp_cleanup(struct swait_queue_head *sq)   
+	if (rcu_nocb_poll)
+		return;
 	swake_up_all(sq);
 }
 
--

2.25.1