Skip to contentSkip to conversation

rcu: Make rcu_normal_after_boot writable on RT

PATCH In review Juri Lelli 3 participants 7 messages Aug 5, 2021 – Aug 9, 2021 RCU

lore
Share & tools
lore

Juri Lelli

author
#0 lore raw

PATCH 1/1 rcu: Make rcu_normal_after_boot writable on RT

Certain configurations (e.g., systems that make heavy use of netns) need to use synchronize_rcu_expedited() to service RCU grace periods even after boot.

Even though synchronize_rcu_expedited() has been traditionally considered harmful for RT for the heavy use of IPIs, it is perfectly usable under certain conditions (e.g. nohz_full).

Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep its default value to 1 (enabled) to avoid regressions. Users who need synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline.

---
 kernel/rcu/update.c | 2 --
 1 file changed, 2 deletions(-)
kernel/rcu/update.c +0 -2
index c21b38cc25e9..0fdbf937edac 100644
@@ -57,9 +57,7 @@
 module_param(rcu_expedited, int, 0);
 module_param(rcu_normal, int, 0);
 static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
-#ifndef CONFIG_PREEMPT_RT
 module_param(rcu_normal_after_boot, int, 0);
-#endif
 #endif /* #ifndef CONFIG_TINY_RCU */
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
Signature · 1 line
2.31.1
maintainer
#1 lore raw
Quoted 8 lines · Juri Lelli

Certain configurations (e.g., systems that make heavy use of netns) need to use synchronize_rcu_expedited() to service RCU grace periods even after boot.

Even though synchronize_rcu_expedited() has been traditionally considered harmful for RT for the heavy use of IPIs, it is perfectly usable under certain conditions (e.g. nohz_full).

Quoted 6 lines

Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep its default value to 1 (enabled) to avoid regressions. Users who need synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline.

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Thanx, Paul

Quoted 21 lines
---
 kernel/rcu/update.c | 2 --
 1 file changed, 2 deletions(-)
kernel/rcu/update.c +0 -2
index c21b38cc25e9..0fdbf937edac 100644
@@ -57,9 +57,7 @@
 module_param(rcu_expedited, int, 0);
 module_param(rcu_normal, int, 0);
 static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
-#ifndef CONFIG_PREEMPT_RT
 module_param(rcu_normal_after_boot, int, 0);
-#endif
 #endif /* #ifndef CONFIG_TINY_RCU */
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
Signature · 1 line
2.31.1
maintainer
#2 lore raw
Quoted 14 lines · Paul E. McKenney
Quoted 13 lines · Juri Lelli

Certain configurations (e.g., systems that make heavy use of netns) need to use synchronize_rcu_expedited() to service RCU grace periods even after boot.

Even though synchronize_rcu_expedited() has been traditionally considered harmful for RT for the heavy use of IPIs, it is perfectly usable under certain conditions (e.g. nohz_full).

Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep its default value to 1 (enabled) to avoid regressions. Users who need synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline.

Quoted 8 lines
Quoted 1 line

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Ah, and this will also need to be reflected in the WARN_ON_ONCE() in synchronize_rcu_expedited_wait() in kernel/rcu/tree_exp.h.

Thanx, Paul

Quoted 21 lines
Quoted 21 lines
---
 kernel/rcu/update.c | 2 --
 1 file changed, 2 deletions(-)
kernel/rcu/update.c +0 -2
index c21b38cc25e9..0fdbf937edac 100644
@@ -57,9 +57,7 @@
 module_param(rcu_expedited, int, 0);
 module_param(rcu_normal, int, 0);
 static int rcu_normal_after_boot = IS_ENABLED(CONFIG_PREEMPT_RT);
-#ifndef CONFIG_PREEMPT_RT
 module_param(rcu_normal_after_boot, int, 0);
-#endif
 #endif /* #ifndef CONFIG_TINY_RCU */
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
Signature · 1 line
2.31.1
author
#3 lore raw

Hi,

Thanks for reviewing Paul.

Quoted 19 lines · Paul E. McKenney
Quoted 18 lines · Paul E. McKenney
Quoted 8 lines · Juri Lelli

Certain configurations (e.g., systems that make heavy use of netns) need to use synchronize_rcu_expedited() to service RCU grace periods even after boot.

Even though synchronize_rcu_expedited() has been traditionally considered harmful for RT for the heavy use of IPIs, it is perfectly usable under certain conditions (e.g. nohz_full).

Quoted 6 lines

Make rcupdate.rcu_normal_after_boot= again writeable on RT, but keep its default value to 1 (enabled) to avoid regressions. Users who need synchronize_rcu_expedited() will boot with rcupdate.rcu_normal_after_ boot=0 in the kernel cmdline.

Makes sense to me!

Quoted 7 lines
Quoted 4 lines

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Ah, and this will also need to be reflected in the WARN_ON_ONCE() in synchronize_rcu_expedited_wait() in kernel/rcu/tree_exp.h.

Indeed. Will add the change as soon as I receive indication about your first point.

Best,
Juri

#4 lore raw
In reply to Paul E. McKenney
Quoted 6 lines · Paul E. McKenney

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more and more synchronize_rcu_expedited() users popping up. I would like to keep that part (default value) since it good to have for most users.

I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate use cases for using "rcu_normal_after_boot = 0".
Paul suggested initially to restrict that option for PREEMPT_RT and I would follow here Paul's guidance to either remove it or restrict it to NO_HZ_FULL in RT's case (as suggested).

Quoted 1 line

Thanx, Paul

Sebastian

maintainer
#5 lore raw
Quoted 12 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Paul E. McKenney

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more and more synchronize_rcu_expedited() users popping up. I would like to keep that part (default value) since it good to have for most users.

Quoted 5 lines

I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate use cases for using "rcu_normal_after_boot = 0".
Paul suggested initially to restrict that option for PREEMPT_RT and I would follow here Paul's guidance to either remove it or restrict it to NO_HZ_FULL in RT's case (as suggested).

Given what I know now, I suggest the following:

o Restrict the option to !PREEMPT_RT unless NO_HZ_FULL.
Maybe "!defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL)".

If there is some non-NO_HZ_FULL PREEMPT_RT configuration that tolerates expedited grace periods, this would need to change.

o Change the permissions from "0" to "0444", if desired. If you
would rather not, I can do this in a follow-up patch. (No idea why I let such an ugly serviceability issue through, but the previous pair of module_param() instances have the same problem.)

Anything I am missing?

Thanx, Paul

author
#6 lore raw
Quoted 27 lines · Paul E. McKenney
Quoted 12 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Paul E. McKenney

Makes sense to me!

But would another of the -rt people be willing to give an Acked-by? For example, maybe they would prefer this kernel boot parameter to be exposed only if (!PREEMPT_RT || NO_HZ_FULL). Or are there !NO_HZ_FULL situations where rcu_normal_after_boot makes sense?

Julia crafted that "rcu_normal_after_boot = 1" for RT after we had more and more synchronize_rcu_expedited() users popping up. I would like to keep that part (default value) since it good to have for most users.

Quoted 5 lines

I don't mind removing CONFIG_PREEMPT_RT part here if there are legitimate use cases for using "rcu_normal_after_boot = 0".
Paul suggested initially to restrict that option for PREEMPT_RT and I would follow here Paul's guidance to either remove it or restrict it to NO_HZ_FULL in RT's case (as suggested).

Given what I know now, I suggest the following:

o Restrict the option to !PREEMPT_RT unless NO_HZ_FULL.
Maybe "!defined(CONFIG_PREEMPT_RT) || defined(CONFIG_NO_HZ_FULL)".

If there is some non-NO_HZ_FULL PREEMPT_RT configuration that tolerates expedited grace periods, this would need to change.

Quoted 6 lines

o Change the permissions from "0" to "0444", if desired. If you
would rather not, I can do this in a follow-up patch. (No idea why I let such an ugly serviceability issue through, but the previous pair of module_param() instances have the same problem.)

Anything I am missing?

Not that I can think of right now. :)

Will implement your suggestions and submit v2 soon. Thank again to you and Sebastian for the review!

Best,
Juri