handle kthread_prio kernel cmdline parameter consistently
Subsystems (2)
cover handle kthread_prio kernel cmdline parameter consistently
When CONFIG_PREEMPT_RT=y, RCU_SOFTIRQ work is moved to dedicated rcuc per-core threads. The rcutree.kthread_prio kernel cmdline parameter controls the priority of those threads.
A provided kthread_prio kernel cmdline parameter also elevates the priority of rcuog threads, but only if they are launched from rcu_spawn_gp_kthread(), not if they are launched from rcu_spawn_one_nocb_kthread(). Fix this inconsistency.
When CONFIG_RCU_NOCB_CPU=y, an rcu_nocbs command-line parameter offloads the work of rcuc on designated cores to new rcuop threads that are not pinned to the cores whose expired grace-period timer callbacks they run. While rcuop threads have the same function as rcuc threads, their priority is not controlled by the kthread_prio parameter. Add this feature and update the documentation accordingly.
Alison Chaiken (4):
RCU: move kthread_prio bounds-check to a separate function
RCU: make priority of grace-period thread consistent
RCU: elevate priority of offloaded callback threads
RCU: update documentation regarding kthread_prio cmdline parameter
.../admin-guide/kernel-parameters.txt | 2 + kernel/rcu/tree.c | 42 +++++++++++-------- kernel/rcu/tree_nocb.h | 7 ++++ 3 files changed, 34 insertions(+), 17 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/4 RCU: move kthread_prio bounds-check to a separate function
Move the bounds-check of the kthread_prio cmdline parameter to a new function in order to faciliate a different callsite.
--- kernel/rcu/tree.c | 39 +++++++++++++++++++++++---------------- 1 file changed, 23 insertions(+), 16 deletions(-)
kernel/rcu/tree.c +23 -16
@@ -4404,26 +4404,10 @@ static int rcu_pm_notify(struct notifier_block *self,static int __init rcu_spawn_gp_kthread (void ) {unsigned long flags; -int kthread_prio_in = kthread_prio;struct rcu_node *rnp;struct sched_param sp;struct task_struct *t; -/* Force priority into range. */ -if (IS_ENABLED (CONFIG_RCU_BOOST ) && kthread_prio <2 - &&IS_BUILTIN (CONFIG_RCU_TORTURE_TEST )) - kthread_prio =2 ; -else if (IS_ENABLED (CONFIG_RCU_BOOST ) && kthread_prio <1 ) - kthread_prio =1 ; -else if (kthread_prio <0 ) - kthread_prio =0 ; -else if (kthread_prio >99 ) - kthread_prio =99 ; - -if (kthread_prio != kthread_prio_in) -pr_alert ("rcu_spawn_gp_kthread(): Limited prio to %d from %d\n" , - kthread_prio, kthread_prio_in); - rcu_scheduler_fully_active =1 ; t =kthread_create (rcu_gp_kthread,NULL ,"%s" , rcu_state.name);if (WARN_ONCE (IS_ERR (t),"%s: Could not start grace-period kthread, OOM is now expected behavior\n" , __func__))
@@ -4548,6 +4532,28 @@ static void __init rcu_init_one(void) } } +/* +* Force priority from the kernel command-line into range. +*/ +static void __init sanitize_kthread_prio (void ) +{ +int kthread_prio_in = kthread_prio; + +if (IS_ENABLED (CONFIG_RCU_BOOST ) && kthread_prio <2 + &&IS_BUILTIN (CONFIG_RCU_TORTURE_TEST )) + kthread_prio =2 ; +else if (IS_ENABLED (CONFIG_RCU_BOOST ) && kthread_prio <1 ) + kthread_prio =1 ; +else if (kthread_prio <0 ) + kthread_prio =0 ; +else if (kthread_prio >99 ) + kthread_prio =99 ; + +if (kthread_prio != kthread_prio_in) +pr_alert ("%s: Limited prio to %d from %d\n" , + __func__, kthread_prio, kthread_prio_in); +} +/* * Compute the rcu_node tree geometry from kernel parameters. This cannot * replace the definitions in tree.h because those are needed to size
@@ -4708,6 +4714,7 @@ void __init rcu_init(void)kfree_rcu_batch_init ();rcu_bootup_announce (); +sanitize_kthread_prio ();rcu_init_geometry ();rcu_init_one ();if (dump_tree)
Signature · 1 line
2.32.0
Paul E. McKenney
maintainerQuoted 16 lines · Alison Chaiken
When
CONFIG_PREEMPT_RT=y,RCU_SOFTIRQwork is moved to dedicated rcuc per-core threads. The rcutree.kthread_prio kernel cmdline parameter controls the priority of those threads.A provided
kthread_priokernel cmdline parameter also elevates the priority of rcuog threads, but only if they are launched fromrcu_spawn_gp_kthread(), not if they are launched fromrcu_spawn_one_nocb_kthread(). Fix this inconsistency.When
CONFIG_RCU_NOCB_CPU=y, anrcu_nocbscommand-line parameter offloads the work of rcuc on designated cores to new rcuop threads that are not pinned to the cores whose expired grace-period timer callbacks they run. While rcuop threads have the same function as rcuc threads, their priority is not controlled by thekthread_prioparameter. Add this feature and update the documentation accordingly.
Quoted 5 lines
Alison Chaiken (4):
RCU: movekthread_priobounds-check to a separate function
RCU: make priority of grace-period thread consistent
RCU: elevate priority of offloaded callback threads
RCU: update documentation regardingkthread_priocmdline parameter
I have queued these for review and testing, thank you!
2/4 appears to have been produced against a pre-merge-window mainline, so I had to adjust it a bit to queue it on -rcu. Please check to make sure that I did not mess anything up.
Thanx, Paul
PATCH 2/4 RCU: make priority of grace-period thread consistent
In reply to Alison ChaikenThe priority of RCU grace period threads is set to kthread_prio when they are launched from rcu_spawn_gp_kthread(). The same is not true of rcu_spawn_one_nocb_kthread(). Accordingly, add priority elevation to rcu_spawn_one_nocb_kthread().
--- kernel/rcu/tree_nocb.h | 4 ++++ 1 file changed, 4 insertions(+)
kernel/rcu/tree_nocb.h +4 -0
@@ -1183,7 +1183,9 @@ static void rcu_spawn_one_nocb_kthread(int cpu)struct rcu_data *rdp =per_cpu_ptr (&rcu_data, cpu);struct rcu_data *rdp_gp;struct task_struct *t; +struct sched_param sp; + sp.sched_priority = kthread_prio;/* * If this isn't a no-CBs CPU or if it already has an rcuo kthread, * then nothing to do.
@@ -1199,6 +1201,8 @@ static void rcu_spawn_one_nocb_kthread(int cpu)if (WARN_ONCE (IS_ERR (t),"%s: Could not start rcuo GP kthread, OOM is now expected behavior\n" , __func__))return ;WRITE_ONCE (rdp_gp->nocb_gp_kthread, t); +if (kthread_prio) +sched_setscheduler_nocheck (t,SCHED_FIFO , &sp); }/* Spawn the kthread for this CPU. */
Signature · 1 line
2.32.0
PATCH 3/4 RCU: elevate priority of offloaded callback threads
In reply to Alison ChaikenWhen CONFIG_PREEMPT_RT=y, the rcutree.kthread_prio command-line parameter signals initialization code to boost the priority of rcuc callbacks to the designated value. With the additional CONFIG_RCU_NOCB_CPU=y configuration and an additional rcu_nocbs command-line parameter, the callbacks on the listed cores are offloaded to new rcuop threads that are not pinned to the cores whose post-grace-period work is performed. While the rcuop threads perform the same function as the rcuc threads they offload, the kthread_prio parameter only boosts the priority of the rcuc threads. Fix this inconsistency by elevating rcuop threads to the same priority as rcuc ones.
--- kernel/rcu/tree.c | 2 +- kernel/rcu/tree_nocb.h | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-)
kernel/rcu/tree.c +1 -1
@@ -153,7 +153,7 @@ static void sync_sched_exp_online_cleanup(int cpu);static void check_cb_ovld_locked (struct rcu_data *rdp,struct rcu_node *rnp);static bool rcu_rdp_is_offloaded (struct rcu_data *rdp); -/* rcuc/rcub kthread realtime priority */ +/* rcuc/rcub/rcuop kthread realtime priority */ static int kthread_prio =IS_ENABLED (CONFIG_RCU_BOOST ) ?1 :0 ;module_param (kthread_prio, int,0444 );
kernel/rcu/tree_nocb.h +3 -0
@@ -1210,6 +1210,9 @@ static void rcu_spawn_one_nocb_kthread(int cpu) "rcuo%c/%d", rcu_state.abbr, cpu);if (WARN_ONCE (IS_ERR (t),"%s: Could not start rcuo CB kthread, OOM is now expected behavior\n" , __func__))return ; + +if (kthread_prio) +sched_setscheduler_nocheck (t,SCHED_FIFO , &sp);WRITE_ONCE (rdp->nocb_cb_kthread, t);WRITE_ONCE (rdp->nocb_gp_kthread, rdp_gp->nocb_gp_kthread); }
Signature · 1 line
2.32.0
PATCH 4/4 RCU: update documentation regarding kthread_prio cmdline parameter
In reply to Alison ChaikenInform readers that the priority of RCU no-callback threads will also be boosted.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ 1 file changed, 2 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -4439,6 +4439,8 @@ (the least-favored priority). Otherwise, when RCU_BOOST is not set, valid values are 0-99 and the default is zero (non-realtime operation). + When RCU_NOCB_CPU is set, also adjust the + priority of NOCB callback kthreads. rcutree.rcu_nocb_gp_stride= [KNL] Set the number of NOCB callback kthreads in
Signature · 1 line
2.32.0