cgroup/cpuset: Enable runtime modification of
Subsystems (9)
Waiman Long
maintainercover cgroup/cpuset: Enable runtime modification of
The "nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.
There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the nohz_full and rcu_nocbs boot kernel parameters.
This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the nohz_full boot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthread
The rcu_nocbs is actually a subset of nohz_full focusing just on the RCU part of the kernel noises. The WQ part has already been handled by the current cpuset code.
This series focuses on the tick and RCU part of the kernel noises by actively changing their internal data structures to track changes in the list of isolated CPUs used by cpuset isolated partitions.
The dynamic update of the lists of housekeeping CPUs at run time will also have impact on the other part of the kernel noises that reference the lists of housekeeping CPUs at run time.
The pending patch series on timer migration[1], when properly integrated will support the timer part too.
The CPU hotplug functionality of the Linux kernel is used to facilitate the runtime change of the nohz_full isolated CPUs with minimal code changes. The CPUs that need to be switched from non-isolated to isolated or vice versa will be brought offline first, making the necessary changes and then brought back online afterward.
The use of CPU hotplug, however, does have a slight drawback of freezing all the other CPUs in part of the offlining process using the stop machine feature of the kernel. That will cause a noticeable latency spikes in other running applications which may be significant to sensitive applications running on isolated CPUs in other isolated partitions at the time. Hopefully we can find a way to solve this problem in the future.
One possible workaround for this is to reserve a set of nohz_full isolated CPUs at boot time using the nohz_full boot command parameter. The bringing of those nohz_full reserved CPUs into and out of isolated partitions will not invoke CPU hotplug and hence will not cause unexpected latency spikes. These reserved CPUs will only be needed if there are other existing isolated partitions running critical applications at the time when an isolated partition needs to be created.
Patches 1-4 updates the CPU isolation code at kernel/sched/isolation.c to enable dynamic update of the lists of housekeeping CPUs.
Patch 5 introduces a new cpuhp_offline_cb() API for shutting down the given set of CPUs, running the given callback method and then bringing those CPUs back online again. This new API will block any incoming hotplug events from interfering this operation.
Patches 6-9 updates the cpuset partition code to use the new cpuhp API to shut down the affect CPUs, making changes to the housekeeping cpumasks and then bring those CPUs online afterward.
Patch 10 works around an issue in the DL server code that block the hotplug operation under certain configurations.
Patch 11-14 updates the timer tick and related code to enable proper updates to the set of CPUs requiring nohz_full dynticks support.
Patch 15 enables runtime modification to the set of isolated CPUs requiring RCU NO-CB CPU support with minor changes to the RCU code.
Patches 16-18 includes other miscellaneous updates to cpuset code and documentation.
This patch series is applied on top of some other cpuset patches[1] posted upstream recently.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
Waiman Long (18):
sched/isolation: Enable runtime update of housekeeping cpumasks sched/isolation: Call sched_tick_offload_init() when
HK_FLAG_KERNEL_NOISE is first set
sched/isolation: Use RCU to delay successive housekeeping cpumask
updates
sched/isolation: Add a debugfs file to dump housekeeping cpumasks cpu/hotplug: Add a new cpuhp_offline_cb() API cgroup/cpuset: Introduce a new top level isolcpus_update_mutex cgroup/cpuset: Allow overwriting HK_TYPE_DOMAIN housekeeping cpumask cgroup/cpuset: Use CPU hotplug to enable runtime nohz_full
modification
cgroup/cpuset: Revert "Include isolated cpuset CPUs in
cpu_is_isolated() check"
sched/core: Ignore DL BW deactivation error if in
cpuhp_offline_cb_mode
tick/nohz: Make nohz_full parameter optional
tick/nohz: Introduce tick_nohz_full_update_cpus() to update
tick_nohz_full_mask
tick/nohz: Allow runtime changes in full dynticks CPUs
tick: Pass timer tick job to an online HK CPU in tick_cpu_dying() cgroup/cpuset: Enable RCU NO-CB CPU offloading of newly isolated CPUs cgroup/cpuset: Don't set have_boot_nohz_full without any boot time
nohz_full CPU
cgroup/cpuset: Documentation updates & don't use CPU 0 for isolated
partition
cgroup/cpuset: Add pr_debug() statements for cpuhp_offline_cb() call
Documentation/admin-guide/cgroup-v2.rst | 33 +- .../admin-guide/kernel-parameters.txt | 19 +- include/linux/context_tracking.h | 8 +- include/linux/cpuhplock.h | 9 + include/linux/cpuset.h | 6 - include/linux/rcupdate.h | 2 + include/linux/sched/isolation.h | 9 +- include/linux/tick.h | 2 + kernel/cgroup/cpuset.c | 344 ++++++++++++------ kernel/context_tracking.c | 21 +- kernel/cpu.c | 47 +++ kernel/rcu/tree_nocb.h | 7 +- kernel/sched/core.c | 8 +- kernel/sched/debug.c | 32 ++ kernel/sched/isolation.c | 151 +++++++- kernel/sched/sched.h | 2 +- kernel/time/tick-common.c | 15 +- kernel/time/tick-sched.c | 24 +- .../selftests/cgroup/test_cpuset_prs.sh | 15 +- 19 files changed, 583 insertions(+), 171 deletions(-)
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 1/18 sched/isolation: Enable runtime update of housekeeping cpumasks
The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full" boot command line options, are used at boot time to exclude selected CPUs from running some kernel background processes to minimize disturbance to latency sensitive userspace applications. Some of housekeeping CPU masks are also checked at run time to avoid using those isolated CPUs.
The cpuset subsystem is now able to dynamically create a set of isolated CPUs to be used in isolated cpuset partitions. The long term goal is to make the degree of isolation as close as possible to what can be done statically using those boot command line options.
This patch is a step in that direction by providing a new housekeep_exclude_cpumask() API to exclude only the given cpumask from the housekeeping cpumasks. Existing boot time "isolcpus" and "nohz_full" cpumask setup, if present, can be overwritten.
Two set of cpumasks are now kept internally. One set are used by the callers while the other set are being updated before the new set are atomically switched on.
--- include/linux/sched/isolation.h | 6 +++ kernel/sched/isolation.c | 95 +++++++++++++++++++++++++++++---- 2 files changed, 91 insertions(+), 10 deletions(-)
include/linux/sched/isolation.h +6 -0
@@ -32,6 +32,7 @@ extern bool housekeeping_enabled(enum hk_type type);extern void housekeeping_affine (struct task_struct *t,enum hk_type type);extern bool housekeeping_test_cpu (int cpu,enum hk_type type);extern void __init housekeeping_init (void ); +extern int housekeeping_exclude_cpumask (struct cpumask *cpumask,unsigned long flags);#else
@@ -59,6 +60,11 @@ static inline bool housekeeping_test_cpu(int cpu, enum hk_type type) }static inline void housekeeping_init (void ) { } + +static inline housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long flags) +{ + return -EOPNOTSUPP; +}#endif /* CONFIG_CPU_ISOLATION */ static inline bool housekeeping_cpu (int cpu,enum hk_type type)
kernel/sched/isolation.c +85 -10
150 lines not on this page · raw message
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 2/18 sched/isolation: Call sched_tick_offload_init() when HK_FLAG_KERNEL_NOISE is first set
In reply to Waiman LongThe sched_tick_offload_init() function is called at boot time whenever "nohz_full" is set. Now housekeeping cpumasks can be updated at run time without the corresponding "nohz_full" kernel parameter. So we have to be able to call sched_tick_offload_init() at run time to allow tick offloading. Remove the __init attribute from sched_tick_offload_init() and call it when the HK_FLAG_KERNEL_NOISE flag is first set.
--- kernel/sched/core.c | 2 +- kernel/sched/isolation.c | 10 +++++++++- kernel/sched/sched.h | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
@@ -5783,7 +5783,7 @@ static void sched_tick_stop(int cpu) }#endif /* CONFIG_HOTPLUG_CPU */ -int __init sched_tick_offload_init (void ) +int sched_tick_offload_init (void ) { tick_work_cpu = alloc_percpu(struct tick_work);BUG_ON (!tick_work_cpu);
kernel/sched/isolation.c +9 -1
@@ -33,6 +33,7 @@ struct housekeeping { };static struct housekeeping housekeeping; +static bool sched_tick_offload_inited;bool housekeeping_enabled (enum hk_type type) {
@@ -103,8 +104,10 @@ void __init housekeeping_init(void)static_branch_enable (&housekeeping_overridden); -if (housekeeping.flags &HK_FLAG_KERNEL_NOISE ) +if (housekeeping.flags &HK_FLAG_KERNEL_NOISE ) {sched_tick_offload_init (); + sched_tick_offload_inited = true; + }for_each_set_bit (type, &housekeeping.flags,HK_TYPE_MAX ) {/* We need at least one CPU to handle housekeeping work */
@@ -324,5 +327,10 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flagsstatic_key_disable (&housekeeping_overridden.key); else if (housekeeping.flags && !static_key_enabled(&housekeeping_overridden)) static_key_enable(&housekeeping_overridden.key); + +if ((housekeeping.flags &HK_FLAG_KERNEL_NOISE ) && !sched_tick_offload_inited) { +sched_tick_offload_init (); + sched_tick_offload_inited = true; + }return 0 ; }
kernel/sched/sched.h +1 -1
@@ -2671,7 +2671,7 @@ extern void post_init_entity_util_avg(struct task_struct *p);#ifdef CONFIG_NO_HZ_FULL extern bool sched_can_stop_tick (struct rq *rq); -extern int __init sched_tick_offload_init (void ); +extern int sched_tick_offload_init (void );/* * Tick may be needed by tasks in the runqueue depending on their policy and
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 3/18 sched/isolation: Use RCU to delay successive housekeeping cpumask updates
In reply to Waiman LongEven though there are 2 separate sets of housekeeping cpumasks for access and update, it is possible that the set of cpumasks to be updated are still being used by the callers of housekeeping functions resulting in the use of an intermediate cpumask between the new and old ones.
To reduce the chance of this, we need to introduce delay between successive housekeeping cpumask updates. One simple way is to make use of the RCU grace period delay. The callers of the housekeeping APIs can optionally hold rcu_read_lock to eliminate the chance of using intermediate housekeeping cpumasks.
--- kernel/sched/isolation.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)
kernel/sched/isolation.c +33 -0
@@ -23,6 +23,9 @@ EXPORT_SYMBOL_GPL(housekeeping_overridden);* The housekeeping cpumasks can now be dynamically updated at run time. * Two set of cpumasks are kept. One set can be used while the other set are * being updated concurrently. +* +* rcu_read_lock() can optionally be held by housekeeping API callers to +* ensure stability of the cpumasks. */ static DEFINE_RAW_SPINLOCK (cpumask_lock);struct housekeeping {
@@ -34,6 +37,8 @@ struct housekeeping {static struct housekeeping housekeeping;static bool sched_tick_offload_inited; +static struct rcu_head rcu_gp[HK_TYPE_MAX ]; +static unsigned long update_flags;bool housekeeping_enabled (enum hk_type type) {
@@ -267,6 +272,18 @@ static int __init housekeeping_isolcpus_setup(char *str) }__setup ("isolcpus=" , housekeeping_isolcpus_setup); +/* +* Bits in update_flags can only turned on with cpumask_lock held and +* cleared by this RCU callback function. +*/ +static void rcu_gp_end (struct rcu_head *rcu) +{ +int type = rcu - rcu_gp; + +/* Atomically clear the corresponding flag bit */ +clear_bit (type, &update_flags); +} +/** * housekeeping_exclude_cpumask - Update housekeeping cpumasks to exclude only the given cpumask * @cpumask: new cpumask to be excluded from housekeeping cpumasks
@@ -306,8 +323,21 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flags }#endif +retry: +/* +* If the RCU grace period for the previous update with conflicting +* flag bits hasn't been completed yet, we have to wait for it. +*/ +while (READ_ONCE (update_flags) & hk_flags) +synchronize_rcu (); +raw_spin_lock (&cpumask_lock); +if (READ_ONCE (update_flags) & hk_flags) { +raw_spin_unlock (&cpumask_lock); + goto retry; + } +for_each_set_bit (type, &hk_flags,HK_TYPE_MAX ) {int idx = ++housekeeping.seq_nrs[type] &1 ;struct cpumask *dst_cpumask = housekeeping.cpumasks[type][idx];
@@ -320,8 +350,11 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flags housekeeping.flags |=BIT (type); }WRITE_ONCE (housekeeping.cpumask_ptrs[type], dst_cpumask); +set_bit (type, &update_flags); }raw_spin_unlock (&cpumask_lock); +for_each_set_bit (type, &hk_flags,HK_TYPE_MAX ) +call_rcu (&rcu_gp[type], rcu_gp_end);if (!housekeeping.flags &&static_key_enabled (&housekeeping_overridden))static_key_disable (&housekeeping_overridden.key);
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 4/18 sched/isolation: Add a debugfs file to dump housekeeping cpumasks
In reply to Waiman LongAs housekeeping cpumasks can now be modified at run time, we need a way to examine the their current values to see if they meet our expectation. Add a new sched debugfs file "housekeeping_cpumasks" to dump out the current values.
--- kernel/sched/debug.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
kernel/sched/debug.c +32 -0
@@ -490,6 +490,35 @@ static void debugfs_fair_server_init(void) } } +#ifdef CONFIG_CPU_ISOLATION +static int hk_cpumasks_show (struct seq_file *m,void *v) +{ +static const char *const hk_type_name[HK_TYPE_MAX ] = { + [HK_TYPE_DOMAIN ] ="domain" , + [HK_TYPE_MANAGED_IRQ ] ="managed_irq" , + [HK_TYPE_KERNEL_NOISE ] ="nohz_full" + }; +int type; + +for (type =0 ; type <HK_TYPE_MAX ; type++) +seq_printf (m,"%s: %*pbl\n" , hk_type_name[type], +cpumask_pr_args (housekeeping_cpumask (type))); +return 0 ; +} + +static int hk_cpumasks_open (struct inode *inode,struct file *filp) +{ +return single_open (filp, hk_cpumasks_show,NULL ); +} + +static const struct file_operations hk_cpumasks_fops = { + .open = hk_cpumasks_open, + .read = seq_read, + .llseek = seq_lseek, + .release = seq_release, +}; +#endif +static __init int sched_init_debug (void ) {struct dentry __maybe_unused *numa;
@@ -525,6 +554,9 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_CPU_ISOLATION +debugfs_create_file ("housekeeing_cpumasks" ,0444 , debugfs_sched,NULL , &hk_cpumasks_fops); +#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 5/18 cpu/hotplug: Add a new cpuhp_offline_cb() API
In reply to Waiman LongAdd a new cpuhp_offline_cb() API that allows us to offline a set of CPUs one-by-one, run the given callback function and then bring those CPUs back online again while inhibiting any concurrent CPU hotplug operations from happening.
This new API can be used to enable runtime adjustment of nohz_full and isolcpus boot command line options. A new cpuhp_offline_cb_mode flag is also added to signal that the system is in this offline callback transient state so that some hotplug operations can be optimized out if we choose to.
--- include/linux/cpuhplock.h | 9 ++++++++ kernel/cpu.c | 47 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/linux/cpuhplock.h +9 -0
@@ -9,7 +9,9 @@#include <linux/cleanup.h> #include <linux/errno.h> +#include <linux/cpumask_types.h> +typedef int (*cpuhp_cb_t )(void *arg);struct device ;extern int lockdep_is_cpus_held (void );
@@ -28,6 +30,8 @@ void clear_tasks_mm_cpumask(int cpu);int remove_cpu (unsigned int cpu);int cpu_device_down (struct device *dev);void smp_shutdown_nonboot_cpus (unsigned int primary_cpu); +int cpuhp_offline_cb (struct cpumask *mask,cpuhp_cb_t func,void *arg); +extern bool cpuhp_offline_cb_mode;#else /* CONFIG_HOTPLUG_CPU */
@@ -42,6 +46,11 @@ static inline void cpu_hotplug_disable(void) { }static inline void cpu_hotplug_enable (void ) { }static inline int remove_cpu (unsigned int cpu) {return -EPERM ; }static inline void smp_shutdown_nonboot_cpus (unsigned int primary_cpu) { } +static inline int cpuhp_offline_cb (struct cpumask *mask,cpuhp_cb_t func,void *arg) +{ +return -EPERM ; +} +#define cpuhp_offline_cb_mode false#endif /* !CONFIG_HOTPLUG_CPU */ DEFINE_LOCK_GUARD_0 (cpus_read_lock,cpus_read_lock (),cpus_read_unlock ())
kernel/cpu.c +47 -0
53 lines not on this page · raw message
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 6/18 cgroup/cpuset: Introduce a new top level isolcpus_update_mutex
In reply to Waiman LongThe current cpuset partition code is able to dynamically update the sched domains of a running system to perform what is essentally the "isolcpus=domain,..." boot command line feature at run time.
To enable runtime modification of nohz_full, we will have to make use of the CPU hotplug functionality to facilitate the proper addition or subtraction of nohz_full CPUs. In other word, we can't hold the cpu_hotplug_lock while doing so. Given the current lock ordering, we will need to introduce a new top level mutex to ensure proper mutual exclusion in case there is a need to update the cpuset states that may require the use of CPU hotplug. This patch introduces a new top level isolcpus_update_mutex for such purpose. This new mutex will be acquired in case the cpuset partition states or the set of isolated CPUs may have to be changed.
The update_unbound_workqueue_cpumask() is now renamed to update_isolation_cpumasks() and moved outside of cpu_hotplug_lock critical regions to enable its future extension to invoke CPU hotplug.
A new global isolcpus_update_state structure is added to track if update_isolation_cpumasks() will need to be invoked. So the existing partition_xcpus_add/del() functions and their callers can now be simplified.
--- kernel/cgroup/cpuset.c | 149 ++++++++++++++++++++++++----------------- 1 file changed, 86 insertions(+), 63 deletions(-)
kernel/cgroup/cpuset.c +86 -63
@@ -215,29 +215,39 @@ static struct cpuset top_cpuset = { };/* -* There are two global locks guarding cpuset structures - cpuset_mutex and -* callback_lock. The cpuset code uses only cpuset_mutex. Other kernel -* subsystems can use cpuset_lock()/cpuset_unlock() to prevent change to cpuset -* structures. Note that cpuset_mutex needs to be a mutex as it is used in -* paths that rely on priority inheritance (e.g. scheduler - on RT) for -* correctness. +* CPUSET Locking Convention +* ------------------------- * -* A task must hold both locks to modify cpusets. If a task holds -* cpuset_mutex, it blocks others, ensuring that it is the only task able to -* also acquire callback_lock and be able to modify cpusets. It can perform -* various checks on the cpuset structure first, knowing nothing will change. -* It can also allocate memory while just holding cpuset_mutex. While it is -* performing these checks, various callback routines can briefly acquire -* callback_lock to query cpusets. Once it is ready to make the changes, it -* takes callback_lock, blocking everyone else. +* Below are the three global locks guarding cpuset structures in lock +* acquisition order: +* - isolcpus_update_mutex +* - cpu_hotplug_lock (cpus_read_lock/cpus_write_lock) +* - cpuset_mutex +* - callback_lock (raw spinlock) * -* Calls to the kernel memory allocator can not be made while holding -* callback_lock, as that would risk double tripping on callback_lock -* from one of the callbacks into the cpuset code from within -* __alloc_pages(). +* The first isolcpus_update_mutex should only be held if the existing set of +* isolated CPUs (in isolated partition) or any of the partition states may be +* changed. Otherwise, it can be skipped. This is used to prevent concurrent +* updates to the set of isolated CPUs. * -* If a task is only holding callback_lock, then it has read-only -* access to cpusets. +* A task must hold all the remaining three locks to modify externally visible +* or used fields of cpusets, though some of the internally used cpuset fields +* can be modified by holding cpu_hotplug_lock and cpuset_mutex only. If only +* reliable read access of the externally used fields are needed, a task can +* hold either cpuset_mutex or callback_lock. +* +* If a task holds cpu_hotplug_lock and cpuset_mutex, it blocks others, +* ensuring that it is the only task able to also acquire callback_lock and +* be able to modify cpusets. It can perform various checks on the cpuset +* structure first, knowing nothing will change. It can also allocate memory +* without holding callback_lock. While it is performing these checks, various +* callback routines can briefly acquire callback_lock to query cpusets. Once +* it is ready to make the changes, it takes callback_lock, blocking everyone +* else. +* +* Calls to the kernel memory allocator cannot be made while holding +* callback_lock which is a spinlock, as the memory allocator may sleep or +* call back into cpuset code and acquire callback_lock. * * Now, the task_struct fields mems_allowed and mempolicy may be changed * by other task, we use alloc_lock in the task_struct fields to protect
@@ -248,6 +258,7 @@ static struct cpuset top_cpuset = {* cpumasks and nodemasks. */ +static DEFINE_MUTEX (isolcpus_update_mutex);static DEFINE_MUTEX (cpuset_mutex);void cpuset_lock (void )
@@ -272,6 +283,17 @@ void cpuset_callback_unlock_irq(void)spin_unlock_irq (&callback_lock); } +/* +* Isolcpus update state (protected by isolcpus_update_mutex mutex) +* +* It contains data related to updating the isolated CPUs configuration in +* isolated partitions. +*/ +static struct { +bool updating;/* Isolcpus updating in progress */ +cpumask_var_t cpus;/* CPUs to be updated */ +} isolcpus_update_state; +static struct workqueue_struct *cpuset_migrate_mm_wq;static DECLARE_WAIT_QUEUE_HEAD (cpuset_attach_wq);
@@ -1273,6 +1295,9 @@ static void isolated_cpus_update(int old_prs, int new_prs, struct cpumask *xcpuscpumask_or (isolated_cpus, isolated_cpus, xcpus); elsecpumask_andnot (isolated_cpus ,isolated_cpus ,xcpus ); + + isolcpus_update_state.updating = true; +cpumask_or (isolcpus_update_state.cpus, isolcpus_update_state.cpus, xcpus); }/*
@@ -1280,31 +1305,26 @@ static void isolated_cpus_update(int old_prs, int new_prs, struct cpumask *xcpus* @new_prs: new partition_root_state * @parent: parent cpuset * @xcpus: exclusive CPUs to be added -* Return: true if isolated_cpus modified, false otherwise * * Remote partition if parent == NULL */ -static bool partition_xcpus_add (int new_prs,struct cpuset *parent, +static void partition_xcpus_add (int new_prs,struct cpuset *parent,struct cpumask *xcpus) { -bool isolcpus_updated; -WARN_ON_ONCE (new_prs <0 );lockdep_assert_held (&callback_lock);if (!parent) parent = &top_cpuset; -if (parent == &top_cpuset)cpumask_or (subpartitions_cpus, subpartitions_cpus, xcpus); - isolcpus_updated = (new_prs != parent->partition_root_state); -if (isolcpus_updated) +if (new_prs != parent->partition_root_state)isolated_cpus_update (parent->partition_root_state, new_prs, xcpus);cpumask_andnot (parent->effective_cpus, parent->effective_cpus, xcpus); -return isolcpus_updated; +return ; }/*
@@ -1312,15 +1332,12 @@ static bool partition_xcpus_add(int new_prs, struct cpuset *parent,* @old_prs: old partition_root_state * @parent: parent cpuset * @xcpus: exclusive CPUs to be removed -* Return: true if isolated_cpus modified, false otherwise * * Remote partition if parent == NULL */ -static bool partition_xcpus_del (int old_prs,struct cpuset *parent, +static void partition_xcpus_del (int old_prs,struct cpuset *parent,struct cpumask *xcpus) { -bool isolcpus_updated; -WARN_ON_ONCE (old_prs <0 );lockdep_assert_held (&callback_lock);if (!parent)
@@ -1329,27 +1346,33 @@ static bool partition_xcpus_del(int old_prs, struct cpuset *parent,if (parent == &top_cpuset)cpumask_andnot (subpartitions_cpus, subpartitions_cpus, xcpus); - isolcpus_updated = (old_prs != parent->partition_root_state); -if (isolcpus_updated) +if (old_prs != parent->partition_root_state)isolated_cpus_update (old_prs, parent->partition_root_state, xcpus);cpumask_and (xcpus, xcpus, cpu_active_mask);cpumask_or (parent->effective_cpus, parent->effective_cpus, xcpus); -return isolcpus_updated; +return ; } -static void update_unbound_workqueue_cpumask (bool isolcpus_updated) +/** +* update_isolation_cpumasks - Update external isolation CPU masks +* +* The following external CPU masks will be updated if necessary: +* - workqueue unbound cpumask +*/ +static void update_isolation_cpumasks (void ) {int ret; -lockdep_assert_cpus_held (); - -if (!isolcpus_updated) +if (!isolcpus_update_state.updating)return ; ret =workqueue_unbound_exclude_cpumask (isolated_cpus);WARN_ON_ONCE (ret <0 ); + +cpumask_clear (isolcpus_update_state.cpus); + isolcpus_update_state.updating = false; }/**
@@ -1441,8 +1464,6 @@ static inline bool is_local_partition(struct cpuset *cs)static int remote_partition_enable (struct cpuset *cs,int new_prs,struct tmpmasks *tmp) { -bool isolcpus_updated; -/* * The user must have sysadmin privilege. */
@@ -1466,11 +1487,10 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,return PERR_INVCPUS ;spin_lock_irq (&callback_lock); - isolcpus_updated =partition_xcpus_add (new_prs,NULL , tmp->new_cpus); +partition_xcpus_add (new_prs,NULL , tmp->new_cpus);list_add (&cs->remote_sibling, &remote_children);cpumask_copy (cs->effective_xcpus, tmp->new_cpus);spin_unlock_irq (&callback_lock); -update_unbound_workqueue_cpumask (isolcpus_updated);cpuset_force_rebuild (); cs->prs_err =0 ;
@@ -1493,15 +1513,12 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,*/ static void remote_partition_disable (struct cpuset *cs,struct tmpmasks *tmp) { -bool isolcpus_updated; -WARN_ON_ONCE (!is_remote_partition (cs));WARN_ON_ONCE (!cpumask_subset (cs->effective_xcpus, subpartitions_cpus));spin_lock_irq (&callback_lock);list_del_init (&cs->remote_sibling); - isolcpus_updated =partition_xcpus_del (cs->partition_root_state, -NULL , cs->effective_xcpus); +partition_xcpus_del (cs->partition_root_state,NULL , cs->effective_xcpus);if (cs->prs_err) cs->partition_root_state = -cs->partition_root_state;else
@@ -1511,7 +1528,6 @@ static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp)compute_effective_exclusive_cpumask (cs,NULL ,NULL );reset_partition_data (cs);spin_unlock_irq (&callback_lock); -update_unbound_workqueue_cpumask (isolcpus_updated);cpuset_force_rebuild ();/*
@@ -1536,7 +1552,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, {bool adding, deleting;int prs = cs->partition_root_state; -int isolcpus_updated =0 ;if (WARN_ON_ONCE (!is_remote_partition (cs)))return ;
@@ -1569,9 +1584,9 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,spin_lock_irq (&callback_lock);if (adding) - isolcpus_updated +=partition_xcpus_add (prs,NULL , tmp->addmask); +partition_xcpus_add (prs,NULL , tmp->addmask);if (deleting) - isolcpus_updated +=partition_xcpus_del (prs,NULL , tmp->delmask); +partition_xcpus_del (prs,NULL , tmp->delmask);/* * Need to update effective_xcpus and exclusive_cpus now as * update_sibling_cpumasks() below may iterate back to the same cs.
@@ -1580,7 +1595,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,if (xcpus)cpumask_copy (cs->exclusive_cpus, xcpus);spin_unlock_irq (&callback_lock); -update_unbound_workqueue_cpumask (isolcpus_updated);if (adding || deleting)cpuset_force_rebuild ();
@@ -1662,7 +1676,6 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,int old_prs, new_prs;int part_error =PERR_NONE ;/* Partition error? */ int subparts_delta =0 ; -int isolcpus_updated =0 ;struct cpumask *xcpus =user_xcpus (cs);bool nocpu;
@@ -1932,18 +1945,15 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,* and vice versa. */ if (adding) - isolcpus_updated +=partition_xcpus_del (old_prs, parent, - tmp->addmask); +partition_xcpus_del (old_prs, parent, tmp->addmask);if (deleting) - isolcpus_updated +=partition_xcpus_add (new_prs, parent, - tmp->delmask); +partition_xcpus_add (new_prs, parent, tmp->delmask);if (is_partition_valid (parent)) { parent->nr_subparts += subparts_delta;WARN_ON_ONCE (parent->nr_subparts <0 ); }spin_unlock_irq (&callback_lock); -update_unbound_workqueue_cpumask (isolcpus_updated);if ((old_prs != new_prs) && (cmd == partcmd_update))update_partition_exclusive_flag (cs, new_prs);
@@ -2968,7 +2978,6 @@ static int update_prstate(struct cpuset *cs, int new_prs)else if (isolcpus_updated)isolated_cpus_update (old_prs, new_prs, cs->effective_xcpus);spin_unlock_irq (&callback_lock); -update_unbound_workqueue_cpumask (isolcpus_updated);/* Force update if switching back to member & update effective_xcpus */ update_cpumasks_hier (cs, &tmpmask, !new_prs);
@@ -3224,6 +3233,7 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,int retval = -ENODEV ; buf =strstrip (buf); +mutex_lock (&isolcpus_update_mutex);cpus_read_lock ();mutex_lock (&cpuset_mutex);if (!is_cpuset_online (cs))
@@ -3256,6 +3266,8 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of, out_unlock:mutex_unlock (&cpuset_mutex);cpus_read_unlock (); +update_isolation_cpumasks (); +mutex_unlock (&isolcpus_update_mutex);flush_workqueue (cpuset_migrate_mm_wq);return retval ?: nbytes; }
@@ -3358,12 +3370,15 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,else return -EINVAL ; +mutex_lock (&isolcpus_update_mutex);cpus_read_lock ();mutex_lock (&cpuset_mutex);if (is_cpuset_online (cs)) retval =update_prstate (cs, val);mutex_unlock (&cpuset_mutex);cpus_read_unlock (); +update_isolation_cpumasks (); +mutex_unlock (&isolcpus_update_mutex);return retval ?: nbytes; }
@@ -3586,15 +3601,22 @@ static void cpuset_css_killed(struct cgroup_subsys_state *css) {struct cpuset *cs =css_cs (css); +mutex_lock (&isolcpus_update_mutex); +/* +* Here the partition root state can't be changed by user again. +*/ +if (!is_partition_valid (cs)) + goto out; +cpus_read_lock ();mutex_lock (&cpuset_mutex); -/* Reset valid partition back to member */ -if (is_partition_valid (cs)) -update_prstate (cs,PRS_MEMBER ); - +update_prstate (cs,PRS_MEMBER );mutex_unlock (&cpuset_mutex);cpus_read_unlock (); +update_isolation_cpumasks (); +out: +mutex_unlock (&isolcpus_update_mutex); }
@@ -3751,6 +3773,7 @@ int __init cpuset_init(void)BUG_ON (!alloc_cpumask_var (&top_cpuset.exclusive_cpus,GFP_KERNEL ));BUG_ON (!zalloc_cpumask_var (&subpartitions_cpus,GFP_KERNEL ));BUG_ON (!zalloc_cpumask_var (&isolated_cpus,GFP_KERNEL )); +BUG_ON (!zalloc_cpumask_var (&isolcpus_update_state.cpus,GFP_KERNEL ));cpumask_setall (top_cpuset.cpus_allowed);nodes_setall (top_cpuset.mems_allowed);
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 7/18 cgroup/cpuset: Allow overwriting HK_TYPE_DOMAIN housekeeping cpumask
In reply to Waiman LongAs we did not modify housekeeping cpumasks in the creation of cpuset partition before, we had to disallow the creation of non-isolated partitions from using any of the HK_TYPE_DOMAIN isolated CPUs. Now we are going to modify housekeeping cpumasks at run time, we will now allow overwriting of HK_TYPE_DOMAIN cpumask when an isolated partition is first created or when the creation of a non-isolated partition conflicts with the boot time HK_TYPE_DOMAIN isolated CPUs. The unnecessary checking code are now being removed. The doc file will be updated in a later patch.
On the other hand, there is still a latency spike problem when CPU hotplug code is used to facilitate the proper functioning of the dynamically modified nohz_full HK_TYPE_KERNEL_NOISE cpumask. So the cpuset code will be modified to maintain the boot-time enabled nohz_full cpumask to avoid using cpu hotplug if all the newly isolated/non-isolated CPUs are already in that cpumask. This code will be removed in the future when the latency spike problem is solved.
--- kernel/cgroup/cpuset.c | 45 ++++++++---------------------------------- 1 file changed, 8 insertions(+), 37 deletions(-)
kernel/cgroup/cpuset.c +8 -37
@@ -59,7 +59,6 @@ static const char * const perr_strings[] = { [PERR_NOCPUS] = "Parent unable to distribute cpu downstream", [PERR_HOTPLUG] = "No cpu available due to hotplug", [PERR_CPUSEMPTY] = "cpuset.cpus and cpuset.cpus.exclusive are empty", - [PERR_HKEEPING] = "partition config conflicts with housekeeping setup", [PERR_ACCESS] = "Enable partition not permitted", [PERR_REMOTE] = "Have remote partition underneath", };
@@ -81,9 +80,10 @@ static cpumask_var_t subpartitions_cpus;static cpumask_var_t isolated_cpus;/* -* Housekeeping (HK_TYPE_DOMAIN) CPUs at boot +* Housekeeping (nohz_full) CPUs at boot */ -static cpumask_var_t boot_hk_cpus; +static cpumask_var_t boot_nohz_full_hk_cpus; +static bool have_boot_nohz_full;static bool have_boot_isolcpus;/* List of remote partition root children */
@@ -1609,26 +1609,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,remote_partition_disable (cs, tmp); } -/* -* prstate_housekeeping_conflict - check for partition & housekeeping conflicts -* @prstate: partition root state to be checked -* @new_cpus: cpu mask -* Return: true if there is conflict, false otherwise -* -* CPUs outside of boot_hk_cpus, if defined, can only be used in an -* isolated partition. -*/ -static bool prstate_housekeeping_conflict (int prstate,struct cpumask *new_cpus) -{ -if (!have_boot_isolcpus) -return false; - -if ((prstate !=PRS_ISOLATED ) && !cpumask_subset (new_cpus, boot_hk_cpus)) -return true; - -return false; -} -/** * update_parent_effective_cpumask - update effective_cpus mask of parent cpuset * @cs: The cpuset that requests change in partition root state
@@ -1737,9 +1717,6 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,if (cpumask_empty (xcpus))return PERR_INVCPUS ; -if (prstate_housekeeping_conflict (new_prs, xcpus)) -return PERR_HKEEPING ; -/* * A parent can be left with no CPU as long as there is no * task directly associated with the parent partition.
@@ -2356,9 +2333,6 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs, cpumask_empty(trialcs->effective_xcpus)) { invalidate = true; cs->prs_err =PERR_INVCPUS ; - } else if (prstate_housekeeping_conflict(old_prs, trialcs->effective_xcpus)) { - invalidate = true; - cs->prs_err =PERR_HKEEPING ; } else if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) { invalidate = true; cs->prs_err =PERR_NOCPUS ;
@@ -2499,9 +2473,6 @@ static int update_exclusive_cpumask(struct cpuset *cs, struct cpuset *trialcs,if (cpumask_empty (trialcs->effective_xcpus)) { invalidate = true; cs->prs_err =PERR_INVCPUS ; - }else if (prstate_housekeeping_conflict (old_prs, trialcs->effective_xcpus)) { - invalidate = true; - cs->prs_err =PERR_HKEEPING ; }else if (tasks_nocpu_error (parent, cs, trialcs->effective_xcpus)) { invalidate = true; cs->prs_err =PERR_NOCPUS ;
@@ -3787,11 +3758,11 @@ int __init cpuset_init(void)BUG_ON (!alloc_cpumask_var (&cpus_attach,GFP_KERNEL )); - have_boot_isolcpus =housekeeping_enabled (HK_TYPE_DOMAIN ); -if (have_boot_isolcpus) { -BUG_ON (!alloc_cpumask_var (&boot_hk_cpus,GFP_KERNEL )); -cpumask_copy (boot_hk_cpus,housekeeping_cpumask (HK_TYPE_DOMAIN )); -cpumask_andnot (isolated_cpus, cpu_possible_mask, boot_hk_cpus); + have_boot_nohz_full =housekeeping_enabled (HK_TYPE_KERNEL_NOISE ); + have_boot_isolcpus =housekeeping_enabled (HK_TYPE_DOMAIN ); +if (have_boot_nohz_full) { +BUG_ON (!alloc_cpumask_var (&boot_nohz_full_hk_cpus,GFP_KERNEL )); +cpumask_copy (boot_nohz_full_hk_cpus,housekeeping_cpumask (HK_TYPE_KERNEL_NOISE )); }return 0 ;
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 8/18 cgroup/cpuset: Use CPU hotplug to enable runtime nohz_full modification
In reply to Waiman LongOne relatively simple way to allow runtime modification of nohz_full, and rcu_nocbs CPUs is to use the CPU hotplug to bring the affected CPUs offline first, making changes to the housekeeping cpumasks and then bring them back online. However, doing this will be rather costly in term of the number of CPU cycles needed. Still it is the easiet way to achieve the desired result and hopefully we can gradually reduce the overhead over time.
Use the newly introduced cpuhp_offline_cb() API to bring the affected CPUs offline, make the necessary housekeeping cpumask changes and then bring those CPUs back online again.
As HK_TYPE_DOMAIN cpumask is going to be updated at run time, we are going to reset any boot time isolcpus domain setting if an isolated partition or a conflicting non-isolated partition is going to be created.
Since rebuild_sched_domains() will be called at the end of update_isolation_cpumasks(), earlier rebuild_sched_domains_locked() calls will be suppressed to avoid unneeded work.
--- kernel/cgroup/cpuset.c | 95 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 92 insertions(+), 3 deletions(-)
kernel/cgroup/cpuset.c +92 -3
@@ -1355,11 +1355,57 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,return ; } +/* +* We are only updating HK_TYPE_DOMAIN and HK_TYPE_KERNEL_NOISE housekeeping +* cpumask for now. HK_TYPE_MANAGED_IRQ will be handled later. +*/ +static int do_housekeeping_exclude_cpumask (void *arg__maybe_unused ) +{ +int ret; +struct cpumask *icpus = isolated_cpus; +unsigned long flags =BIT (HK_TYPE_DOMAIN ) |BIT (HK_TYPE_KERNEL_NOISE ); + +/* +* The boot time isolcpus setting will be overwritten if set. +*/ + have_boot_isolcpus = false; + +if (have_boot_nohz_full) { +/* +* Need to separate the handling of HK_TYPE_KERNEL_NOISE and +* HK_TYPE_DOMAIN as different cpumasks will be used for each. +*/ + ret =housekeeping_exclude_cpumask (icpus,BIT (HK_TYPE_DOMAIN )); +WARN_ON_ONCE ((ret <0 ) && (ret != -EOPNOTSUPP )); + +if (cpumask_empty (isolcpus_update_state.cpus)) +return ret; + flags =BIT (HK_TYPE_KERNEL_NOISE ); + icpus =kmalloc (cpumask_size (),GFP_KERNEL ); +if (WARN_ON_ONCE (!icpus)) +return -ENOMEM ; + +/* +* Add boot time nohz_full CPUs into the isolated CPUs list +* for exclusion from HK_TYPE_KERNEL_NOISE CPUs. +*/ +cpumask_andnot (icpus, cpu_possible_mask, boot_nohz_full_hk_cpus); +cpumask_or (icpus, icpus, isolated_cpus); + } + ret =housekeeping_exclude_cpumask (icpus, flags); +WARN_ON_ONCE ((ret <0 ) && (ret != -EOPNOTSUPP )); + +if (icpus != isolated_cpus) +kfree (icpus); +return ret; +} +/** * update_isolation_cpumasks - Update external isolation CPU masks * * The following external CPU masks will be updated if necessary: * - workqueue unbound cpumask +* - housekeeping cpumasks */ static void update_isolation_cpumasks (void ) {
@@ -1371,7 +1417,41 @@ static void update_isolation_cpumasks(void) ret =workqueue_unbound_exclude_cpumask (isolated_cpus);WARN_ON_ONCE (ret <0 ); +/* +* Mask out offline and boot-time nohz_full non-housekeeping +* CPUs from isolcpus_update_state.cpus to compute the set +* of CPUs that need to be brought offline before calling +* do_housekeeping_exclude_cpumask(). +*/ +cpumask_and (isolcpus_update_state.cpus, + isolcpus_update_state.cpus, cpu_active_mask); +if (have_boot_nohz_full) +cpumask_and (isolcpus_update_state.cpus, + isolcpus_update_state.cpus, boot_nohz_full_hk_cpus); + +/* +* Without any change in the set of nohz_full CPUs, we don't really +* need to use CPU hotplug for making change in HK cpumasks. +*/ +if (cpumask_empty (isolcpus_update_state.cpus)) + ret =do_housekeeping_exclude_cpumask (NULL ); +else + ret =cpuhp_offline_cb (isolcpus_update_state.cpus, + do_housekeeping_exclude_cpumask,NULL ); +/* +* A errno value of -EPERM may be returned from cpuhp_offline_cb() if +* any one of the CPUs in isolcpus_update_state.cpus can't be brought +* offline. This can happen for the boot CPU (normally CPU 0) which +* cannot be shut down. This CPU should not be used for creating +* isolated partition. +*/ +if (ret == -EPERM ) +pr_warn_once ("cpuset: The boot CPU shouldn't be used for isolated partition\n" ); +else +WARN_ON_ONCE (ret <0 ); +cpumask_clear (isolcpus_update_state.cpus); +rebuild_sched_domains (); isolcpus_update_state.updating = false; }
@@ -2961,7 +3041,16 @@ static int update_prstate(struct cpuset *cs, int new_prs)update_partition_sd_lb (cs, old_prs);notify_partition_change (cs, old_prs); -if (force_sd_rebuild) + +/* +* If boot time domain isolcpus exists and it conflicts with the CPUs +* in the new partition, we will have to reset HK_TYPE_DOMAIN cpumask. +*/ +if (have_boot_isolcpus && (new_prs >PRS_MEMBER ) && + !cpumask_subset (cs->effective_xcpus,housekeeping_cpumask (HK_TYPE_DOMAIN ))) + isolcpus_update_state.updating = true; + +if (force_sd_rebuild && !isolcpus_update_state.updating)rebuild_sched_domains_locked ();free_cpumasks (NULL , &tmpmask);return 0 ;
@@ -3232,7 +3321,7 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of, }free_cpuset (trialcs); -if (force_sd_rebuild) +if (force_sd_rebuild && !isolcpus_update_state.updating)rebuild_sched_domains_locked (); out_unlock:mutex_unlock (&cpuset_mutex);
@@ -3999,7 +4088,7 @@ static void cpuset_handle_hotplug(void) }/* rebuild sched domains if necessary */ -if (force_sd_rebuild) +if (force_sd_rebuild && !isolcpus_update_state.updating)rebuild_sched_domains_cpuslocked ();free_cpumasks (NULL , ptmp);
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 9/18 cgroup/cpuset: Revert "Include isolated cpuset CPUs in cpu_is_isolated() check"
In reply to Waiman LongNow that the HK_TYPE_DOMAIN cpumask is updated at run time to reflect changes made in isolated cpuset partitions. We no longer need a separate cpuset_cpu_is_isolated() function for checking isolated CPUs generated by cpuset. Revert commit 3232e7aad11e ("cgroup/cpuset: Include isolated cpuset CPUs in cpu_is_isolated() check").
--- include/linux/cpuset.h | 6 ------ include/linux/sched/isolation.h | 3 +-- kernel/cgroup/cpuset.c | 11 ----------- 3 files changed, 1 insertion(+), 19 deletions(-)
include/linux/cpuset.h +0 -6
@@ -76,7 +76,6 @@ extern void cpuset_lock(void);extern void cpuset_unlock (void );extern void cpuset_cpus_allowed (struct task_struct *p,struct cpumask *mask);extern bool cpuset_cpus_allowed_fallback (struct task_struct *p); -extern bool cpuset_cpu_is_isolated (int cpu);extern nodemask_t cpuset_mems_allowed (struct task_struct *p);#define cpuset_current_mems_allowed (current->mems_allowed)void cpuset_init_current_mems_allowed (void );
@@ -206,11 +205,6 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)return false; } -static inline bool cpuset_cpu_is_isolated (int cpu) -{ -return false; -} -static inline nodemask_t cpuset_mems_allowed (struct task_struct *p) {return node_possible_map;
include/linux/sched/isolation.h +1 -2
@@ -79,8 +79,7 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)static inline bool cpu_is_isolated (int cpu) {return !housekeeping_test_cpu (cpu,HK_TYPE_DOMAIN ) || - !housekeeping_test_cpu (cpu,HK_TYPE_TICK ) || -cpuset_cpu_is_isolated (cpu); + !housekeeping_test_cpu (cpu,HK_TYPE_TICK ); }#endif /* _LINUX_SCHED_ISOLATION_H */
kernel/cgroup/cpuset.c +0 -11
@@ -1455,17 +1455,6 @@ static void update_isolation_cpumasks(void) isolcpus_update_state.updating = false; } -/** -* cpuset_cpu_is_isolated - Check if the given CPU is isolated -* @cpu: the CPU number to be checked -* Return: true if CPU is used in an isolated partition, false otherwise -*/ -bool cpuset_cpu_is_isolated (int cpu) -{ -return cpumask_test_cpu (cpu, isolated_cpus); -} -EXPORT_SYMBOL_GPL (cpuset_cpu_is_isolated); -/* * compute_effective_exclusive_cpumask - compute effective exclusive CPUs * @cs: cpuset
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 10/18 sched/core: Ignore DL BW deactivation error if in cpuhp_offline_cb_mode
In reply to Waiman LongWith the new strategy of using CPU hotplug to improve CPU isolation and the optimization of delaying sched domain rebuild until the whole process completes, we can run into a problem in shutting down the last CPU of a partition and a -EBUSY error may be returned. This -EBUSY error is caused by failing the DL BW check in dl_bw_deactivate().
As the CPU deactivation is only temporary and it will be brought back up again in a short moment, there is no point in failing the operation because of this DL BW error in this transitioning period. Fix this problem by ignoring this error when in CPU hotplug offline callback mode (cpuhp_offline_cb_mode is on).
--- kernel/sched/core.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/core.c +5 -1
@@ -8469,7 +8469,11 @@ int sched_cpu_deactivate(unsigned int cpu) ret =dl_bw_deactivate (cpu); -if (ret) +/* +* Ignore DL BW error if in cpuhp offline callback mode as CPU +* deactivation is only temporary. +*/ +if (ret && !cpuhp_offline_cb_mode)return ret;/*
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 11/18 tick/nohz: Make nohz_full parameter optional
In reply to Waiman LongTo provide nohz_full tick support, there is a set of tick dependency masks that need to be evaluated on every IRQ and context switch. Switching on nohz_full tick support at runtime will be problematic as some of the tick dependency masks may not be properly set causing problem down the road.
Allow nohz_full boot option to be specified without any parameter to force enable nohz_full tick support without any CPU in the tick_nohz_full_mask yet. The context_tracking_key and tick_nohz_full_running flag will be enabled in this case to make tick_nohz_full_enabled() return true.
There is still a small performance overhead by force enable nohz_full this way. So it should only be used if there is a chance that some CPUs may become isolated later via the cpuset isolated partition functionality and better CPU isolation closed to nohz_full is desired.
--- .../admin-guide/kernel-parameters.txt | 19 ++++++++++++------- include/linux/context_tracking.h | 7 ++++++- kernel/context_tracking.c | 4 +++- kernel/sched/isolation.c | 13 ++++++++++++- kernel/time/tick-sched.c | 11 +++++++++-- 5 files changed, 42 insertions(+), 12 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +12 -7
@@ -4260,15 +4260,20 @@ Valid arguments: on, off Default: on - nohz_full= [KNL,BOOT,SMP,ISOL] - The argument is a cpu list, as described above. + nohz_full[=cpu-list] + [KNL,BOOT,SMP,ISOL] In kernels built with CONFIG_NO_HZ_FULL=y, set - the specified list of CPUs whose tick will be stopped - whenever possible. The boot CPU will be forced outside - the range to maintain the timekeeping. Any CPUs - in this list will have their RCU callbacks offloaded, + the specified list of CPUs whose tick will be + stopped whenever possible. If the argument is + not specified, nohz_full will be forced enabled + without any CPU in the nohz_full list yet. + The boot CPU will be forced outside the range + to maintain the timekeeping. Any CPUs in this + list will have their RCU callbacks offloaded, just as if they had also been called out in the - rcu_nocbs= boot parameter. + rcu_nocbs= boot parameter. There is no need + to use rcu_nocbs= boot parameter if nohz_full + has been set which will override rcu_nocbs. Note that this argument takes precedence over the CONFIG_RCU_NOCB_CPU_DEFAULT_ALL option.
include/linux/context_tracking.h +6 -1
@@ -9,8 +9,13 @@#include <asm/ptrace.h> -#ifdef CONFIG_CONTEXT_TRACKING_USER +/* +* Pass CONTEXT_TRACKING_FORCE_ENABLE to ct_cpu_track_user() to force enable +* user context tracking. +*/ +#define CONTEXT_TRACKING_FORCE_ENABLE (-1) +extern void ct_cpu_track_user (int cpu);/* Called with interrupts disabled. */
kernel/context_tracking.c +3 -1
@@ -698,7 +698,9 @@ void __init ct_cpu_track_user(int cpu) {static __initdata bool initialized = false; -if (!per_cpu (context_tracking.active, cpu)) { +if (cpu ==CONTEXT_TRACKING_FORCE_ENABLE ) { +static_branch_inc (&context_tracking_key); + }else if (!per_cpu (context_tracking.active, cpu)) {per_cpu (context_tracking.active, cpu) = true;static_branch_inc (&context_tracking_key); }
kernel/sched/isolation.c +12 -1
@@ -146,6 +146,7 @@ static int __init housekeeping_setup(char *str, unsigned long flags) }alloc_bootmem_cpumask_var (&non_housekeeping_mask); +if (cpulist_parse (str, non_housekeeping_mask) <0 ) {pr_warn ("Housekeeping: nohz_full= or isolcpus= incorrect CPU range\n" ); goto free_non_housekeeping_mask;
@@ -155,6 +156,13 @@ static int __init housekeeping_setup(char *str, unsigned long flags)cpumask_andnot (housekeeping_staging, cpu_possible_mask, non_housekeeping_mask); +/* +* Allow "nohz_full" without parameter to force enable nohz_full +* at boot time without any CPUs in the nohz_full list yet. +*/ +if ((flags &HK_FLAG_KERNEL_NOISE ) && !*str) + goto setup_housekeeping_staging; + first_cpu =cpumask_first_and (cpu_present_mask, housekeeping_staging);if (first_cpu >= nr_cpu_ids || first_cpu >= setup_max_cpus) {__cpumask_set_cpu (smp_processor_id (), housekeeping_staging);
@@ -168,6 +176,7 @@ static int __init housekeeping_setup(char *str, unsigned long flags)if (cpumask_empty (non_housekeeping_mask)) goto free_housekeeping_staging; +setup_housekeeping_staging:if (!housekeeping.flags) {/* First setup call ("nohz_full=" or "isolcpus=") */ enum hk_type type;
@@ -212,10 +221,12 @@ static int __init housekeeping_nohz_full_setup(char *str)unsigned long flags; flags =HK_FLAG_KERNEL_NOISE ; +if (*str =='=' ) + str++;return housekeeping_setup (str, flags); } -__setup ("nohz_full=" , housekeeping_nohz_full_setup); +__setup ("nohz_full" , housekeeping_nohz_full_setup);static int __init housekeeping_isolcpus_setup (char *str) {
kernel/time/tick-sched.c +9 -2
@@ -651,8 +651,15 @@ void __init tick_nohz_init(void) } } -for_each_cpu (cpu, tick_nohz_full_mask) -ct_cpu_track_user (cpu); +/* +* Force enable context_tracking_key if tick_nohz_full_mask empty +*/ +if (cpumask_empty (tick_nohz_full_mask)) { +ct_cpu_track_user (CONTEXT_TRACKING_FORCE_ENABLE ); + }else { +for_each_cpu (cpu, tick_nohz_full_mask) +ct_cpu_track_user (cpu); + } ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "kernel/nohz:predown", NULL,
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 12/18 tick/nohz: Introduce tick_nohz_full_update_cpus() to update tick_nohz_full_mask
In reply to Waiman LongWhen the list of HK_FLAG_KERNEL_NOISE housekeeping CPUs are changed, we will need to update tick_nohz_full_mask so that dynticks can work correctly. Introduce a new tick_nohz_full_update_cpus() function that can be called at run time to update tick_nohz_full_mask.
--- include/linux/tick.h | 2 ++ kernel/time/tick-sched.c | 6 ++++++ 2 files changed, 8 insertions(+)
include/linux/tick.h +2 -0
@@ -272,6 +272,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,extern void tick_nohz_full_kick_cpu (int cpu);extern void __tick_nohz_task_switch (void );extern void __init tick_nohz_full_setup (cpumask_var_t cpumask); +extern void tick_nohz_full_update_cpus (cpumask_var_t cpumask);#else static inline bool tick_nohz_full_enabled (void ) {return false; }static inline bool tick_nohz_full_cpu (int cpu) {return false; }
@@ -297,6 +298,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,static inline void tick_nohz_full_kick_cpu (int cpu) { }static inline void __tick_nohz_task_switch (void ) { }static inline void tick_nohz_full_setup (cpumask_var_t cpumask) { } +static inline void tick_nohz_full_update_cpus (cpumask_var_t cpumask) {return false; }#endif static inline void tick_nohz_task_switch (void )
kernel/time/tick-sched.c +6 -0
@@ -604,6 +604,12 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask) tick_nohz_full_running = true; } +/* Get the new set of run-time nohz CPU list from cpuset */ +void tick_nohz_full_update_cpus (cpumask_var_t cpumask) +{ +cpumask_copy (tick_nohz_full_mask, cpumask); +} +bool tick_nohz_cpu_hotpluggable (unsigned int cpu) {/*
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 13/18 tick/nohz: Allow runtime changes in full dynticks CPUs
In reply to Waiman LongFull dynticks can only be enabled if "nohz_full" boot option has been been specified with or without parameter. Any change in the list of nohz_full CPUs have to be reflected in tick_nohz_full_mask. So the newly introduced tick_nohz_full_update_cpus() will be called to update the mask.
We also need to enable CPU context tracking for those CPUs that are in tick_nohz_full_mask. So remove __init from tick_nohz_init() and ct_cpu_track_user() so that they be called later when an isolated cpuset partition is being created. The __ro_after_init attribute is taken away from context_tracking_key as well.
Also add a new ct_cpu_untrack_user() function to reverse the action of ct_cpu_track_user() in case we need to disable the nohz_full mode of a CPU.
With nohz_full enabled, the boot CPU (typically CPU 0) will be the tick CPU which cannot be shut down easily. So the boot CPU should not be used in an isolated cpuset partition.
With runtime modification of nohz_full CPUs, tick_do_timer_cpu can become TICK_DO_TIMER_NONE. So remove the two TICK_DO_TIMER_NONE WARN_ON_ONCE() calls in tick-sched.c to avoid unnecessary warnings.
--- include/linux/context_tracking.h | 1 + kernel/cgroup/cpuset.c | 23 ++++++++++++++++++++++- kernel/context_tracking.c | 17 ++++++++++++++--- kernel/time/tick-sched.c | 7 ------- 4 files changed, 37 insertions(+), 11 deletions(-)
include/linux/context_tracking.h +1 -0
@@ -17,6 +17,7 @@#define CONTEXT_TRACKING_FORCE_ENABLE (-1)extern void ct_cpu_track_user (int cpu); +extern void ct_cpu_untrack_user (int cpu);/* Called with interrupts disabled. */ extern void __ct_user_enter (enum ctx_state state);
kernel/cgroup/cpuset.c +22 -1
@@ -23,6 +23,7 @@*/ #include "cpuset-internal.h" +#include <linux/context_tracking.h> #include <linux/init.h> #include <linux/interrupt.h> #include <linux/kernel.h>
@@ -1361,7 +1362,7 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,*/ static int do_housekeeping_exclude_cpumask (void *arg__maybe_unused ) { -int ret; +int cpu, ret;struct cpumask *icpus = isolated_cpus;unsigned long flags =BIT (HK_TYPE_DOMAIN ) |BIT (HK_TYPE_KERNEL_NOISE );
@@ -1395,6 +1396,26 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused) ret =housekeeping_exclude_cpumask (icpus, flags);WARN_ON_ONCE ((ret <0 ) && (ret != -EOPNOTSUPP )); +#ifdef CONFIG_NO_HZ_FULL +/* +* To properly enable/disable nohz_full dynticks for the affected CPUs, +* the new nohz_full CPUs have to be copied to tick_nohz_full_mask and +* ct_cpu_track_user/ct_cpu_untrack_user() will have to be called +* for those CPUs that have their states changed. +*/ +if (tick_nohz_full_enabled ()) { +tick_nohz_full_update_cpus (icpus); +for_each_cpu (cpu, isolcpus_update_state.cpus) { +if (cpumask_test_cpu (cpu, icpus)) +ct_cpu_track_user (cpu); +else +ct_cpu_untrack_user (cpu); + } + }else { +pr_warn_once ("Full dynticks cannot be enabled without the nohz_full kernel boot parameter!\n" ); + } +#endif +if (icpus != isolated_cpus)kfree (icpus);return ret;
kernel/context_tracking.c +14 -3
@@ -431,7 +431,7 @@ static __always_inline void ct_kernel_enter(bool user, int offset) { }#define CREATE_TRACE_POINTS #include <trace/events/context_tracking.h> -DEFINE_STATIC_KEY_FALSE_RO (context_tracking_key); +DEFINE_STATIC_KEY_FALSE (context_tracking_key);EXPORT_SYMBOL_GPL (context_tracking_key); static noinstr bool context_tracking_recursion_enter(void)
@@ -694,9 +694,9 @@ void user_exit_callable(void) }NOKPROBE_SYMBOL (user_exit_callable); -void __init ct_cpu_track_user (int cpu) +void ct_cpu_track_user (int cpu) { -static __initdata bool initialized = false; +static bool initialized;if (cpu ==CONTEXT_TRACKING_FORCE_ENABLE ) {static_branch_inc (&context_tracking_key);
@@ -720,6 +720,17 @@ void __init ct_cpu_track_user(int cpu) initialized = true; } +void ct_cpu_untrack_user (int cpu) +{ +#ifndef CONFIG_CONTEXT_TRACKING_USER_FORCE +if (!per_cpu (context_tracking.active, cpu)) +return ; + +per_cpu (context_tracking.active, cpu) = false; +static_branch_dec (&context_tracking_key); +#endif +} +#ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE void __init context_tracking_init (void ) {
kernel/time/tick-sched.c +0 -7
@@ -220,9 +220,6 @@ static void tick_sched_do_timer(struct tick_sched *ts, ktime_t now) tick_cpu =READ_ONCE (tick_do_timer_cpu);if (IS_ENABLED (CONFIG_NO_HZ_COMMON ) &&unlikely (tick_cpu ==TICK_DO_TIMER_NONE )) { -#ifdef CONFIG_NO_HZ_FULL -WARN_ON_ONCE (tick_nohz_full_running); -#endif WRITE_ONCE (tick_do_timer_cpu, cpu); tick_cpu = cpu; }
@@ -1201,10 +1198,6 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)*/ if (tick_cpu == cpu)return false; - -/* Should not happen for nohz-full */ -if (WARN_ON_ONCE (tick_cpu ==TICK_DO_TIMER_NONE )) -return false; }return true;
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 14/18 tick: Pass timer tick job to an online HK CPU in tick_cpu_dying()
In reply to Waiman LongIn tick_cpu_dying(), if the dying CPU is the current timekeeper, it has to pass the job over to another CPU. The current code passes it to another online CPU. However, that CPU may not be a timer tick housekeeping CPU. If that happens, another CPU will have to manually take it over again later. Avoid this unnecessary work by directly assigning an online housekeeping CPU.
Use READ_ONCE/WRITE_ONCE() to access tick_do_timer_cpu in case the non-HK CPUs may not be in stop machine in the future.
--- kernel/time/tick-common.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/time/tick-common.c +11 -4
@@ -17,6 +17,7 @@#include <linux/profile.h> #include <linux/sched.h> #include <linux/module.h> +#include <linux/sched/isolation.h> #include <trace/events/power.h> #include <asm/irq_regs.h>
@@ -394,12 +395,18 @@ int tick_cpu_dying(unsigned int dying_cpu) {/* * If the current CPU is the timekeeper, it's the only one that can -* safely hand over its duty. Also all online CPUs are in stop -* machine, guaranteed not to be idle, therefore there is no +* safely hand over its duty. Also all online housekeeping CPUs are +* in stop machine, guaranteed not to be idle, therefore there is no * concurrency and it's safe to pick any online successor. */ -if (tick_do_timer_cpu == dying_cpu) - tick_do_timer_cpu =cpumask_first (cpu_online_mask); +if (READ_ONCE (tick_do_timer_cpu) == dying_cpu) { +unsigned int new_cpu; + + new_cpu =cpumask_first_and (cpu_online_mask,housekeeping_cpumask (HK_TYPE_TICK )); +if (WARN_ON_ONCE (new_cpu >= nr_cpu_ids)) + new_cpu =cpumask_first (cpu_online_mask); +WRITE_ONCE (tick_do_timer_cpu, new_cpu); + }/* Make sure the CPU won't try to retake the timekeeping duty */ tick_sched_timer_dying (dying_cpu);
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 15/18 cgroup/cpuset: Enable RCU NO-CB CPU offloading of newly isolated CPUs
In reply to Waiman LongMake use of the provided rcu_nocb_cpu_offload()/rcu_nocb_cpu_deoffload() APIs to enable RCU NO-CB CPU offloading of newly isolated CPUs and deoffloading of de-isolated CPUs.
Also add a new rcu_nocbs_enabled() helper function to determine if RCU NO-CB CPU offloading can be done.
As nohz_full can now be specified without any CPU list, drop the test for cpumask_empty(tick_nohz_full_mask) in rcu_init_nohz().
The RCU NO-CB CPU offloading feature can only used if either "rcs_nocbs" or the "nohz_full" boot command parameters are used so that the proper RCU NO-CB resources are properly initialized at boot time.
--- include/linux/rcupdate.h | 2 ++ kernel/cgroup/cpuset.c | 14 ++++++++++++++ kernel/rcu/tree_nocb.h | 7 ++++++- 3 files changed, 22 insertions(+), 1 deletion(-)
include/linux/rcupdate.h +2 -0
@@ -140,6 +140,7 @@ void rcu_init_nohz(void);int rcu_nocb_cpu_offload (int cpu);int rcu_nocb_cpu_deoffload (int cpu);void rcu_nocb_flush_deferred_wakeup (void ); +bool rcu_nocbs_enabled (void );#define RCU_NOCB_LOCKDEP_WARN (c, s) RCU_LOCKDEP_WARN(c, s)
@@ -149,6 +150,7 @@ static inline void rcu_init_nohz(void) { }static inline int rcu_nocb_cpu_offload (int cpu) {return -EINVAL ; }static inline int rcu_nocb_cpu_deoffload (int cpu) {return 0 ; }static inline void rcu_nocb_flush_deferred_wakeup (void ) { } +static inline bool rcu_nocbs_enabled (void ) {return false; }#define RCU_NOCB_LOCKDEP_WARN (c, s)
kernel/cgroup/cpuset.c +14 -0
@@ -1416,6 +1416,20 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused) }#endif +if (rcu_nocbs_enabled ()) { +/* +* Enable RCU NO-CB CPU offloading/deoffloading for the affected CPUs +*/ +for_each_cpu (cpu, isolcpus_update_state.cpus) { +if (cpumask_test_cpu (cpu, icpus)) + ret =rcu_nocb_cpu_offload (cpu); +else + ret =rcu_nocb_cpu_deoffload (cpu); +if (WARN_ON_ONCE (ret)) +break ; + } + } +if (icpus != isolated_cpus)kfree (icpus);return ret;
kernel/rcu/tree_nocb.h +6 -1
@@ -1293,7 +1293,7 @@ void __init rcu_init_nohz(void)struct shrinker *__maybe_unused lazy_rcu_shrinker;#if defined(CONFIG_NO_HZ_FULL ) -if (tick_nohz_full_running && !cpumask_empty (tick_nohz_full_mask)) +if (tick_nohz_full_running) cpumask = tick_nohz_full_mask;#endif
@@ -1365,6 +1365,11 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)mutex_init (&rdp->nocb_gp_kthread_mutex); } +bool rcu_nocbs_enabled (void ) +{ +return !!rcu_state.nocb_is_setup; +} +/* * If the specified CPU is a no-CBs CPU that does not already have its * rcuo CB kthread, spawn it. Additionally, if the rcuo GP kthread
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 16/18 cgroup/cpuset: Don't set have_boot_nohz_full without any boot time nohz_full CPU
In reply to Waiman LongAs HK_TYPE_KERNEL_NOISE bit can now be set without any nohz_full CPU specified at boot time, don't set have_boot_nohz_full in this case.
--- kernel/cgroup/cpuset.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/cgroup/cpuset.c +6 -1
@@ -3871,7 +3871,12 @@ int __init cpuset_init(void)BUG_ON (!alloc_cpumask_var (&cpus_attach,GFP_KERNEL )); - have_boot_nohz_full =housekeeping_enabled (HK_TYPE_KERNEL_NOISE ); +/* +* HK_TYPE_KERNEL_NOISE bit can be set without any nohz_full CPU +*/ + have_boot_nohz_full =housekeeping_enabled (HK_TYPE_KERNEL_NOISE ) && + !cpumask_equal (cpu_possible_mask, +housekeeping_cpumask (HK_TYPE_KERNEL_NOISE )); have_boot_isolcpus =housekeeping_enabled (HK_TYPE_DOMAIN );if (have_boot_nohz_full) {BUG_ON (!alloc_cpumask_var (&boot_nohz_full_hk_cpus,GFP_KERNEL ));
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 17/18 cgroup/cpuset: Documentation updates & don't use CPU 0 for isolated partition
In reply to Waiman LongAs CPU hotplug is now used to improve CPU isolation of CPUs in isolated partitions. The boot CPU (typically CPU 0) cannot be put offline impacting the amount of CPU isolation available. Now we have to advise users that the boot CPU should never be used for isolated partitions. A warning will be printed when boot CPU is used and the cgroup-v2.rst is updated accordingly. The test_cpuset_prs.sh selftest is also updated to remove CPU 0 when forming isolated partitions.
Also update the cgroup-v2.rst file to show the need to specify the "nohz_full" kernel boot parameter to enable better nohz_full behavior for the CPUs in isolated partitions as well as the latency spike issue with using CPU hotplug.
--- Documentation/admin-guide/cgroup-v2.rst | 33 +++++++++++++++---- .../selftests/cgroup/test_cpuset_prs.sh | 8 ++--- 2 files changed, 31 insertions(+), 10 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +27 -6
@@ -2556,11 +2556,12 @@ Cpuset Interface Files It accepts only the following input values when written to. - ========== ===================================== + ========== =============================================== "member" Non-root member of a partition "root" Partition root - "isolated" Partition root without load balancing - ========== ===================================== + "isolated" Partition root without load balancing and other + OS noises + ========== =============================================== A cpuset partition is a collection of cpuset-enabled cgroups with a partition root at the top of the hierarchy and its descendants
@@ -2593,9 +2594,29 @@ Cpuset Interface Files When set to "isolated", the CPUs in that partition will be in an isolated state without any load balancing from the scheduler - and excluded from the unbound workqueues. Tasks placed in such - a partition with multiple CPUs should be carefully distributed - and bound to each of the individual CPUs for optimal performance. + and excluded from the unbound workqueues as well as without + other OS noises. Tasks placed in such a partition with multiple + CPUs should be carefully distributed and bound to each of the + individual CPUs for optimal performance. + + As CPU hotplug, if supported, is used to improve the degree of + CPU isolation close to the "nohz_full" kernel boot parameter. + The boot CPU (typically CPU 0) cannot be brought offline, so the + boot CPU should not be used for forming isolated partitions. + The "nohz_full" kernel boot parameter needs to be present to + enable full dynticks support and RCU no-callback CPU mode for + CPUs in isolated partitions even if the optional cpu list + isn't provided. Without that, adding the "rcu_nocbs" boot + kernel parameter without the cpu list can be used to enable + RCU no-callback CPU mode without full dynticks. + + Using CPU hotplug for creating or destroying an isolated + partition can cause latency spike in applications running + in other isolated partitions. A reserved list of CPUs can + optionally be put in the "nohz_full" kernel boot parameter to + alleviate this problem. When these reserved CPUs are used for + isolated partitions, CPU hotplug won't need to be invoked and + so there won't be latency spike in other isolated partitions. A partition root ("root" or "isolated") can be in one of the two possible states - valid or invalid. An invalid partition
tools/testing/selftests/cgroup/test_cpuset_prs.sh +4 -4
@@ -318,8 +318,8 @@ TEST_MATRIX=( # Invalid to valid local partition direct transition tests " C1-3:S+:P2 X4:P2 . . . . . . 0 A1:1-3|XA1:1-3|A2:1-3:XA2: A1:P2|A2:P-2 1-3" " C1-3:S+:P2 X4:P2 . . . X3:P2 . . 0 A1:1-2|XA1:1-3|A2:3:XA2:3 A1:P2|A2:P2 1-3" - " C0-3:P2 . . C4-6 C0-4 . . . 0 A1:0-4|B1:4-6 A1:P-2|B1:P0" - " C0-3:P2 . . C4-6 C0-4:C0-3 . . . 0 A1:0-3|B1:4-6 A1:P2|B1:P0 0-3" + " C1-3:P2 . . C4-6 C1-4 . . . 0 A1:1-4|B1:4-6 A1:P-2|B1:P0" + " C1-3:P2 . . C4-6 C1-4:C1-3 . . . 0 A1:1-3|B1:4-6 A1:P2|B1:P0 1-3" # Local partition invalidation tests " C0-3:X1-3:S+:P2 C1-3:X2-3:S+:P2 C2-3:X3:P2 \
@@ -329,8 +329,8 @@ TEST_MATRIX=( " C0-3:X1-3:S+:P2 C1-3:X2-3:S+:P2 C2-3:X3:P2 \ . . C4:X . . 0 A1:1-3|A2:1-3|A3:2-3|XA2:|XA3: A1:P2|A2:P-2|A3:P-2 1-3" # Local partition CPU change tests - " C0-5:S+:P2 C4-5:S+:P1 . . . C3-5 . . 0 A1:0-2|A2:3-5 A1:P2|A2:P1 0-2" - " C0-5:S+:P2 C4-5:S+:P1 . . C1-5 . . . 0 A1:1-3|A2:4-5 A1:P2|A2:P1 1-3" + " C1-5:S+:P2 C4-5:S+:P1 . . . C3-5 . . 0 A1:1-2|A2:3-5 A1:P2|A2:P1 1-2" + " C1-5:S+:P2 C4-5:S+:P1 . . C2-5 . . . 0 A1:2-3|A2:4-5 A1:P2|A2:P1 2-3" # cpus_allowed/exclusive_cpus update tests " C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \
Signature · 1 line
2.50.0
Waiman Long
maintainerRFC PATCH 18/18 cgroup/cpuset: Add pr_debug() statements for cpuhp_offline_cb() call
In reply to Waiman LongAdd some pr_debug() statements to actions performed related to the cpuhp_offline_cb() call to aid debugging. Since rcu_nocb_cpu_offload() and rcu_nocb_cpu_deoffload() will print out some info text, there is no need to add pr_debug() statements for them.
Also update test_cpuset_prs.sh test script to enable printing of dynamic debug messages for the kernel/cgroup/cpuset.c file when loglevel is 7 (debug).
--- kernel/cgroup/cpuset.c | 18 +++++++++++++----- .../selftests/cgroup/test_cpuset_prs.sh | 7 +++++++ 2 files changed, 20 insertions(+), 5 deletions(-)
kernel/cgroup/cpuset.c +13 -5
@@ -21,6 +21,7 @@* License. See the file COPYING in the main directory of the Linux * distribution for more details. */ +#define pr_fmt (fmt) "cpuset: " fmt#include "cpuset-internal.h" #include <linux/context_tracking.h>
@@ -1406,10 +1407,13 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)if (tick_nohz_full_enabled ()) {tick_nohz_full_update_cpus (icpus);for_each_cpu (cpu, isolcpus_update_state.cpus) { -if (cpumask_test_cpu (cpu, icpus)) +if (cpumask_test_cpu (cpu, icpus)) { +pr_debug ("Add CPU %d to nohz_full\n" , cpu);ct_cpu_track_user (cpu); -else + }else { +pr_debug ("Remove CPU %d from nohz_full\n" , cpu);ct_cpu_untrack_user (cpu); + } } }else {pr_warn_once ("Full dynticks cannot be enabled without the nohz_full kernel boot parameter!\n" );
@@ -1425,6 +1429,7 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused) ret =rcu_nocb_cpu_offload (cpu); else ret =rcu_nocb_cpu_deoffload (cpu); +if (WARN_ON_ONCE (ret))break ; }
@@ -1468,11 +1473,14 @@ static void update_isolation_cpumasks(void)* Without any change in the set of nohz_full CPUs, we don't really * need to use CPU hotplug for making change in HK cpumasks. */ -if (cpumask_empty (isolcpus_update_state.cpus)) +if (cpumask_empty (isolcpus_update_state.cpus)) { ret =do_housekeeping_exclude_cpumask (NULL ); -else + }else { +pr_debug ("cpuhp_offline_cb() called for CPUs %*pbl\n" , +cpumask_pr_args (isolcpus_update_state.cpus)); ret =cpuhp_offline_cb (isolcpus_update_state.cpus, do_housekeeping_exclude_cpumask,NULL ); + }/* * A errno value of -EPERM may be returned from cpuhp_offline_cb() if * any one of the CPUs in isolcpus_update_state.cpus can't be brought
@@ -1481,7 +1489,7 @@ static void update_isolation_cpumasks(void)* isolated partition. */ if (ret == -EPERM ) -pr_warn_once ("cpuset: The boot CPU shouldn't be used for isolated partition\n" ); +pr_warn_once ("The boot CPU shouldn't be used for isolated partition\n" );else WARN_ON_ONCE (ret <0 );
tools/testing/selftests/cgroup/test_cpuset_prs.sh +7 -0
@@ -67,6 +67,13 @@ then echo Y > /sys/kernel/debug/sched/verbose fi +# Enable dynamic debug messages for cpuset only +DYN_DEBUG=/sys/kernel/debug/dynamic_debug/control +[[ -f $DYN_DEBUG ]] && { + echo "-p" > $DYN_DEBUG + echo "file kernel/cgroup/cpuset.c +p" > $DYN_DEBUG +} + cd $CGROUP2 echo +cpuset > cgroup.subtree_control
Signature · 1 line
2.50.0
Frederic Weisbecker
maintainerQuoted 13 lines · Waiman Long
The "
nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the
nohz_fullandrcu_nocbsboot kernel parameters.
Quoted 13 lines
This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the
nohz_fullboot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthread
Thanks for working on that, I'm about to leave for 2 weeks vacation so I won't have the time to check this until I'm back.
However this series is highly conflicting with mine (cpuset/isolation: Honour kthreads preferred affinity). Your patchset even redoes things I'm doing (housekeeping cpumask update, RCU synchronization, HK_TYPE_DOMAIN to include cpusets, etc...)
I have a v2 that is almost ready to post.
Wouldn't it be better to wait for it and its infrastructure changes before proceeding with nohz_full?
Thanks.
Signature · 2 lines
Frederic Weisbecker SUSE Labs
Quoted 35 lines · Frederic Weisbecker
Quoted 13 lines · Waiman Long
The "
nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the
nohz_fullandrcu_nocbsboot kernel parameters.Quoted 13 lines
This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the
nohz_fullboot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthreadThanks for working on that, I'm about to leave for 2 weeks vacation so I won't have the time to check this until I'm back.
However this series is highly conflicting with mine (cpuset/isolation: Honour kthreads preferred affinity). Your patchset even redoes things I'm doing (housekeeping cpumask update, RCU synchronization,
HK_TYPE_DOMAINto include cpusets, etc...)
Quoted 4 lines
I have a v2 that is almost ready to post.
Wouldn't it be better to wait for it and its infrastructure changes before proceeding with
nohz_full?
Sure. I am just posting this RFC patch series to show my current idea that I have. I will wait for your v2 and integrate on top.
Looking forward to your upcoming v2 patch.
Cheers,
Longman