Skip to contentSkip to conversation

cgroup/cpuset: Enable runtime modification of

RFC PATCH In review Waiman Long 3 participants 21 messages Aug 8, 2025

Subsystems (9)
lore
Share & tools
lore

Waiman Long

author maintainer
#0 lore raw

cover cgroup/cpuset: Enable runtime modification of

The "nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.

There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the nohz_full and rcu_nocbs boot kernel parameters.

This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the nohz_full boot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthread

The rcu_nocbs is actually a subset of nohz_full focusing just on the RCU part of the kernel noises. The WQ part has already been handled by the current cpuset code.

This series focuses on the tick and RCU part of the kernel noises by actively changing their internal data structures to track changes in the list of isolated CPUs used by cpuset isolated partitions.

The dynamic update of the lists of housekeeping CPUs at run time will also have impact on the other part of the kernel noises that reference the lists of housekeeping CPUs at run time.

The pending patch series on timer migration[1], when properly integrated will support the timer part too.

The CPU hotplug functionality of the Linux kernel is used to facilitate the runtime change of the nohz_full isolated CPUs with minimal code changes. The CPUs that need to be switched from non-isolated to isolated or vice versa will be brought offline first, making the necessary changes and then brought back online afterward.

The use of CPU hotplug, however, does have a slight drawback of freezing all the other CPUs in part of the offlining process using the stop machine feature of the kernel. That will cause a noticeable latency spikes in other running applications which may be significant to sensitive applications running on isolated CPUs in other isolated partitions at the time. Hopefully we can find a way to solve this problem in the future.

One possible workaround for this is to reserve a set of nohz_full isolated CPUs at boot time using the nohz_full boot command parameter. The bringing of those nohz_full reserved CPUs into and out of isolated partitions will not invoke CPU hotplug and hence will not cause unexpected latency spikes. These reserved CPUs will only be needed if there are other existing isolated partitions running critical applications at the time when an isolated partition needs to be created.

Patches 1-4 updates the CPU isolation code at kernel/sched/isolation.c to enable dynamic update of the lists of housekeeping CPUs.

Patch 5 introduces a new cpuhp_offline_cb() API for shutting down the given set of CPUs, running the given callback method and then bringing those CPUs back online again. This new API will block any incoming hotplug events from interfering this operation.

Patches 6-9 updates the cpuset partition code to use the new cpuhp API to shut down the affect CPUs, making changes to the housekeeping cpumasks and then bring those CPUs online afterward.

Patch 10 works around an issue in the DL server code that block the hotplug operation under certain configurations.

Patch 11-14 updates the timer tick and related code to enable proper updates to the set of CPUs requiring nohz_full dynticks support.

Patch 15 enables runtime modification to the set of isolated CPUs requiring RCU NO-CB CPU support with minor changes to the RCU code.

Patches 16-18 includes other miscellaneous updates to cpuset code and documentation.

This patch series is applied on top of some other cpuset patches[1] posted upstream recently.

[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/

Waiman Long (18):
sched/isolation: Enable runtime update of housekeeping cpumasks sched/isolation: Call sched_tick_offload_init() when
HK_FLAG_KERNEL_NOISE is first set
sched/isolation: Use RCU to delay successive housekeeping cpumask
updates
sched/isolation: Add a debugfs file to dump housekeeping cpumasks cpu/hotplug: Add a new cpuhp_offline_cb() API cgroup/cpuset: Introduce a new top level isolcpus_update_mutex cgroup/cpuset: Allow overwriting HK_TYPE_DOMAIN housekeeping cpumask cgroup/cpuset: Use CPU hotplug to enable runtime nohz_full
modification
cgroup/cpuset: Revert "Include isolated cpuset CPUs in
cpu_is_isolated() check"
sched/core: Ignore DL BW deactivation error if in
cpuhp_offline_cb_mode
tick/nohz: Make nohz_full parameter optional
tick/nohz: Introduce tick_nohz_full_update_cpus() to update
tick_nohz_full_mask
tick/nohz: Allow runtime changes in full dynticks CPUs
tick: Pass timer tick job to an online HK CPU in tick_cpu_dying() cgroup/cpuset: Enable RCU NO-CB CPU offloading of newly isolated CPUs cgroup/cpuset: Don't set have_boot_nohz_full without any boot time
nohz_full CPU
cgroup/cpuset: Documentation updates & don't use CPU 0 for isolated
partition
cgroup/cpuset: Add pr_debug() statements for cpuhp_offline_cb() call

 Documentation/admin-guide/cgroup-v2.rst       |  33 +-
 .../admin-guide/kernel-parameters.txt         |  19 +-
 include/linux/context_tracking.h              |   8 +-
 include/linux/cpuhplock.h                     |   9 +
 include/linux/cpuset.h                        |   6 -
 include/linux/rcupdate.h                      |   2 +
 include/linux/sched/isolation.h               |   9 +-
 include/linux/tick.h                          |   2 +
 kernel/cgroup/cpuset.c                        | 344 ++++++++++++------
 kernel/context_tracking.c                     |  21 +-
 kernel/cpu.c                                  |  47 +++
 kernel/rcu/tree_nocb.h                        |   7 +-
 kernel/sched/core.c                           |   8 +-
 kernel/sched/debug.c                          |  32 ++
 kernel/sched/isolation.c                      | 151 +++++++-
 kernel/sched/sched.h                          |   2 +-
 kernel/time/tick-common.c                     |  15 +-
 kernel/time/tick-sched.c                      |  24 +-
 .../selftests/cgroup/test_cpuset_prs.sh       |  15 +-
 19 files changed, 583 insertions(+), 171 deletions(-)
Signature · 1 line
2.50.0
author maintainer
#1 lore raw

RFC PATCH 1/18 sched/isolation: Enable runtime update of housekeeping cpumasks

The housekeeping CPU masks, set up by the "isolcpus" and "nohz_full" boot command line options, are used at boot time to exclude selected CPUs from running some kernel background processes to minimize disturbance to latency sensitive userspace applications. Some of housekeeping CPU masks are also checked at run time to avoid using those isolated CPUs.

The cpuset subsystem is now able to dynamically create a set of isolated CPUs to be used in isolated cpuset partitions. The long term goal is to make the degree of isolation as close as possible to what can be done statically using those boot command line options.

This patch is a step in that direction by providing a new housekeep_exclude_cpumask() API to exclude only the given cpumask from the housekeeping cpumasks. Existing boot time "isolcpus" and "nohz_full" cpumask setup, if present, can be overwritten.

Two set of cpumasks are now kept internally. One set are used by the callers while the other set are being updated before the new set are atomically switched on.

---
 include/linux/sched/isolation.h |  6 +++
 kernel/sched/isolation.c        | 95 +++++++++++++++++++++++++++++----
 2 files changed, 91 insertions(+), 10 deletions(-)
include/linux/sched/isolation.h +6 -0
index d8501f4709b5..af38d21d0d00 100644
@@ -32,6 +32,7 @@ extern bool housekeeping_enabled(enum hk_type type);
 extern void housekeeping_affine(struct task_struct *t, enum hk_type type);
 extern bool housekeeping_test_cpu(int cpu, enum hk_type type);
 extern void __init housekeeping_init(void);
+extern int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long flags);
 
 #else
 
@@ -59,6 +60,11 @@ static inline bool housekeeping_test_cpu(int cpu, enum hk_type type)
 }
 
 static inline void housekeeping_init(void) { }
+
+static inline housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long flags)
+{
+	return -EOPNOTSUPP;
+}
 #endif /* CONFIG_CPU_ISOLATION */
 
 static inline bool housekeeping_cpu(int cpu, enum hk_type type)
kernel/sched/isolation.c +85 -10

150 lines not on this page · raw message

Signature · 1 line
2.50.0
author maintainer
#2 lore raw

RFC PATCH 2/18 sched/isolation: Call sched_tick_offload_init() when HK_FLAG_KERNEL_NOISE is first set

In reply to Waiman Long

The sched_tick_offload_init() function is called at boot time whenever "nohz_full" is set. Now housekeeping cpumasks can be updated at run time without the corresponding "nohz_full" kernel parameter. So we have to be able to call sched_tick_offload_init() at run time to allow tick offloading. Remove the __init attribute from sched_tick_offload_init() and call it when the HK_FLAG_KERNEL_NOISE flag is first set.

---
 kernel/sched/core.c      |  2 +-
 kernel/sched/isolation.c | 10 +++++++++-
 kernel/sched/sched.h     |  2 +-
 3 files changed, 11 insertions(+), 3 deletions(-)
kernel/sched/core.c +1 -1
index be00629f0ba4..9f02c047e25b 100644
@@ -5783,7 +5783,7 @@ static void sched_tick_stop(int cpu)
 }
 #endif /* CONFIG_HOTPLUG_CPU */
 
-int __init sched_tick_offload_init(void)
+int sched_tick_offload_init(void)
 {
 	tick_work_cpu = alloc_percpu(struct tick_work);
 	BUG_ON(!tick_work_cpu);
kernel/sched/isolation.c +9 -1
index 3fb0e8ccce26..ee396ae13719 100644
@@ -33,6 +33,7 @@ struct housekeeping {
 };
 
 static struct housekeeping housekeeping;
+static bool sched_tick_offload_inited;
 
 bool housekeeping_enabled(enum hk_type type)
 {
@@ -103,8 +104,10 @@ void __init housekeeping_init(void)
 
 	static_branch_enable(&housekeeping_overridden);
 
-	if (housekeeping.flags & HK_FLAG_KERNEL_NOISE)
+	if (housekeeping.flags & HK_FLAG_KERNEL_NOISE) {
 		sched_tick_offload_init();
+		sched_tick_offload_inited = true;
+	}
 
 	for_each_set_bit(type, &housekeeping.flags, HK_TYPE_MAX) {
 		/* We need at least one CPU to handle housekeeping work */
@@ -324,5 +327,10 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flags
 		static_key_disable(&housekeeping_overridden.key);
 	else if (housekeeping.flags && !static_key_enabled(&housekeeping_overridden))
 		static_key_enable(&housekeeping_overridden.key);
+
+	if ((housekeeping.flags & HK_FLAG_KERNEL_NOISE) && !sched_tick_offload_inited) {
+		sched_tick_offload_init();
+		sched_tick_offload_inited = true;
+	}
 	return 0;
 }
kernel/sched/sched.h +1 -1
index be9745d104f7..d4676305e099 100644
@@ -2671,7 +2671,7 @@ extern void post_init_entity_util_avg(struct task_struct *p);
 
 #ifdef CONFIG_NO_HZ_FULL
 extern bool sched_can_stop_tick(struct rq *rq);
-extern int __init sched_tick_offload_init(void);
+extern int sched_tick_offload_init(void);
 
 /*
  * Tick may be needed by tasks in the runqueue depending on their policy and
Signature · 1 line
2.50.0
author maintainer
#3 lore raw

RFC PATCH 3/18 sched/isolation: Use RCU to delay successive housekeeping cpumask updates

In reply to Waiman Long

Even though there are 2 separate sets of housekeeping cpumasks for access and update, it is possible that the set of cpumasks to be updated are still being used by the callers of housekeeping functions resulting in the use of an intermediate cpumask between the new and old ones.

To reduce the chance of this, we need to introduce delay between successive housekeeping cpumask updates. One simple way is to make use of the RCU grace period delay. The callers of the housekeeping APIs can optionally hold rcu_read_lock to eliminate the chance of using intermediate housekeeping cpumasks.

---
 kernel/sched/isolation.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
kernel/sched/isolation.c +33 -0
index ee396ae13719..f26708667754 100644
@@ -23,6 +23,9 @@ EXPORT_SYMBOL_GPL(housekeeping_overridden);
  * The housekeeping cpumasks can now be dynamically updated at run time.
  * Two set of cpumasks are kept. One set can be used while the other set are
  * being updated concurrently.
+ *
+ * rcu_read_lock() can optionally be held by housekeeping API callers to
+ * ensure stability of the cpumasks.
  */
 static DEFINE_RAW_SPINLOCK(cpumask_lock);
 struct housekeeping {
@@ -34,6 +37,8 @@ struct housekeeping {
 
 static struct housekeeping housekeeping;
 static bool sched_tick_offload_inited;
+static struct rcu_head rcu_gp[HK_TYPE_MAX];
+static unsigned long update_flags;
 
 bool housekeeping_enabled(enum hk_type type)
 {
@@ -267,6 +272,18 @@ static int __init housekeeping_isolcpus_setup(char *str)
 }
 __setup("isolcpus=", housekeeping_isolcpus_setup);
 
+/*
+ * Bits in update_flags can only turned on with cpumask_lock held and
+ * cleared by this RCU callback function.
+ */
+static void rcu_gp_end(struct rcu_head *rcu)
+{
+	int type = rcu - rcu_gp;
+
+	/* Atomically clear the corresponding flag bit */
+	clear_bit(type, &update_flags);
+}
+
 /**
  * housekeeping_exclude_cpumask - Update housekeeping cpumasks to exclude only the given cpumask
  * @cpumask:  new cpumask to be excluded from housekeeping cpumasks
@@ -306,8 +323,21 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flags
 	}
 #endif
 
+retry:
+	/*
+	 * If the RCU grace period for the previous update with conflicting
+	 * flag bits hasn't been completed yet, we have to wait for it.
+	 */
+	while (READ_ONCE(update_flags) & hk_flags)
+		synchronize_rcu();
+
 	raw_spin_lock(&cpumask_lock);
 
+	if (READ_ONCE(update_flags) & hk_flags) {
+		raw_spin_unlock(&cpumask_lock);
+		goto retry;
+	}
+
 	for_each_set_bit(type, &hk_flags, HK_TYPE_MAX) {
 		int idx = ++housekeeping.seq_nrs[type] & 1;
 		struct cpumask *dst_cpumask = housekeeping.cpumasks[type][idx];
@@ -320,8 +350,11 @@ int housekeeping_exclude_cpumask(struct cpumask *cpumask, unsigned long hk_flags
 			housekeeping.flags |= BIT(type);
 		}
 		WRITE_ONCE(housekeeping.cpumask_ptrs[type], dst_cpumask);
+		set_bit(type, &update_flags);
 	}
 	raw_spin_unlock(&cpumask_lock);
+	for_each_set_bit(type, &hk_flags, HK_TYPE_MAX)
+		call_rcu(&rcu_gp[type], rcu_gp_end);
 
 	if (!housekeeping.flags && static_key_enabled(&housekeeping_overridden))
 		static_key_disable(&housekeeping_overridden.key);
Signature · 1 line
2.50.0
author maintainer
#4 lore raw

RFC PATCH 4/18 sched/isolation: Add a debugfs file to dump housekeeping cpumasks

In reply to Waiman Long

As housekeeping cpumasks can now be modified at run time, we need a way to examine the their current values to see if they meet our expectation. Add a new sched debugfs file "housekeeping_cpumasks" to dump out the current values.

---
 kernel/sched/debug.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
kernel/sched/debug.c +32 -0
index 3f06ab84d53f..ba8f0334c15e 100644
@@ -490,6 +490,35 @@ static void debugfs_fair_server_init(void)
 	}
 }
 
+#ifdef CONFIG_CPU_ISOLATION
+static int hk_cpumasks_show(struct seq_file *m, void *v)
+{
+	static const char * const hk_type_name[HK_TYPE_MAX] = {
+		[HK_TYPE_DOMAIN]	= "domain",
+		[HK_TYPE_MANAGED_IRQ]	= "managed_irq",
+		[HK_TYPE_KERNEL_NOISE]	= "nohz_full"
+	};
+	int type;
+
+	for (type = 0; type < HK_TYPE_MAX; type++)
+		seq_printf(m, "%s: %*pbl\n", hk_type_name[type],
+			   cpumask_pr_args(housekeeping_cpumask(type)));
+	return 0;
+}
+
+static int hk_cpumasks_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, hk_cpumasks_show, NULL);
+}
+
+static const struct file_operations hk_cpumasks_fops = {
+	.open		= hk_cpumasks_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+#endif
+
 static __init int sched_init_debug(void)
 {
 	struct dentry __maybe_unused *numa;
@@ -525,6 +554,9 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
+#ifdef CONFIG_CPU_ISOLATION
+	debugfs_create_file("housekeeing_cpumasks", 0444, debugfs_sched, NULL, &hk_cpumasks_fops);
+#endif
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
 
 	debugfs_fair_server_init();
Signature · 1 line
2.50.0
author maintainer
#5 lore raw

RFC PATCH 5/18 cpu/hotplug: Add a new cpuhp_offline_cb() API

In reply to Waiman Long

Add a new cpuhp_offline_cb() API that allows us to offline a set of CPUs one-by-one, run the given callback function and then bring those CPUs back online again while inhibiting any concurrent CPU hotplug operations from happening.

This new API can be used to enable runtime adjustment of nohz_full and isolcpus boot command line options. A new cpuhp_offline_cb_mode flag is also added to signal that the system is in this offline callback transient state so that some hotplug operations can be optimized out if we choose to.

---
 include/linux/cpuhplock.h |  9 ++++++++
 kernel/cpu.c              | 47 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/linux/cpuhplock.h +9 -0
index f7aa20f62b87..b42b81361abc 100644
@@ -9,7 +9,9 @@
 
 #include <linux/cleanup.h>
 #include <linux/errno.h>
+#include <linux/cpumask_types.h>
 
+typedef int (*cpuhp_cb_t)(void *arg);
 struct device;
 
 extern int lockdep_is_cpus_held(void);
@@ -28,6 +30,8 @@ void clear_tasks_mm_cpumask(int cpu);
 int remove_cpu(unsigned int cpu);
 int cpu_device_down(struct device *dev);
 void smp_shutdown_nonboot_cpus(unsigned int primary_cpu);
+int cpuhp_offline_cb(struct cpumask *mask, cpuhp_cb_t func, void *arg);
+extern bool cpuhp_offline_cb_mode;
 
 #else /* CONFIG_HOTPLUG_CPU */
 
@@ -42,6 +46,11 @@ static inline void cpu_hotplug_disable(void) { }
 static inline void cpu_hotplug_enable(void) { }
 static inline int remove_cpu(unsigned int cpu) { return -EPERM; }
 static inline void smp_shutdown_nonboot_cpus(unsigned int primary_cpu) { }
+static inline int cpuhp_offline_cb(struct cpumask *mask, cpuhp_cb_t func, void *arg)
+{
+	return -EPERM;
+}
+#define cpuhp_offline_cb_mode	false
 #endif	/* !CONFIG_HOTPLUG_CPU */
 
 DEFINE_LOCK_GUARD_0(cpus_read_lock, cpus_read_lock(), cpus_read_unlock())
kernel/cpu.c +47 -0

53 lines not on this page · raw message

Signature · 1 line
2.50.0
author maintainer
#6 lore raw

RFC PATCH 6/18 cgroup/cpuset: Introduce a new top level isolcpus_update_mutex

In reply to Waiman Long

The current cpuset partition code is able to dynamically update the sched domains of a running system to perform what is essentally the "isolcpus=domain,..." boot command line feature at run time.

To enable runtime modification of nohz_full, we will have to make use of the CPU hotplug functionality to facilitate the proper addition or subtraction of nohz_full CPUs. In other word, we can't hold the cpu_hotplug_lock while doing so. Given the current lock ordering, we will need to introduce a new top level mutex to ensure proper mutual exclusion in case there is a need to update the cpuset states that may require the use of CPU hotplug. This patch introduces a new top level isolcpus_update_mutex for such purpose. This new mutex will be acquired in case the cpuset partition states or the set of isolated CPUs may have to be changed.

The update_unbound_workqueue_cpumask() is now renamed to update_isolation_cpumasks() and moved outside of cpu_hotplug_lock critical regions to enable its future extension to invoke CPU hotplug.

A new global isolcpus_update_state structure is added to track if update_isolation_cpumasks() will need to be invoked. So the existing partition_xcpus_add/del() functions and their callers can now be simplified.

---
 kernel/cgroup/cpuset.c | 149 ++++++++++++++++++++++++-----------------
 1 file changed, 86 insertions(+), 63 deletions(-)
kernel/cgroup/cpuset.c +86 -63
index 27adb04df675..2190efd33efb 100644
@@ -215,29 +215,39 @@ static struct cpuset top_cpuset = {
 };
 
 /*
- * There are two global locks guarding cpuset structures - cpuset_mutex and
- * callback_lock. The cpuset code uses only cpuset_mutex. Other kernel
- * subsystems can use cpuset_lock()/cpuset_unlock() to prevent change to cpuset
- * structures. Note that cpuset_mutex needs to be a mutex as it is used in
- * paths that rely on priority inheritance (e.g. scheduler - on RT) for
- * correctness.
+ * CPUSET Locking Convention
+ * -------------------------
  *
- * A task must hold both locks to modify cpusets.  If a task holds
- * cpuset_mutex, it blocks others, ensuring that it is the only task able to
- * also acquire callback_lock and be able to modify cpusets.  It can perform
- * various checks on the cpuset structure first, knowing nothing will change.
- * It can also allocate memory while just holding cpuset_mutex.  While it is
- * performing these checks, various callback routines can briefly acquire
- * callback_lock to query cpusets.  Once it is ready to make the changes, it
- * takes callback_lock, blocking everyone else.
+ * Below are the three global locks guarding cpuset structures in lock
+ * acquisition order:
+ *  - isolcpus_update_mutex
+ *  - cpu_hotplug_lock (cpus_read_lock/cpus_write_lock)
+ *  - cpuset_mutex
+ *  - callback_lock (raw spinlock)
  *
- * Calls to the kernel memory allocator can not be made while holding
- * callback_lock, as that would risk double tripping on callback_lock
- * from one of the callbacks into the cpuset code from within
- * __alloc_pages().
+ * The first isolcpus_update_mutex should only be held if the existing set of
+ * isolated CPUs (in isolated partition) or any of the partition states may be
+ * changed. Otherwise, it can be skipped. This is used to prevent concurrent
+ * updates to the set of isolated CPUs.
  *
- * If a task is only holding callback_lock, then it has read-only
- * access to cpusets.
+ * A task must hold all the remaining three locks to modify externally visible
+ * or used fields of cpusets, though some of the internally used cpuset fields
+ * can be modified by holding cpu_hotplug_lock and cpuset_mutex only. If only
+ * reliable read access of the externally used fields are needed, a task can
+ * hold either cpuset_mutex or callback_lock.
+ *
+ * If a task holds cpu_hotplug_lock and cpuset_mutex, it blocks others,
+ * ensuring that it is the only task able to also acquire callback_lock and
+ * be able to modify cpusets.  It can perform various checks on the cpuset
+ * structure first, knowing nothing will change. It can also allocate memory
+ * without holding callback_lock. While it is performing these checks, various
+ * callback routines can briefly acquire callback_lock to query cpusets.  Once
+ * it is ready to make the changes, it takes callback_lock, blocking everyone
+ * else.
+ *
+ * Calls to the kernel memory allocator cannot be made while holding
+ * callback_lock which is a spinlock, as the memory allocator may sleep or
+ * call back into cpuset code and acquire callback_lock.
  *
  * Now, the task_struct fields mems_allowed and mempolicy may be changed
  * by other task, we use alloc_lock in the task_struct fields to protect
@@ -248,6 +258,7 @@ static struct cpuset top_cpuset = {
  * cpumasks and nodemasks.
  */
 
+static DEFINE_MUTEX(isolcpus_update_mutex);
 static DEFINE_MUTEX(cpuset_mutex);
 
 void cpuset_lock(void)
@@ -272,6 +283,17 @@ void cpuset_callback_unlock_irq(void)
 	spin_unlock_irq(&callback_lock);
 }
 
+/*
+ * Isolcpus update state (protected by isolcpus_update_mutex mutex)
+ *
+ * It contains data related to updating the isolated CPUs configuration in
+ * isolated partitions.
+ */
+static struct {
+	bool updating;		/* Isolcpus updating in progress */
+	cpumask_var_t cpus;	/* CPUs to be updated */
+} isolcpus_update_state;
+
 static struct workqueue_struct *cpuset_migrate_mm_wq;
 
 static DECLARE_WAIT_QUEUE_HEAD(cpuset_attach_wq);
@@ -1273,6 +1295,9 @@ static void isolated_cpus_update(int old_prs, int new_prs, struct cpumask *xcpus
 		cpumask_or(isolated_cpus, isolated_cpus, xcpus);
 	else
 		cpumask_andnot(isolated_cpus, isolated_cpus, xcpus);
+
+	isolcpus_update_state.updating = true;
+	cpumask_or(isolcpus_update_state.cpus, isolcpus_update_state.cpus, xcpus);
 }
 
 /*
@@ -1280,31 +1305,26 @@ static void isolated_cpus_update(int old_prs, int new_prs, struct cpumask *xcpus
  * @new_prs: new partition_root_state
  * @parent: parent cpuset
  * @xcpus: exclusive CPUs to be added
- * Return: true if isolated_cpus modified, false otherwise
  *
  * Remote partition if parent == NULL
  */
-static bool partition_xcpus_add(int new_prs, struct cpuset *parent,
+static void partition_xcpus_add(int new_prs, struct cpuset *parent,
 				struct cpumask *xcpus)
 {
-	bool isolcpus_updated;
-
 	WARN_ON_ONCE(new_prs < 0);
 	lockdep_assert_held(&callback_lock);
 	if (!parent)
 		parent = &top_cpuset;
 
-
 	if (parent == &top_cpuset)
 		cpumask_or(subpartitions_cpus, subpartitions_cpus, xcpus);
 
-	isolcpus_updated = (new_prs != parent->partition_root_state);
-	if (isolcpus_updated)
+	if (new_prs != parent->partition_root_state)
 		isolated_cpus_update(parent->partition_root_state, new_prs,
 				     xcpus);
 
 	cpumask_andnot(parent->effective_cpus, parent->effective_cpus, xcpus);
-	return isolcpus_updated;
+	return;
 }
 
 /*
@@ -1312,15 +1332,12 @@ static bool partition_xcpus_add(int new_prs, struct cpuset *parent,
  * @old_prs: old partition_root_state
  * @parent: parent cpuset
  * @xcpus: exclusive CPUs to be removed
- * Return: true if isolated_cpus modified, false otherwise
  *
  * Remote partition if parent == NULL
  */
-static bool partition_xcpus_del(int old_prs, struct cpuset *parent,
+static void partition_xcpus_del(int old_prs, struct cpuset *parent,
 				struct cpumask *xcpus)
 {
-	bool isolcpus_updated;
-
 	WARN_ON_ONCE(old_prs < 0);
 	lockdep_assert_held(&callback_lock);
 	if (!parent)
@@ -1329,27 +1346,33 @@ static bool partition_xcpus_del(int old_prs, struct cpuset *parent,
 	if (parent == &top_cpuset)
 		cpumask_andnot(subpartitions_cpus, subpartitions_cpus, xcpus);
 
-	isolcpus_updated = (old_prs != parent->partition_root_state);
-	if (isolcpus_updated)
+	if (old_prs != parent->partition_root_state)
 		isolated_cpus_update(old_prs, parent->partition_root_state,
 				     xcpus);
 
 	cpumask_and(xcpus, xcpus, cpu_active_mask);
 	cpumask_or(parent->effective_cpus, parent->effective_cpus, xcpus);
-	return isolcpus_updated;
+	return;
 }
 
-static void update_unbound_workqueue_cpumask(bool isolcpus_updated)
+/**
+ * update_isolation_cpumasks - Update external isolation CPU masks
+ *
+ * The following external CPU masks will be updated if necessary:
+ * - workqueue unbound cpumask
+ */
+static void update_isolation_cpumasks(void)
 {
 	int ret;
 
-	lockdep_assert_cpus_held();
-
-	if (!isolcpus_updated)
+	if (!isolcpus_update_state.updating)
 		return;
 
 	ret = workqueue_unbound_exclude_cpumask(isolated_cpus);
 	WARN_ON_ONCE(ret < 0);
+
+	cpumask_clear(isolcpus_update_state.cpus);
+	isolcpus_update_state.updating = false;
 }
 
 /**
@@ -1441,8 +1464,6 @@ static inline bool is_local_partition(struct cpuset *cs)
 static int remote_partition_enable(struct cpuset *cs, int new_prs,
 				   struct tmpmasks *tmp)
 {
-	bool isolcpus_updated;
-
 	/*
 	 * The user must have sysadmin privilege.
 	 */
@@ -1466,11 +1487,10 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
 		return PERR_INVCPUS;
 
 	spin_lock_irq(&callback_lock);
-	isolcpus_updated = partition_xcpus_add(new_prs, NULL, tmp->new_cpus);
+	partition_xcpus_add(new_prs, NULL, tmp->new_cpus);
 	list_add(&cs->remote_sibling, &remote_children);
 	cpumask_copy(cs->effective_xcpus, tmp->new_cpus);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
 	cpuset_force_rebuild();
 	cs->prs_err = 0;
 
@@ -1493,15 +1513,12 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
  */
 static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp)
 {
-	bool isolcpus_updated;
-
 	WARN_ON_ONCE(!is_remote_partition(cs));
 	WARN_ON_ONCE(!cpumask_subset(cs->effective_xcpus, subpartitions_cpus));
 
 	spin_lock_irq(&callback_lock);
 	list_del_init(&cs->remote_sibling);
-	isolcpus_updated = partition_xcpus_del(cs->partition_root_state,
-					       NULL, cs->effective_xcpus);
+	partition_xcpus_del(cs->partition_root_state, NULL, cs->effective_xcpus);
 	if (cs->prs_err)
 		cs->partition_root_state = -cs->partition_root_state;
 	else
@@ -1511,7 +1528,6 @@ static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp)
 	compute_effective_exclusive_cpumask(cs, NULL, NULL);
 	reset_partition_data(cs);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
 	cpuset_force_rebuild();
 
 	/*
@@ -1536,7 +1552,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 {
 	bool adding, deleting;
 	int prs = cs->partition_root_state;
-	int isolcpus_updated = 0;
 
 	if (WARN_ON_ONCE(!is_remote_partition(cs)))
 		return;
@@ -1569,9 +1584,9 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 
 	spin_lock_irq(&callback_lock);
 	if (adding)
-		isolcpus_updated += partition_xcpus_add(prs, NULL, tmp->addmask);
+		partition_xcpus_add(prs, NULL, tmp->addmask);
 	if (deleting)
-		isolcpus_updated += partition_xcpus_del(prs, NULL, tmp->delmask);
+		partition_xcpus_del(prs, NULL, tmp->delmask);
 	/*
 	 * Need to update effective_xcpus and exclusive_cpus now as
 	 * update_sibling_cpumasks() below may iterate back to the same cs.
@@ -1580,7 +1595,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 	if (xcpus)
 		cpumask_copy(cs->exclusive_cpus, xcpus);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
 	if (adding || deleting)
 		cpuset_force_rebuild();
 
@@ -1662,7 +1676,6 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 	int old_prs, new_prs;
 	int part_error = PERR_NONE;	/* Partition error? */
 	int subparts_delta = 0;
-	int isolcpus_updated = 0;
 	struct cpumask *xcpus = user_xcpus(cs);
 	bool nocpu;
 
@@ -1932,18 +1945,15 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 	 * and vice versa.
 	 */
 	if (adding)
-		isolcpus_updated += partition_xcpus_del(old_prs, parent,
-							tmp->addmask);
+		partition_xcpus_del(old_prs, parent, tmp->addmask);
 	if (deleting)
-		isolcpus_updated += partition_xcpus_add(new_prs, parent,
-							tmp->delmask);
+		partition_xcpus_add(new_prs, parent, tmp->delmask);
 
 	if (is_partition_valid(parent)) {
 		parent->nr_subparts += subparts_delta;
 		WARN_ON_ONCE(parent->nr_subparts < 0);
 	}
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
 
 	if ((old_prs != new_prs) && (cmd == partcmd_update))
 		update_partition_exclusive_flag(cs, new_prs);
@@ -2968,7 +2978,6 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 	else if (isolcpus_updated)
 		isolated_cpus_update(old_prs, new_prs, cs->effective_xcpus);
 	spin_unlock_irq(&callback_lock);
-	update_unbound_workqueue_cpumask(isolcpus_updated);
 
 	/* Force update if switching back to member & update effective_xcpus */
 	update_cpumasks_hier(cs, &tmpmask, !new_prs);
@@ -3224,6 +3233,7 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 	int retval = -ENODEV;
 
 	buf = strstrip(buf);
+	mutex_lock(&isolcpus_update_mutex);
 	cpus_read_lock();
 	mutex_lock(&cpuset_mutex);
 	if (!is_cpuset_online(cs))
@@ -3256,6 +3266,8 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 out_unlock:
 	mutex_unlock(&cpuset_mutex);
 	cpus_read_unlock();
+	update_isolation_cpumasks();
+	mutex_unlock(&isolcpus_update_mutex);
 	flush_workqueue(cpuset_migrate_mm_wq);
 	return retval ?: nbytes;
 }
@@ -3358,12 +3370,15 @@ static ssize_t cpuset_partition_write(struct kernfs_open_file *of, char *buf,
 	else
 		return -EINVAL;
 
+	mutex_lock(&isolcpus_update_mutex);
 	cpus_read_lock();
 	mutex_lock(&cpuset_mutex);
 	if (is_cpuset_online(cs))
 		retval = update_prstate(cs, val);
 	mutex_unlock(&cpuset_mutex);
 	cpus_read_unlock();
+	update_isolation_cpumasks();
+	mutex_unlock(&isolcpus_update_mutex);
 	return retval ?: nbytes;
 }
 
@@ -3586,15 +3601,22 @@ static void cpuset_css_killed(struct cgroup_subsys_state *css)
 {
 	struct cpuset *cs = css_cs(css);
 
+	mutex_lock(&isolcpus_update_mutex);
+	/*
+	 * Here the partition root state can't be changed by user again.
+	 */
+	if (!is_partition_valid(cs))
+		goto out;
+
 	cpus_read_lock();
 	mutex_lock(&cpuset_mutex);
-
 	/* Reset valid partition back to member */
-	if (is_partition_valid(cs))
-		update_prstate(cs, PRS_MEMBER);
-
+	update_prstate(cs, PRS_MEMBER);
 	mutex_unlock(&cpuset_mutex);
 	cpus_read_unlock();
+	update_isolation_cpumasks();
+out:
+	mutex_unlock(&isolcpus_update_mutex);
 
 }
 
@@ -3751,6 +3773,7 @@ int __init cpuset_init(void)
 	BUG_ON(!alloc_cpumask_var(&top_cpuset.exclusive_cpus, GFP_KERNEL));
 	BUG_ON(!zalloc_cpumask_var(&subpartitions_cpus, GFP_KERNEL));
 	BUG_ON(!zalloc_cpumask_var(&isolated_cpus, GFP_KERNEL));
+	BUG_ON(!zalloc_cpumask_var(&isolcpus_update_state.cpus, GFP_KERNEL));
 
 	cpumask_setall(top_cpuset.cpus_allowed);
 	nodes_setall(top_cpuset.mems_allowed);
Signature · 1 line
2.50.0
author maintainer
#7 lore raw

RFC PATCH 7/18 cgroup/cpuset: Allow overwriting HK_TYPE_DOMAIN housekeeping cpumask

In reply to Waiman Long

As we did not modify housekeeping cpumasks in the creation of cpuset partition before, we had to disallow the creation of non-isolated partitions from using any of the HK_TYPE_DOMAIN isolated CPUs. Now we are going to modify housekeeping cpumasks at run time, we will now allow overwriting of HK_TYPE_DOMAIN cpumask when an isolated partition is first created or when the creation of a non-isolated partition conflicts with the boot time HK_TYPE_DOMAIN isolated CPUs. The unnecessary checking code are now being removed. The doc file will be updated in a later patch.

On the other hand, there is still a latency spike problem when CPU hotplug code is used to facilitate the proper functioning of the dynamically modified nohz_full HK_TYPE_KERNEL_NOISE cpumask. So the cpuset code will be modified to maintain the boot-time enabled nohz_full cpumask to avoid using cpu hotplug if all the newly isolated/non-isolated CPUs are already in that cpumask. This code will be removed in the future when the latency spike problem is solved.

---
 kernel/cgroup/cpuset.c | 45 ++++++++----------------------------------
 1 file changed, 8 insertions(+), 37 deletions(-)
kernel/cgroup/cpuset.c +8 -37
index 2190efd33efb..87e9ee7922cd 100644
@@ -59,7 +59,6 @@ static const char * const perr_strings[] = {
 	[PERR_NOCPUS]    = "Parent unable to distribute cpu downstream",
 	[PERR_HOTPLUG]   = "No cpu available due to hotplug",
 	[PERR_CPUSEMPTY] = "cpuset.cpus and cpuset.cpus.exclusive are empty",
-	[PERR_HKEEPING]  = "partition config conflicts with housekeeping setup",
 	[PERR_ACCESS]    = "Enable partition not permitted",
 	[PERR_REMOTE]    = "Have remote partition underneath",
 };
@@ -81,9 +80,10 @@ static cpumask_var_t	subpartitions_cpus;
 static cpumask_var_t	isolated_cpus;
 
 /*
- * Housekeeping (HK_TYPE_DOMAIN) CPUs at boot
+ * Housekeeping (nohz_full) CPUs at boot
  */
-static cpumask_var_t	boot_hk_cpus;
+static cpumask_var_t	boot_nohz_full_hk_cpus;
+static bool		have_boot_nohz_full;
 static bool		have_boot_isolcpus;
 
 /* List of remote partition root children */
@@ -1609,26 +1609,6 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
 	remote_partition_disable(cs, tmp);
 }
 
-/*
- * prstate_housekeeping_conflict - check for partition & housekeeping conflicts
- * @prstate: partition root state to be checked
- * @new_cpus: cpu mask
- * Return: true if there is conflict, false otherwise
- *
- * CPUs outside of boot_hk_cpus, if defined, can only be used in an
- * isolated partition.
- */
-static bool prstate_housekeeping_conflict(int prstate, struct cpumask *new_cpus)
-{
-	if (!have_boot_isolcpus)
-		return false;
-
-	if ((prstate != PRS_ISOLATED) && !cpumask_subset(new_cpus, boot_hk_cpus))
-		return true;
-
-	return false;
-}
-
 /**
  * update_parent_effective_cpumask - update effective_cpus mask of parent cpuset
  * @cs:      The cpuset that requests change in partition root state
@@ -1737,9 +1717,6 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		if (cpumask_empty(xcpus))
 			return PERR_INVCPUS;
 
-		if (prstate_housekeeping_conflict(new_prs, xcpus))
-			return PERR_HKEEPING;
-
 		/*
 		 * A parent can be left with no CPU as long as there is no
 		 * task directly associated with the parent partition.
@@ -2356,9 +2333,6 @@ static int update_cpumask(struct cpuset *cs, struct cpuset *trialcs,
 		    cpumask_empty(trialcs->effective_xcpus)) {
 			invalidate = true;
 			cs->prs_err = PERR_INVCPUS;
-		} else if (prstate_housekeeping_conflict(old_prs, trialcs->effective_xcpus)) {
-			invalidate = true;
-			cs->prs_err = PERR_HKEEPING;
 		} else if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) {
 			invalidate = true;
 			cs->prs_err = PERR_NOCPUS;
@@ -2499,9 +2473,6 @@ static int update_exclusive_cpumask(struct cpuset *cs, struct cpuset *trialcs,
 		if (cpumask_empty(trialcs->effective_xcpus)) {
 			invalidate = true;
 			cs->prs_err = PERR_INVCPUS;
-		} else if (prstate_housekeeping_conflict(old_prs, trialcs->effective_xcpus)) {
-			invalidate = true;
-			cs->prs_err = PERR_HKEEPING;
 		} else if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) {
 			invalidate = true;
 			cs->prs_err = PERR_NOCPUS;
@@ -3787,11 +3758,11 @@ int __init cpuset_init(void)
 
 	BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
 
-	have_boot_isolcpus = housekeeping_enabled(HK_TYPE_DOMAIN);
-	if (have_boot_isolcpus) {
-		BUG_ON(!alloc_cpumask_var(&boot_hk_cpus, GFP_KERNEL));
-		cpumask_copy(boot_hk_cpus, housekeeping_cpumask(HK_TYPE_DOMAIN));
-		cpumask_andnot(isolated_cpus, cpu_possible_mask, boot_hk_cpus);
+	have_boot_nohz_full = housekeeping_enabled(HK_TYPE_KERNEL_NOISE);
+	have_boot_isolcpus  = housekeeping_enabled(HK_TYPE_DOMAIN);
+	if (have_boot_nohz_full) {
+		BUG_ON(!alloc_cpumask_var(&boot_nohz_full_hk_cpus, GFP_KERNEL));
+		cpumask_copy(boot_nohz_full_hk_cpus, housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
 	}
 
 	return 0;
Signature · 1 line
2.50.0
author maintainer
#8 lore raw

RFC PATCH 8/18 cgroup/cpuset: Use CPU hotplug to enable runtime nohz_full modification

In reply to Waiman Long

One relatively simple way to allow runtime modification of nohz_full, and rcu_nocbs CPUs is to use the CPU hotplug to bring the affected CPUs offline first, making changes to the housekeeping cpumasks and then bring them back online. However, doing this will be rather costly in term of the number of CPU cycles needed. Still it is the easiet way to achieve the desired result and hopefully we can gradually reduce the overhead over time.

Use the newly introduced cpuhp_offline_cb() API to bring the affected CPUs offline, make the necessary housekeeping cpumask changes and then bring those CPUs back online again.

As HK_TYPE_DOMAIN cpumask is going to be updated at run time, we are going to reset any boot time isolcpus domain setting if an isolated partition or a conflicting non-isolated partition is going to be created.

Since rebuild_sched_domains() will be called at the end of update_isolation_cpumasks(), earlier rebuild_sched_domains_locked() calls will be suppressed to avoid unneeded work.

---
 kernel/cgroup/cpuset.c | 95 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 92 insertions(+), 3 deletions(-)
kernel/cgroup/cpuset.c +92 -3
index 87e9ee7922cd..60f336e50b05 100644
@@ -1355,11 +1355,57 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,
 	return;
 }
 
+/*
+ * We are only updating HK_TYPE_DOMAIN and HK_TYPE_KERNEL_NOISE housekeeping
+ * cpumask for now. HK_TYPE_MANAGED_IRQ will be handled later.
+ */
+static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
+{
+	int ret;
+	struct cpumask *icpus = isolated_cpus;
+	unsigned long flags = BIT(HK_TYPE_DOMAIN) | BIT(HK_TYPE_KERNEL_NOISE);
+
+	/*
+	 * The boot time isolcpus setting will be overwritten if set.
+	 */
+	have_boot_isolcpus = false;
+
+	if (have_boot_nohz_full) {
+		/*
+		 * Need to separate the handling of HK_TYPE_KERNEL_NOISE and
+		 * HK_TYPE_DOMAIN as different cpumasks will be used for each.
+		 */
+		ret = housekeeping_exclude_cpumask(icpus, BIT(HK_TYPE_DOMAIN));
+		WARN_ON_ONCE((ret < 0) && (ret != -EOPNOTSUPP));
+
+		if (cpumask_empty(isolcpus_update_state.cpus))
+			return ret;
+		flags = BIT(HK_TYPE_KERNEL_NOISE);
+		icpus = kmalloc(cpumask_size(), GFP_KERNEL);
+		if (WARN_ON_ONCE(!icpus))
+			return -ENOMEM;
+
+		/*
+		 * Add boot time nohz_full CPUs into the isolated CPUs list
+		 * for exclusion from HK_TYPE_KERNEL_NOISE CPUs.
+		 */
+		cpumask_andnot(icpus, cpu_possible_mask, boot_nohz_full_hk_cpus);
+		cpumask_or(icpus, icpus, isolated_cpus);
+	}
+	ret = housekeeping_exclude_cpumask(icpus, flags);
+	WARN_ON_ONCE((ret < 0) && (ret != -EOPNOTSUPP));
+
+	if (icpus != isolated_cpus)
+		kfree(icpus);
+	return ret;
+}
+
 /**
  * update_isolation_cpumasks - Update external isolation CPU masks
  *
  * The following external CPU masks will be updated if necessary:
  * - workqueue unbound cpumask
+ * - housekeeping cpumasks
  */
 static void update_isolation_cpumasks(void)
 {
@@ -1371,7 +1417,41 @@ static void update_isolation_cpumasks(void)
 	ret = workqueue_unbound_exclude_cpumask(isolated_cpus);
 	WARN_ON_ONCE(ret < 0);
 
+	 /*
+	  * Mask out offline and boot-time nohz_full non-housekeeping
+	  * CPUs from isolcpus_update_state.cpus to compute the set
+	  * of CPUs that need to be brought offline before calling
+	  * do_housekeeping_exclude_cpumask().
+	  */
+	cpumask_and(isolcpus_update_state.cpus,
+		    isolcpus_update_state.cpus, cpu_active_mask);
+	if (have_boot_nohz_full)
+		cpumask_and(isolcpus_update_state.cpus,
+			    isolcpus_update_state.cpus, boot_nohz_full_hk_cpus);
+
+	/*
+	 * Without any change in the set of nohz_full CPUs, we don't really
+	 * need to use CPU hotplug for making change in HK cpumasks.
+	 */
+	if (cpumask_empty(isolcpus_update_state.cpus))
+		ret = do_housekeeping_exclude_cpumask(NULL);
+	else
+		ret = cpuhp_offline_cb(isolcpus_update_state.cpus,
+				       do_housekeeping_exclude_cpumask, NULL);
+	/*
+	 * A errno value of -EPERM may be returned from cpuhp_offline_cb() if
+	 * any one of the CPUs in isolcpus_update_state.cpus can't be brought
+	 * offline. This can happen for the boot CPU (normally CPU 0) which
+	 * cannot be shut down. This CPU should not be used for creating
+	 * isolated partition.
+	 */
+	if (ret == -EPERM)
+		pr_warn_once("cpuset: The boot CPU shouldn't be used for isolated partition\n");
+	else
+		WARN_ON_ONCE(ret < 0);
+
 	cpumask_clear(isolcpus_update_state.cpus);
+	rebuild_sched_domains();
 	isolcpus_update_state.updating = false;
 }
 
@@ -2961,7 +3041,16 @@ static int update_prstate(struct cpuset *cs, int new_prs)
 	update_partition_sd_lb(cs, old_prs);
 
 	notify_partition_change(cs, old_prs);
-	if (force_sd_rebuild)
+
+	/*
+	 * If boot time domain isolcpus exists and it conflicts with the CPUs
+	 * in the new partition, we will have to reset HK_TYPE_DOMAIN cpumask.
+	 */
+	if (have_boot_isolcpus && (new_prs > PRS_MEMBER) &&
+	    !cpumask_subset(cs->effective_xcpus, housekeeping_cpumask(HK_TYPE_DOMAIN)))
+		isolcpus_update_state.updating = true;
+
+	if (force_sd_rebuild && !isolcpus_update_state.updating)
 		rebuild_sched_domains_locked();
 	free_cpumasks(NULL, &tmpmask);
 	return 0;
@@ -3232,7 +3321,7 @@ ssize_t cpuset_write_resmask(struct kernfs_open_file *of,
 	}
 
 	free_cpuset(trialcs);
-	if (force_sd_rebuild)
+	if (force_sd_rebuild && !isolcpus_update_state.updating)
 		rebuild_sched_domains_locked();
 out_unlock:
 	mutex_unlock(&cpuset_mutex);
@@ -3999,7 +4088,7 @@ static void cpuset_handle_hotplug(void)
 	}
 
 	/* rebuild sched domains if necessary */
-	if (force_sd_rebuild)
+	if (force_sd_rebuild && !isolcpus_update_state.updating)
 		rebuild_sched_domains_cpuslocked();
 
 	free_cpumasks(NULL, ptmp);
Signature · 1 line
2.50.0
author maintainer
#9 lore raw

RFC PATCH 9/18 cgroup/cpuset: Revert "Include isolated cpuset CPUs in cpu_is_isolated() check"

In reply to Waiman Long

Now that the HK_TYPE_DOMAIN cpumask is updated at run time to reflect changes made in isolated cpuset partitions. We no longer need a separate cpuset_cpu_is_isolated() function for checking isolated CPUs generated by cpuset. Revert commit 3232e7aad11e ("cgroup/cpuset: Include isolated cpuset CPUs in cpu_is_isolated() check").

---
 include/linux/cpuset.h          |  6 ------
 include/linux/sched/isolation.h |  3 +--
 kernel/cgroup/cpuset.c          | 11 -----------
 3 files changed, 1 insertion(+), 19 deletions(-)
include/linux/cpuset.h +0 -6
index 2ddb256187b5..a2ea8efebf36 100644
@@ -76,7 +76,6 @@ extern void cpuset_lock(void);
 extern void cpuset_unlock(void);
 extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
 extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
-extern bool cpuset_cpu_is_isolated(int cpu);
 extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
 #define cpuset_current_mems_allowed (current->mems_allowed)
 void cpuset_init_current_mems_allowed(void);
@@ -206,11 +205,6 @@ static inline bool cpuset_cpus_allowed_fallback(struct task_struct *p)
 	return false;
 }
 
-static inline bool cpuset_cpu_is_isolated(int cpu)
-{
-	return false;
-}
-
 static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
 {
 	return node_possible_map;
include/linux/sched/isolation.h +1 -2
index af38d21d0d00..0bc4b3368d39 100644
@@ -79,8 +79,7 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type)
 static inline bool cpu_is_isolated(int cpu)
 {
 	return !housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN) ||
-	       !housekeeping_test_cpu(cpu, HK_TYPE_TICK) ||
-	       cpuset_cpu_is_isolated(cpu);
+	       !housekeeping_test_cpu(cpu, HK_TYPE_TICK);
 }
 
 #endif /* _LINUX_SCHED_ISOLATION_H */
kernel/cgroup/cpuset.c +0 -11
index 60f336e50b05..6308bb14e018 100644
@@ -1455,17 +1455,6 @@ static void update_isolation_cpumasks(void)
 	isolcpus_update_state.updating = false;
 }
 
-/**
- * cpuset_cpu_is_isolated - Check if the given CPU is isolated
- * @cpu: the CPU number to be checked
- * Return: true if CPU is used in an isolated partition, false otherwise
- */
-bool cpuset_cpu_is_isolated(int cpu)
-{
-	return cpumask_test_cpu(cpu, isolated_cpus);
-}
-EXPORT_SYMBOL_GPL(cpuset_cpu_is_isolated);
-
 /*
  * compute_effective_exclusive_cpumask - compute effective exclusive CPUs
  * @cs: cpuset
Signature · 1 line
2.50.0
author maintainer
#10 lore raw

RFC PATCH 10/18 sched/core: Ignore DL BW deactivation error if in cpuhp_offline_cb_mode

In reply to Waiman Long

With the new strategy of using CPU hotplug to improve CPU isolation and the optimization of delaying sched domain rebuild until the whole process completes, we can run into a problem in shutting down the last CPU of a partition and a -EBUSY error may be returned. This -EBUSY error is caused by failing the DL BW check in dl_bw_deactivate().

As the CPU deactivation is only temporary and it will be brought back up again in a short moment, there is no point in failing the operation because of this DL BW error in this transitioning period. Fix this problem by ignoring this error when in CPU hotplug offline callback mode (cpuhp_offline_cb_mode is on).

---
 kernel/sched/core.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/sched/core.c +5 -1
index 9f02c047e25b..78f4ba73a9f2 100644
@@ -8469,7 +8469,11 @@ int sched_cpu_deactivate(unsigned int cpu)
 
 	ret = dl_bw_deactivate(cpu);
 
-	if (ret)
+	/*
+	 * Ignore DL BW error if in cpuhp offline callback mode as CPU
+	 * deactivation is only temporary.
+	 */
+	if (ret && !cpuhp_offline_cb_mode)
 		return ret;
 
 	/*
Signature · 1 line
2.50.0
author maintainer
#11 lore raw

RFC PATCH 11/18 tick/nohz: Make nohz_full parameter optional

In reply to Waiman Long

To provide nohz_full tick support, there is a set of tick dependency masks that need to be evaluated on every IRQ and context switch. Switching on nohz_full tick support at runtime will be problematic as some of the tick dependency masks may not be properly set causing problem down the road.

Allow nohz_full boot option to be specified without any parameter to force enable nohz_full tick support without any CPU in the tick_nohz_full_mask yet. The context_tracking_key and tick_nohz_full_running flag will be enabled in this case to make tick_nohz_full_enabled() return true.

There is still a small performance overhead by force enable nohz_full this way. So it should only be used if there is a chance that some CPUs may become isolated later via the cpuset isolated partition functionality and better CPU isolation closed to nohz_full is desired.

---
 .../admin-guide/kernel-parameters.txt         | 19 ++++++++++++-------
 include/linux/context_tracking.h              |  7 ++++++-
 kernel/context_tracking.c                     |  4 +++-
 kernel/sched/isolation.c                      | 13 ++++++++++++-
 kernel/time/tick-sched.c                      | 11 +++++++++--
 5 files changed, 42 insertions(+), 12 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +12 -7
index 747a55abf494..89a8161475b5 100644
@@ -4260,15 +4260,20 @@
 			Valid arguments: on, off
 			Default: on
 
-	nohz_full=	[KNL,BOOT,SMP,ISOL]
-			The argument is a cpu list, as described above.
+	nohz_full[=cpu-list]
+			[KNL,BOOT,SMP,ISOL]
 			In kernels built with CONFIG_NO_HZ_FULL=y, set
-			the specified list of CPUs whose tick will be stopped
-			whenever possible. The boot CPU will be forced outside
-			the range to maintain the timekeeping.  Any CPUs
-			in this list will have their RCU callbacks offloaded,
+			the specified list of CPUs whose tick will be
+			stopped whenever possible.  If the argument is
+			not specified, nohz_full will be forced enabled
+			without any CPU in the nohz_full list yet.
+			The boot CPU will be forced outside the range
+			to maintain the timekeeping.  Any CPUs in this
+			list will have their RCU callbacks offloaded,
 			just as if they had also been called out in the
-			rcu_nocbs= boot parameter.
+			rcu_nocbs= boot parameter.  There is no need
+			to use rcu_nocbs= boot parameter if nohz_full
+			has been set which will override rcu_nocbs.
 
 			Note that this argument takes precedence over
 			the CONFIG_RCU_NOCB_CPU_DEFAULT_ALL option.
include/linux/context_tracking.h +6 -1
index af9fe87a0922..a3fea7f9fef6 100644
@@ -9,8 +9,13 @@
 
 #include <asm/ptrace.h>
 
-
 #ifdef CONFIG_CONTEXT_TRACKING_USER
+/*
+ * Pass CONTEXT_TRACKING_FORCE_ENABLE to ct_cpu_track_user() to force enable
+ * user context tracking.
+ */
+#define CONTEXT_TRACKING_FORCE_ENABLE	(-1)
+
 extern void ct_cpu_track_user(int cpu);
 
 /* Called with interrupts disabled.  */
kernel/context_tracking.c +3 -1
index fb5be6e9b423..734354bbfdbb 100644
@@ -698,7 +698,9 @@ void __init ct_cpu_track_user(int cpu)
 {
 	static __initdata bool initialized = false;
 
-	if (!per_cpu(context_tracking.active, cpu)) {
+	if (cpu == CONTEXT_TRACKING_FORCE_ENABLE) {
+		static_branch_inc(&context_tracking_key);
+	} else if (!per_cpu(context_tracking.active, cpu)) {
 		per_cpu(context_tracking.active, cpu) = true;
 		static_branch_inc(&context_tracking_key);
 	}
kernel/sched/isolation.c +12 -1
index f26708667754..2bed4b2f9ec5 100644
@@ -146,6 +146,7 @@ static int __init housekeeping_setup(char *str, unsigned long flags)
 	}
 
 	alloc_bootmem_cpumask_var(&non_housekeeping_mask);
+
 	if (cpulist_parse(str, non_housekeeping_mask) < 0) {
 		pr_warn("Housekeeping: nohz_full= or isolcpus= incorrect CPU range\n");
 		goto free_non_housekeeping_mask;
@@ -155,6 +156,13 @@ static int __init housekeeping_setup(char *str, unsigned long flags)
 	cpumask_andnot(housekeeping_staging,
 		       cpu_possible_mask, non_housekeeping_mask);
 
+	/*
+	 * Allow "nohz_full" without parameter to force enable nohz_full
+	 * at boot time without any CPUs in the nohz_full list yet.
+	 */
+	if ((flags & HK_FLAG_KERNEL_NOISE) && !*str)
+		goto setup_housekeeping_staging;
+
 	first_cpu = cpumask_first_and(cpu_present_mask, housekeeping_staging);
 	if (first_cpu >= nr_cpu_ids || first_cpu >= setup_max_cpus) {
 		__cpumask_set_cpu(smp_processor_id(), housekeeping_staging);
@@ -168,6 +176,7 @@ static int __init housekeeping_setup(char *str, unsigned long flags)
 	if (cpumask_empty(non_housekeeping_mask))
 		goto free_housekeeping_staging;
 
+setup_housekeeping_staging:
 	if (!housekeeping.flags) {
 		/* First setup call ("nohz_full=" or "isolcpus=") */
 		enum hk_type type;
@@ -212,10 +221,12 @@ static int __init housekeeping_nohz_full_setup(char *str)
 	unsigned long flags;
 
 	flags = HK_FLAG_KERNEL_NOISE;
+	if (*str == '=')
+		str++;
 
 	return housekeeping_setup(str, flags);
 }
-__setup("nohz_full=", housekeeping_nohz_full_setup);
+__setup("nohz_full", housekeeping_nohz_full_setup);
 
 static int __init housekeeping_isolcpus_setup(char *str)
 {
kernel/time/tick-sched.c +9 -2
index c527b421c865..87b26a4471e7 100644
@@ -651,8 +651,15 @@ void __init tick_nohz_init(void)
 		}
 	}
 
-	for_each_cpu(cpu, tick_nohz_full_mask)
-		ct_cpu_track_user(cpu);
+	/*
+	 * Force enable context_tracking_key if tick_nohz_full_mask empty
+	 */
+	if (cpumask_empty(tick_nohz_full_mask)) {
+		ct_cpu_track_user(CONTEXT_TRACKING_FORCE_ENABLE);
+	} else {
+		for_each_cpu(cpu, tick_nohz_full_mask)
+			ct_cpu_track_user(cpu);
+	}
 
 	ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
 					"kernel/nohz:predown", NULL,
Signature · 1 line
2.50.0
author maintainer
#12 lore raw

RFC PATCH 12/18 tick/nohz: Introduce tick_nohz_full_update_cpus() to update tick_nohz_full_mask

In reply to Waiman Long

When the list of HK_FLAG_KERNEL_NOISE housekeeping CPUs are changed, we will need to update tick_nohz_full_mask so that dynticks can work correctly. Introduce a new tick_nohz_full_update_cpus() function that can be called at run time to update tick_nohz_full_mask.

---
 include/linux/tick.h     | 2 ++
 kernel/time/tick-sched.c | 6 ++++++
 2 files changed, 8 insertions(+)
include/linux/tick.h +2 -0
index ac76ae9fa36d..34907c0b632c 100644
@@ -272,6 +272,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,
 extern void tick_nohz_full_kick_cpu(int cpu);
 extern void __tick_nohz_task_switch(void);
 extern void __init tick_nohz_full_setup(cpumask_var_t cpumask);
+extern void tick_nohz_full_update_cpus(cpumask_var_t cpumask);
 #else
 static inline bool tick_nohz_full_enabled(void) { return false; }
 static inline bool tick_nohz_full_cpu(int cpu) { return false; }
@@ -297,6 +298,7 @@ static inline void tick_dep_clear_signal(struct signal_struct *signal,
 static inline void tick_nohz_full_kick_cpu(int cpu) { }
 static inline void __tick_nohz_task_switch(void) { }
 static inline void tick_nohz_full_setup(cpumask_var_t cpumask) { }
+static inline void tick_nohz_full_update_cpus(cpumask_var_t cpumask) { return false; }
 #endif
 
 static inline void tick_nohz_task_switch(void)
kernel/time/tick-sched.c +6 -0
index 87b26a4471e7..9204808b7a55 100644
@@ -604,6 +604,12 @@ void __init tick_nohz_full_setup(cpumask_var_t cpumask)
 	tick_nohz_full_running = true;
 }
 
+/* Get the new set of run-time nohz CPU list from cpuset */
+void tick_nohz_full_update_cpus(cpumask_var_t cpumask)
+{
+	cpumask_copy(tick_nohz_full_mask, cpumask);
+}
+
 bool tick_nohz_cpu_hotpluggable(unsigned int cpu)
 {
 	/*
Signature · 1 line
2.50.0
author maintainer
#13 lore raw

RFC PATCH 13/18 tick/nohz: Allow runtime changes in full dynticks CPUs

In reply to Waiman Long

Full dynticks can only be enabled if "nohz_full" boot option has been been specified with or without parameter. Any change in the list of nohz_full CPUs have to be reflected in tick_nohz_full_mask. So the newly introduced tick_nohz_full_update_cpus() will be called to update the mask.

We also need to enable CPU context tracking for those CPUs that are in tick_nohz_full_mask. So remove __init from tick_nohz_init() and ct_cpu_track_user() so that they be called later when an isolated cpuset partition is being created. The __ro_after_init attribute is taken away from context_tracking_key as well.

Also add a new ct_cpu_untrack_user() function to reverse the action of ct_cpu_track_user() in case we need to disable the nohz_full mode of a CPU.

With nohz_full enabled, the boot CPU (typically CPU 0) will be the tick CPU which cannot be shut down easily. So the boot CPU should not be used in an isolated cpuset partition.

With runtime modification of nohz_full CPUs, tick_do_timer_cpu can become TICK_DO_TIMER_NONE. So remove the two TICK_DO_TIMER_NONE WARN_ON_ONCE() calls in tick-sched.c to avoid unnecessary warnings.

---
 include/linux/context_tracking.h |  1 +
 kernel/cgroup/cpuset.c           | 23 ++++++++++++++++++++++-
 kernel/context_tracking.c        | 17 ++++++++++++++---
 kernel/time/tick-sched.c         |  7 -------
 4 files changed, 37 insertions(+), 11 deletions(-)
include/linux/context_tracking.h +1 -0
index a3fea7f9fef6..1a6b816f1ad6 100644
@@ -17,6 +17,7 @@
 #define CONTEXT_TRACKING_FORCE_ENABLE	(-1)
 
 extern void ct_cpu_track_user(int cpu);
+extern void ct_cpu_untrack_user(int cpu);
 
 /* Called with interrupts disabled.  */
 extern void __ct_user_enter(enum ctx_state state);
kernel/cgroup/cpuset.c +22 -1
index 6308bb14e018..45c82c18bec4 100644
@@ -23,6 +23,7 @@
  */
 #include "cpuset-internal.h"
 
+#include <linux/context_tracking.h>
 #include <linux/init.h>
 #include <linux/interrupt.h>
 #include <linux/kernel.h>
@@ -1361,7 +1362,7 @@ static void partition_xcpus_del(int old_prs, struct cpuset *parent,
  */
 static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
 {
-	int ret;
+	int cpu, ret;
 	struct cpumask *icpus = isolated_cpus;
 	unsigned long flags = BIT(HK_TYPE_DOMAIN) | BIT(HK_TYPE_KERNEL_NOISE);
 
@@ -1395,6 +1396,26 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
 	ret = housekeeping_exclude_cpumask(icpus, flags);
 	WARN_ON_ONCE((ret < 0) && (ret != -EOPNOTSUPP));
 
+#ifdef CONFIG_NO_HZ_FULL
+	/*
+	 * To properly enable/disable nohz_full dynticks for the affected CPUs,
+	 * the new nohz_full CPUs have to be copied to tick_nohz_full_mask and
+	 * ct_cpu_track_user/ct_cpu_untrack_user() will have to be called
+	 * for those CPUs that have their states changed.
+	 */
+	if (tick_nohz_full_enabled()) {
+		tick_nohz_full_update_cpus(icpus);
+		for_each_cpu(cpu, isolcpus_update_state.cpus) {
+			if (cpumask_test_cpu(cpu, icpus))
+				ct_cpu_track_user(cpu);
+			else
+				ct_cpu_untrack_user(cpu);
+		}
+	} else {
+		pr_warn_once("Full dynticks cannot be enabled without the nohz_full kernel boot parameter!\n");
+	}
+#endif
+
 	if (icpus != isolated_cpus)
 		kfree(icpus);
 	return ret;
kernel/context_tracking.c +14 -3
index 734354bbfdbb..ed5653a3d6f7 100644
@@ -431,7 +431,7 @@ static __always_inline void ct_kernel_enter(bool user, int offset) { }
 #define CREATE_TRACE_POINTS
 #include <trace/events/context_tracking.h>
 
-DEFINE_STATIC_KEY_FALSE_RO(context_tracking_key);
+DEFINE_STATIC_KEY_FALSE(context_tracking_key);
 EXPORT_SYMBOL_GPL(context_tracking_key);
 
 static noinstr bool context_tracking_recursion_enter(void)
@@ -694,9 +694,9 @@ void user_exit_callable(void)
 }
 NOKPROBE_SYMBOL(user_exit_callable);
 
-void __init ct_cpu_track_user(int cpu)
+void ct_cpu_track_user(int cpu)
 {
-	static __initdata bool initialized = false;
+	static bool initialized;
 
 	if (cpu == CONTEXT_TRACKING_FORCE_ENABLE) {
 		static_branch_inc(&context_tracking_key);
@@ -720,6 +720,17 @@ void __init ct_cpu_track_user(int cpu)
 	initialized = true;
 }
 
+void ct_cpu_untrack_user(int cpu)
+{
+#ifndef CONFIG_CONTEXT_TRACKING_USER_FORCE
+	if (!per_cpu(context_tracking.active, cpu))
+		return;
+
+	per_cpu(context_tracking.active, cpu) = false;
+	static_branch_dec(&context_tracking_key);
+#endif
+}
+
 #ifdef CONFIG_CONTEXT_TRACKING_USER_FORCE
 void __init context_tracking_init(void)
 {
kernel/time/tick-sched.c +0 -7
index 9204808b7a55..c16250c6a79f 100644
@@ -220,9 +220,6 @@ static void tick_sched_do_timer(struct tick_sched *ts, ktime_t now)
 	tick_cpu = READ_ONCE(tick_do_timer_cpu);
 
 	if (IS_ENABLED(CONFIG_NO_HZ_COMMON) && unlikely(tick_cpu == TICK_DO_TIMER_NONE)) {
-#ifdef CONFIG_NO_HZ_FULL
-		WARN_ON_ONCE(tick_nohz_full_running);
-#endif
 		WRITE_ONCE(tick_do_timer_cpu, cpu);
 		tick_cpu = cpu;
 	}
@@ -1201,10 +1198,6 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 		 */
 		if (tick_cpu == cpu)
 			return false;
-
-		/* Should not happen for nohz-full */
-		if (WARN_ON_ONCE(tick_cpu == TICK_DO_TIMER_NONE))
-			return false;
 	}
 
 	return true;
Signature · 1 line
2.50.0
author maintainer
#14 lore raw

RFC PATCH 14/18 tick: Pass timer tick job to an online HK CPU in tick_cpu_dying()

In reply to Waiman Long

In tick_cpu_dying(), if the dying CPU is the current timekeeper, it has to pass the job over to another CPU. The current code passes it to another online CPU. However, that CPU may not be a timer tick housekeeping CPU. If that happens, another CPU will have to manually take it over again later. Avoid this unnecessary work by directly assigning an online housekeeping CPU.

Use READ_ONCE/WRITE_ONCE() to access tick_do_timer_cpu in case the non-HK CPUs may not be in stop machine in the future.

---
 kernel/time/tick-common.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/time/tick-common.c +11 -4
index 9a3859443c04..6d5ff85281cc 100644
@@ -17,6 +17,7 @@
 #include <linux/profile.h>
 #include <linux/sched.h>
 #include <linux/module.h>
+#include <linux/sched/isolation.h>
 #include <trace/events/power.h>
 
 #include <asm/irq_regs.h>
@@ -394,12 +395,18 @@ int tick_cpu_dying(unsigned int dying_cpu)
 {
 	/*
 	 * If the current CPU is the timekeeper, it's the only one that can
-	 * safely hand over its duty. Also all online CPUs are in stop
-	 * machine, guaranteed not to be idle, therefore there is no
+	 * safely hand over its duty. Also all online housekeeping CPUs are
+	 * in stop machine, guaranteed not to be idle, therefore there is no
 	 * concurrency and it's safe to pick any online successor.
 	 */
-	if (tick_do_timer_cpu == dying_cpu)
-		tick_do_timer_cpu = cpumask_first(cpu_online_mask);
+	if (READ_ONCE(tick_do_timer_cpu) == dying_cpu) {
+		unsigned int new_cpu;
+
+		new_cpu = cpumask_first_and(cpu_online_mask, housekeeping_cpumask(HK_TYPE_TICK));
+		if (WARN_ON_ONCE(new_cpu >= nr_cpu_ids))
+			new_cpu = cpumask_first(cpu_online_mask);
+		WRITE_ONCE(tick_do_timer_cpu, new_cpu);
+	}
 
 	/* Make sure the CPU won't try to retake the timekeeping duty */
 	tick_sched_timer_dying(dying_cpu);
Signature · 1 line
2.50.0
author maintainer
#15 lore raw

RFC PATCH 15/18 cgroup/cpuset: Enable RCU NO-CB CPU offloading of newly isolated CPUs

In reply to Waiman Long

Make use of the provided rcu_nocb_cpu_offload()/rcu_nocb_cpu_deoffload() APIs to enable RCU NO-CB CPU offloading of newly isolated CPUs and deoffloading of de-isolated CPUs.

Also add a new rcu_nocbs_enabled() helper function to determine if RCU NO-CB CPU offloading can be done.

As nohz_full can now be specified without any CPU list, drop the test for cpumask_empty(tick_nohz_full_mask) in rcu_init_nohz().

The RCU NO-CB CPU offloading feature can only used if either "rcs_nocbs" or the "nohz_full" boot command parameters are used so that the proper RCU NO-CB resources are properly initialized at boot time.

---
 include/linux/rcupdate.h |  2 ++
 kernel/cgroup/cpuset.c   | 14 ++++++++++++++
 kernel/rcu/tree_nocb.h   |  7 ++++++-
 3 files changed, 22 insertions(+), 1 deletion(-)
include/linux/rcupdate.h +2 -0
index 120536f4c6eb..642b80a4f071 100644
@@ -140,6 +140,7 @@ void rcu_init_nohz(void);
 int rcu_nocb_cpu_offload(int cpu);
 int rcu_nocb_cpu_deoffload(int cpu);
 void rcu_nocb_flush_deferred_wakeup(void);
+bool rcu_nocbs_enabled(void);
 
 #define RCU_NOCB_LOCKDEP_WARN(c, s) RCU_LOCKDEP_WARN(c, s)
 
@@ -149,6 +150,7 @@ static inline void rcu_init_nohz(void) { }
 static inline int rcu_nocb_cpu_offload(int cpu) { return -EINVAL; }
 static inline int rcu_nocb_cpu_deoffload(int cpu) { return 0; }
 static inline void rcu_nocb_flush_deferred_wakeup(void) { }
+static inline bool rcu_nocbs_enabled(void) { return false; }
 
 #define RCU_NOCB_LOCKDEP_WARN(c, s)
 
kernel/cgroup/cpuset.c +14 -0
index 45c82c18bec4..de9cb92a0fc7 100644
@@ -1416,6 +1416,20 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
 	}
 #endif
 
+	if (rcu_nocbs_enabled()) {
+		/*
+		 * Enable RCU NO-CB CPU offloading/deoffloading for the affected CPUs
+		 */
+		for_each_cpu(cpu, isolcpus_update_state.cpus) {
+			if (cpumask_test_cpu(cpu, icpus))
+				ret = rcu_nocb_cpu_offload(cpu);
+			else
+				ret = rcu_nocb_cpu_deoffload(cpu);
+			if (WARN_ON_ONCE(ret))
+				break;
+		}
+	}
+
 	if (icpus != isolated_cpus)
 		kfree(icpus);
 	return ret;
kernel/rcu/tree_nocb.h +6 -1
index e6cd56603cad..4d49a745b871 100644
@@ -1293,7 +1293,7 @@ void __init rcu_init_nohz(void)
 	struct shrinker * __maybe_unused lazy_rcu_shrinker;
 
 #if defined(CONFIG_NO_HZ_FULL)
-	if (tick_nohz_full_running && !cpumask_empty(tick_nohz_full_mask))
+	if (tick_nohz_full_running)
 		cpumask = tick_nohz_full_mask;
 #endif
 
@@ -1365,6 +1365,11 @@ static void __init rcu_boot_init_nocb_percpu_data(struct rcu_data *rdp)
 	mutex_init(&rdp->nocb_gp_kthread_mutex);
 }
 
+bool rcu_nocbs_enabled(void)
+{
+	return !!rcu_state.nocb_is_setup;
+}
+
 /*
  * If the specified CPU is a no-CBs CPU that does not already have its
  * rcuo CB kthread, spawn it.  Additionally, if the rcuo GP kthread
Signature · 1 line
2.50.0
author maintainer
#16 lore raw

RFC PATCH 16/18 cgroup/cpuset: Don't set have_boot_nohz_full without any boot time nohz_full CPU

In reply to Waiman Long

As HK_TYPE_KERNEL_NOISE bit can now be set without any nohz_full CPU specified at boot time, don't set have_boot_nohz_full in this case.

---
 kernel/cgroup/cpuset.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/cgroup/cpuset.c +6 -1
index de9cb92a0fc7..489708f4e096 100644
@@ -3871,7 +3871,12 @@ int __init cpuset_init(void)
 
 	BUG_ON(!alloc_cpumask_var(&cpus_attach, GFP_KERNEL));
 
-	have_boot_nohz_full = housekeeping_enabled(HK_TYPE_KERNEL_NOISE);
+	/*
+	 * HK_TYPE_KERNEL_NOISE bit can be set without any nohz_full CPU
+	 */
+	have_boot_nohz_full = housekeeping_enabled(HK_TYPE_KERNEL_NOISE) &&
+			      !cpumask_equal(cpu_possible_mask,
+					     housekeeping_cpumask(HK_TYPE_KERNEL_NOISE));
 	have_boot_isolcpus  = housekeeping_enabled(HK_TYPE_DOMAIN);
 	if (have_boot_nohz_full) {
 		BUG_ON(!alloc_cpumask_var(&boot_nohz_full_hk_cpus, GFP_KERNEL));
Signature · 1 line
2.50.0
author maintainer
#17 lore raw

RFC PATCH 17/18 cgroup/cpuset: Documentation updates & don't use CPU 0 for isolated partition

In reply to Waiman Long

As CPU hotplug is now used to improve CPU isolation of CPUs in isolated partitions. The boot CPU (typically CPU 0) cannot be put offline impacting the amount of CPU isolation available. Now we have to advise users that the boot CPU should never be used for isolated partitions. A warning will be printed when boot CPU is used and the cgroup-v2.rst is updated accordingly. The test_cpuset_prs.sh selftest is also updated to remove CPU 0 when forming isolated partitions.

Also update the cgroup-v2.rst file to show the need to specify the "nohz_full" kernel boot parameter to enable better nohz_full behavior for the CPUs in isolated partitions as well as the latency spike issue with using CPU hotplug.

---
 Documentation/admin-guide/cgroup-v2.rst       | 33 +++++++++++++++----
 .../selftests/cgroup/test_cpuset_prs.sh       |  8 ++---
 2 files changed, 31 insertions(+), 10 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +27 -6
index d9d3cc7df348..26213383b34b 100644
@@ -2556,11 +2556,12 @@ Cpuset Interface Files
 
 	It accepts only the following input values when written to.
 
-	  ==========	=====================================
+	  ==========	===============================================
 	  "member"	Non-root member of a partition
 	  "root"	Partition root
-	  "isolated"	Partition root without load balancing
-	  ==========	=====================================
+	  "isolated"	Partition root without load balancing and other
+		        OS noises
+	  ==========	===============================================
 
 	A cpuset partition is a collection of cpuset-enabled cgroups with
 	a partition root at the top of the hierarchy and its descendants
@@ -2593,9 +2594,29 @@ Cpuset Interface Files
 
 	When set to "isolated", the CPUs in that partition will be in
 	an isolated state without any load balancing from the scheduler
-	and excluded from the unbound workqueues.  Tasks placed in such
-	a partition with multiple CPUs should be carefully distributed
-	and bound to each of the individual CPUs for optimal performance.
+	and excluded from the unbound workqueues as well as without
+	other OS noises.  Tasks placed in such a partition with multiple
+	CPUs should be carefully distributed and bound to each of the
+	individual CPUs for optimal performance.
+
+	As CPU hotplug, if supported, is used to improve the degree of
+	CPU isolation close to the "nohz_full" kernel boot parameter.
+	The boot CPU (typically CPU 0) cannot be brought offline, so the
+	boot CPU should not be used for forming isolated partitions.
+	The "nohz_full" kernel boot parameter needs to be present to
+	enable full dynticks support and RCU no-callback CPU mode for
+	CPUs in isolated partitions even if the optional cpu list
+	isn't provided.  Without that, adding the "rcu_nocbs" boot
+	kernel parameter without the cpu list can be used to enable
+	RCU no-callback CPU mode without full dynticks.
+
+	Using CPU hotplug for creating or destroying an isolated
+	partition can cause latency spike in applications running
+	in other isolated partitions.  A reserved list of CPUs can
+	optionally be put in the "nohz_full" kernel boot parameter to
+	alleviate this problem.  When these reserved CPUs are used for
+	isolated partitions, CPU hotplug won't need to be invoked and
+	so there won't be latency spike in other isolated partitions.
 
 	A partition root ("root" or "isolated") can be in one of the
 	two possible states - valid or invalid.  An invalid partition
tools/testing/selftests/cgroup/test_cpuset_prs.sh +4 -4
index a17256d9f88a..f61369be8bf6 100755
@@ -318,8 +318,8 @@ TEST_MATRIX=(
 	# Invalid to valid local partition direct transition tests
 	" C1-3:S+:P2 X4:P2  .      .      .      .      .      .     0 A1:1-3|XA1:1-3|A2:1-3:XA2: A1:P2|A2:P-2 1-3"
 	" C1-3:S+:P2 X4:P2  .      .      .    X3:P2    .      .     0 A1:1-2|XA1:1-3|A2:3:XA2:3 A1:P2|A2:P2 1-3"
-	"  C0-3:P2   .      .    C4-6   C0-4     .      .      .     0 A1:0-4|B1:4-6 A1:P-2|B1:P0"
-	"  C0-3:P2   .      .    C4-6 C0-4:C0-3  .      .      .     0 A1:0-3|B1:4-6 A1:P2|B1:P0 0-3"
+	"  C1-3:P2   .      .    C4-6   C1-4     .      .      .     0 A1:1-4|B1:4-6 A1:P-2|B1:P0"
+	"  C1-3:P2   .      .    C4-6 C1-4:C1-3  .      .      .     0 A1:1-3|B1:4-6 A1:P2|B1:P0 1-3"
 
 	# Local partition invalidation tests
 	" C0-3:X1-3:S+:P2 C1-3:X2-3:S+:P2 C2-3:X3:P2 \
@@ -329,8 +329,8 @@ TEST_MATRIX=(
 	" C0-3:X1-3:S+:P2 C1-3:X2-3:S+:P2 C2-3:X3:P2 \
 				   .      .    C4:X     .      .     0 A1:1-3|A2:1-3|A3:2-3|XA2:|XA3: A1:P2|A2:P-2|A3:P-2 1-3"
 	# Local partition CPU change tests
-	" C0-5:S+:P2 C4-5:S+:P1 .  .      .    C3-5     .      .     0 A1:0-2|A2:3-5 A1:P2|A2:P1 0-2"
-	" C0-5:S+:P2 C4-5:S+:P1 .  .    C1-5     .      .      .     0 A1:1-3|A2:4-5 A1:P2|A2:P1 1-3"
+	" C1-5:S+:P2 C4-5:S+:P1 .  .      .    C3-5     .      .     0 A1:1-2|A2:3-5 A1:P2|A2:P1 1-2"
+	" C1-5:S+:P2 C4-5:S+:P1 .  .    C2-5     .      .      .     0 A1:2-3|A2:4-5 A1:P2|A2:P1 2-3"
 
 	# cpus_allowed/exclusive_cpus update tests
 	" C0-3:X2-3:S+ C1-3:X2-3:S+ C2-3:X2-3 \
Signature · 1 line
2.50.0
author maintainer
#18 lore raw

RFC PATCH 18/18 cgroup/cpuset: Add pr_debug() statements for cpuhp_offline_cb() call

In reply to Waiman Long

Add some pr_debug() statements to actions performed related to the cpuhp_offline_cb() call to aid debugging. Since rcu_nocb_cpu_offload() and rcu_nocb_cpu_deoffload() will print out some info text, there is no need to add pr_debug() statements for them.

Also update test_cpuset_prs.sh test script to enable printing of dynamic debug messages for the kernel/cgroup/cpuset.c file when loglevel is 7 (debug).

---
 kernel/cgroup/cpuset.c                         | 18 +++++++++++++-----
 .../selftests/cgroup/test_cpuset_prs.sh        |  7 +++++++
 2 files changed, 20 insertions(+), 5 deletions(-)
kernel/cgroup/cpuset.c +13 -5
index 489708f4e096..30632e4b5899 100644
@@ -21,6 +21,7 @@
  *  License.  See the file COPYING in the main directory of the Linux
  *  distribution for more details.
  */
+#define pr_fmt(fmt) "cpuset: " fmt
 #include "cpuset-internal.h"
 
 #include <linux/context_tracking.h>
@@ -1406,10 +1407,13 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
 	if (tick_nohz_full_enabled()) {
 		tick_nohz_full_update_cpus(icpus);
 		for_each_cpu(cpu, isolcpus_update_state.cpus) {
-			if (cpumask_test_cpu(cpu, icpus))
+			if (cpumask_test_cpu(cpu, icpus)) {
+				pr_debug("Add CPU %d to nohz_full\n", cpu);
 				ct_cpu_track_user(cpu);
-			else
+			} else {
+				pr_debug("Remove CPU %d from nohz_full\n", cpu);
 				ct_cpu_untrack_user(cpu);
+			}
 		}
 	} else {
 		pr_warn_once("Full dynticks cannot be enabled without the nohz_full kernel boot parameter!\n");
@@ -1425,6 +1429,7 @@ static int do_housekeeping_exclude_cpumask(void *arg __maybe_unused)
 				ret = rcu_nocb_cpu_offload(cpu);
 			else
 				ret = rcu_nocb_cpu_deoffload(cpu);
+
 			if (WARN_ON_ONCE(ret))
 				break;
 		}
@@ -1468,11 +1473,14 @@ static void update_isolation_cpumasks(void)
 	 * Without any change in the set of nohz_full CPUs, we don't really
 	 * need to use CPU hotplug for making change in HK cpumasks.
 	 */
-	if (cpumask_empty(isolcpus_update_state.cpus))
+	if (cpumask_empty(isolcpus_update_state.cpus)) {
 		ret = do_housekeeping_exclude_cpumask(NULL);
-	else
+	} else {
+		pr_debug("cpuhp_offline_cb() called for CPUs %*pbl\n",
+			 cpumask_pr_args(isolcpus_update_state.cpus));
 		ret = cpuhp_offline_cb(isolcpus_update_state.cpus,
 				       do_housekeeping_exclude_cpumask, NULL);
+	}
 	/*
 	 * A errno value of -EPERM may be returned from cpuhp_offline_cb() if
 	 * any one of the CPUs in isolcpus_update_state.cpus can't be brought
@@ -1481,7 +1489,7 @@ static void update_isolation_cpumasks(void)
 	 * isolated partition.
 	 */
 	if (ret == -EPERM)
-		pr_warn_once("cpuset: The boot CPU shouldn't be used for isolated partition\n");
+		pr_warn_once("The boot CPU shouldn't be used for isolated partition\n");
 	else
 		WARN_ON_ONCE(ret < 0);
 
tools/testing/selftests/cgroup/test_cpuset_prs.sh +7 -0
index f61369be8bf6..43a12690775e 100755
@@ -67,6 +67,13 @@ then
 	echo Y > /sys/kernel/debug/sched/verbose
 fi
 
+# Enable dynamic debug messages for cpuset only
+DYN_DEBUG=/sys/kernel/debug/dynamic_debug/control
+[[ -f $DYN_DEBUG ]] && {
+	echo "-p" > $DYN_DEBUG
+	echo "file kernel/cgroup/cpuset.c +p" > $DYN_DEBUG
+}
+
 cd $CGROUP2
 echo +cpuset > cgroup.subtree_control
 
Signature · 1 line
2.50.0
#19 lore raw
In reply to Waiman Long
Quoted 13 lines · Waiman Long

The "nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.

There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the nohz_full and rcu_nocbs boot kernel parameters.

Quoted 13 lines

This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the nohz_full boot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthread

Thanks for working on that, I'm about to leave for 2 weeks vacation so I won't have the time to check this until I'm back.

However this series is highly conflicting with mine (cpuset/isolation: Honour kthreads preferred affinity). Your patchset even redoes things I'm doing (housekeeping cpumask update, RCU synchronization, HK_TYPE_DOMAIN to include cpusets, etc...)

I have a v2 that is almost ready to post.

Wouldn't it be better to wait for it and its infrastructure changes before proceeding with nohz_full?

Thanks.

Signature · 2 lines
Frederic Weisbecker
SUSE Labs
#20 lore raw
Quoted 35 lines · Frederic Weisbecker
Quoted 13 lines · Waiman Long

The "nohz_full" and "rcu_nocbs" boot command parameters can be used to remove a lot of kernel overhead on a specific set of isolated CPUs which can be used to run some latency/bandwidth sensitive workloads with as little kernel disturbance/noise as possible. The problem with this mode of operation is the fact that it is a static configuration which cannot be changed after boot to adjust for changes in application loading.

There is always a desire to enable runtime modification of the number of isolated CPUs that can be dedicated to this type of demanding workloads. This patchset is an attempt to do just that with an amount of CPU isolation close to what can be done with the nohz_full and rcu_nocbs boot kernel parameters.

Quoted 13 lines

This patch series provides the ability to change the set of housekeeping CPUs at run time via the cpuset isolated partition functionality. Currently, the cpuset isolated partition is able to disable scheduler load balancing and the CPU affinity of the unbound workqueue to avoid the isolated CPUs. This patch series will extend that with other kernel noises associated with the nohz_full boot command line parameter which has the following sub-categories:
- tick
- timer
- RCU
- MISC
- WQ
- kthread

Thanks for working on that, I'm about to leave for 2 weeks vacation so I won't have the time to check this until I'm back.

However this series is highly conflicting with mine (cpuset/isolation: Honour kthreads preferred affinity). Your patchset even redoes things I'm doing (housekeeping cpumask update, RCU synchronization, HK_TYPE_DOMAIN to include cpusets, etc...)

Quoted 4 lines

I have a v2 that is almost ready to post.

Wouldn't it be better to wait for it and its infrastructure changes before proceeding with nohz_full?

Sure. I am just posting this RFC patch series to show my current idea that I have. I will wait for your v2 and integrate on top.

Looking forward to your upcoming v2 patch.

Cheers,
Longman