Skip to contentSkip to conversation

cgroup/cpuset: More fix patches

PATCH Superseded Waiman Long 3 participants 8 messages Nov 11 – Nov 12

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Waiman Long

author maintainer
#0 lore raw

v3 cover cgroup/cpuset: More fix patches

[v3]
- Drop patch 1, 4 & 5 of v2 as they are merged.
- Update the new patch 1 as suggested by Chen Ridong
- Add a new fix patch 3.

[v2]
- Reorder the series by putting the original patch 4 before patch 2.
- Update the Fixes tags in patch 1 and 5 as suggested by sashiko.
- Update the new patch 3 to remove obsolete WARN_ON_ONCE() and update comment as reported by sashiko.
- Minor fix to the test case in patch 5 as reported by sashiko.

This patch series contains an additional set of cpuset fix patches.

Waiman Long (3):
cgroup/cpuset: Correctly set cpus_updated in
cpuset_hotplug_update_tasks()
cgroup/cpuset: Fix incorrect partition invalidation logic in
cpuset_hotplug_update_tasks()
cgroup/cpuset: Check remote partition underneath before making an
invalid partition valid

 kernel/cgroup/cpuset.c                        | 70 ++++++++++---------
 .../selftests/cgroup/test_cpuset_prs.sh       |  4 ++
 2 files changed, 42 insertions(+), 32 deletions(-)
Signature · 1 line
2.56.0
author maintainer
#1 lore raw

PATCH v3 1/3 cgroup/cpuset: Correctly set cpus_updated in cpuset_hotplug_update_tasks()

When cpuset_hotplug_update_tasks() is called with the conditions that remote_partition_disable() has to be called, the cs->effective_cpus will be updated such that it matches what is computed in new_cpus. In such case, cpus_updated can be incorrectly set to false leading to stale CPU afinity remained in the tasks not matching the new cpuset value. Fix that by forcing cpus_updated to true after calling remote_partition_disable().

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
 kernel/cgroup/cpuset.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/cgroup/cpuset.c +7 -2
index 8a8a424a389f..862864b44960 100644
@@ -4031,7 +4031,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 {
 	static cpumask_t new_cpus;
 	static nodemask_t new_mems;
-	bool cpus_updated;
+	bool cpus_updated = false;
 	bool mems_updated;
 	int partcmd = -1;
 	struct cpuset *parent;
@@ -4044,12 +4044,17 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	if (!tmp || !cs->partition_root_state)
 		goto update_tasks;
 
+	/*
+	 * cpus_updated should be set to true when remote_partition_disable()
+	 * is called as cs->effective_cpus will be modified.
+	 */
 	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
 				       (cpumask_empty(&new_cpus) &&
 					partition_is_populated(cs, NULL)))) {
 		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
 		remote_partition_disable(cs, tmp);
 		compute_effective_cpumask(&new_cpus, cs, parent);
+		cpus_updated = true;
 	}
 
 	/*
@@ -4082,7 +4087,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	}
 
 update_tasks:
-	cpus_updated = !cpumask_equal(&new_cpus, cs->effective_cpus);
+	cpus_updated |= !cpumask_equal(&new_cpus, cs->effective_cpus);
 	mems_updated = !nodes_equal(new_mems, cs->effective_mems);
 	if (!cpus_updated && !mems_updated)
 		goto unlock;	/* Hotplug doesn't affect this cpuset */
Signature · 1 line
2.56.0
reviewer Reviewed-by on v3 1/3
#2 lore raw

Re: [PATCH-next v3 1/3] cgroup/cpuset: Correctly set cpus_updated in cpuset_hotplug_update_tasks()

Quoted 47 lines · Waiman Long

When cpuset_hotplug_update_tasks() is called with the conditions that remote_partition_disable() has to be called, the cs->effective_cpus will be updated such that it matches what is computed in new_cpus. In such case, cpus_updated can be incorrectly set to false leading to stale CPU afinity remained in the tasks not matching the new cpuset value. Fix that by forcing cpus_updated to true after calling remote_partition_disable().

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
  kernel/cgroup/cpuset.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
kernel/cgroup/cpuset.c +6 -1
index 8a8a424a389f..862864b44960 100644
@@ -4031,7 +4031,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
  {
  	static cpumask_t new_cpus;
  	static nodemask_t new_mems;
-	bool cpus_updated;
+	bool cpus_updated = false;
  	bool mems_updated;
  	int partcmd = -1;
  	struct cpuset *parent;
@@ -4044,12 +4044,17 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
  	if (!tmp || !cs->partition_root_state)
  		goto update_tasks;
 
+	/*
+	 * cpus_updated should be set to true when remote_partition_disable()
+	 * is called as cs->effective_cpus will be modified.
+	 */
  	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
  				       (cpumask_empty(&new_cpus) &&
  					partition_is_populated(cs, NULL)))) {
  		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
  		remote_partition_disable(cs, tmp);
  		compute_effective_cpumask(&new_cpus, cs, parent);
+		cpus_updated = true;
  	}
 
  	/*

kernel/cgroup/cpuset.c static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) · line 4093

Quoted 8 lines
kernel/cgroup/cpuset.c +1 -1
@@ -4082,7 +4087,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
  	}
 
  update_tasks:
-	cpus_updated = !cpumask_equal(&new_cpus, cs->effective_cpus);
+	cpus_updated |= !cpumask_equal(&new_cpus, cs->effective_cpus);
  	mems_updated = !nodes_equal(new_mems, cs->effective_mems);
  	if (!cpus_updated && !mems_updated)
  		goto unlock;	/* Hotplug doesn't affect this cpuset */

Thanks.

Signature · 2 lines
Best regards
Ridong
Reviewed-by Tested-by on v3 1/3
#3 lore raw

Re: [PATCH-next v3 1/3] cgroup/cpuset: Correctly set cpus_updated in cpuset_hotplug_update_tasks()

In reply to Waiman Long
Quoted 47 lines · Waiman Long

When cpuset_hotplug_update_tasks() is called with the conditions that remote_partition_disable() has to be called, the cs->effective_cpus will be updated such that it matches what is computed in new_cpus. In such case, cpus_updated can be incorrectly set to false leading to stale CPU afinity remained in the tasks not matching the new cpuset value. Fix that by forcing cpus_updated to true after calling remote_partition_disable().

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
 kernel/cgroup/cpuset.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
kernel/cgroup/cpuset.c +6 -1
index 8a8a424a389f..862864b44960 100644
@@ -4031,7 +4031,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 {
 	static cpumask_t new_cpus;
 	static nodemask_t new_mems;
-	bool cpus_updated;
+	bool cpus_updated = false;
 	bool mems_updated;
 	int partcmd = -1;
 	struct cpuset *parent;
@@ -4044,12 +4044,17 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	if (!tmp || !cs->partition_root_state)
 		goto update_tasks;
 
+	/*
+	 * cpus_updated should be set to true when remote_partition_disable()
+	 * is called as cs->effective_cpus will be modified.
+	 */
 	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
 				       (cpumask_empty(&new_cpus) &&
 					partition_is_populated(cs, NULL)))) {
 		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
 		remote_partition_disable(cs, tmp);
 		compute_effective_cpumask(&new_cpus, cs, parent);
+		cpus_updated = true;
 	}
 
 	/*

kernel/cgroup/cpuset.c static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) · line 4093

Quoted 8 lines
kernel/cgroup/cpuset.c +1 -1
@@ -4082,7 +4087,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	}
 
 update_tasks:
-	cpus_updated = !cpumask_equal(&new_cpus, cs->effective_cpus);
+	cpus_updated |= !cpumask_equal(&new_cpus, cs->effective_cpus);
 	mems_updated = !nodes_equal(new_mems, cs->effective_mems);
 	if (!cpus_updated && !mems_updated)
 		goto unlock;	/* Hotplug doesn't affect this cpuset */

Thanks,
Guopeng

author maintainer
#4 lore raw

PATCH v3 2/3 cgroup/cpuset: Fix incorrect partition invalidation logic in cpuset_hotplug_update_tasks()

In reply to Waiman Long

When most of the active CPUs of a system are allocated to sub-partitions below the top_cpuset partition, it is possible that a CPU offline operation may remove the last active CPU in the top_cpuset partition. When that really happens, cpuset_handle_hotplug() will clear the subpartitions_cpus to make sure active CPUs are present in the top partition and let the other sub-partitions get invalidated and then competes for their exclusive CPUs again.

In cpuset_hotplug_update_tasks(), the current code use "cpumask_empty(subpartitions_cpus)" to determine if that partition should be invalidated. That is fine if there is only one sub-partition. As reported by sashiko that when there are more than one sub-partitions and the first sub-partition will get invalidated and then it will try to acquire the exclusive CPUs and become valid again. When this happens, the subpartitions_cpus will no longer be empty. The remaining sub-partitions will assume their exclusive CPUs are safe and proceed as normal. As a result, active CPUs owned by those sub-partitions remain in both top_cpuset.effective_cpus and the sub-partitions' effective_cpus while missing from subpartitions_cpus, breaking partition CPU exclusivity and triggering WARN_ON_ONCE() in generate_sched_domains() and remote_partition_disable().

Fix that by checking if the sub-partitions' effective_xcpus are subset of subpartitions_cpus to determine if they should be invalidated. Also remove the obsolete WARN_ON_ONCE() call in remote_partition_disable().

Fixes 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2")
---
 kernel/cgroup/cpuset.c | 19 +++++--------------
 1 file changed, 5 insertions(+), 14 deletions(-)
kernel/cgroup/cpuset.c +5 -14
index 862864b44960..8b684361c94e 100644
@@ -1766,15 +1766,6 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs,
 static void remote_partition_disable(struct cpuset *cs, struct tmpmasks *tmp)
 {
 	WARN_ON_ONCE(!is_remote_partition(cs));
-	/*
-	 * When a CPU is offlined, top_cpuset may end up with no available CPUs,
-	 * which should clear subpartitions_cpus. We should not emit a warning for this
-	 * scenario: the hierarchy is updated from top to bottom, so subpartitions_cpus
-	 * may already be cleared when disabling the partition.
-	 */
-	WARN_ON_ONCE(!cpumask_subset(cs->effective_xcpus, subpartitions_cpus) &&
-		     !cpumask_empty(subpartitions_cpus));
-
 	spin_lock_irq(&callback_lock);
 	cs->remote_partition = false;
 	partition_xcpus_del(cs->partition_root_state, NULL, cs->effective_xcpus);
@@ -4048,9 +4039,9 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	 * cpus_updated should be set to true when remote_partition_disable()
 	 * is called as cs->effective_cpus will be modified.
 	 */
-	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
-				       (cpumask_empty(&new_cpus) &&
-					partition_is_populated(cs, NULL)))) {
+	if (is_remote_partition(cs) &&
+	    (!cpumask_subset(cs->effective_xcpus, subpartitions_cpus) ||
+	     (cpumask_empty(&new_cpus) && partition_is_populated(cs, NULL)))) {
 		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
 		remote_partition_disable(cs, tmp);
 		compute_effective_cpumask(&new_cpus, cs, parent);
@@ -4063,12 +4054,12 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	 * 1) empty effective cpus but not valid empty partition.
 	 * 2) parent is invalid or doesn't grant any cpus to child
 	 *    partitions.
-	 * 3) subpartitions_cpus is empty.
+	 * 3) effective_xcpus isn't a subset of subpartitions_cpus.
 	 */
 	if (is_local_partition(cs) &&
 	    (!is_partition_valid(parent) ||
 	     tasks_nocpu_error(parent, cs, &new_cpus) ||
-	     cpumask_empty(subpartitions_cpus)))
+	     !cpumask_subset(cs->effective_xcpus, subpartitions_cpus)))
 		partcmd = partcmd_invalidate;
 	/*
 	 * On the other hand, an invalid partition root may be transitioned
Signature · 1 line
2.56.0
author maintainer
#5 lore raw

PATCH v3 3/3 cgroup/cpuset: Check remote partition underneath before making an invalid partition valid

In reply to Waiman Long

When re-evaluating an invalid local partition via partcmd_update, update_parent_effective_cpumask() does not check whether a descendant remote partition is using its exclusive CPUs, promoting the cpuset to a valid partition root over the remote partition. This condition should only be possible if parent is the top_cpuset and exclusive_cpus isn't empty.

This leaves an active remote partition underneath a valid local partition root with overlapping exclusive CPUs, breaking scheduler domain partitioning and triggering warnings in compute_partition_effective_cpumask() and generate_sched_domains().

Fix that by checking the effective_xcpus against subpartitions_cpus when an underneath remote partition is possible to make sure that all the exclusive CPUs are available before allowing the invalid partition to be switched back to valid.

Also make sure that effective_xcpus is non-empty before swtiching an invalid partition back to valid.

Add a test case in test_cpuset_prs.sh to verify that an invalid partition with a remote partition underneath cannot be switched back to valid with a hotplug event.

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
 kernel/cgroup/cpuset.c                        | 42 ++++++++++++-------
 .../selftests/cgroup/test_cpuset_prs.sh       |  4 ++
 2 files changed, 30 insertions(+), 16 deletions(-)
kernel/cgroup/cpuset.c +26 -16
index 8b684361c94e..af5fd543fe88 100644
@@ -2034,11 +2034,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		 * tasks_nocpu_error() with this very mask, and a failing
 		 * write is routed to partcmd_invalidate instead.
 		 */
-		if (is_partition_invalid(cs)) {
-			adding = false;
-			deleting = cpumask_and(tmp->delmask,
-					newmask, parent->effective_xcpus);
-		} else {
+		if (is_partition_valid(cs)) {
 			cpumask_andnot(tmp->addmask, cs->effective_xcpus, newmask);
 			adding = cpumask_and(tmp->addmask, tmp->addmask,
 					     parent->effective_xcpus);
@@ -2046,6 +2042,17 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 			cpumask_andnot(tmp->delmask, newmask, cs->effective_xcpus);
 			deleting = cpumask_and(tmp->delmask, tmp->delmask,
 					       parent->effective_xcpus);
+		} else {
+			deleting = cpumask_and(tmp->delmask,
+					newmask, parent->effective_xcpus);
+
+			/* Check if a remote partition may be underneath */
+			if ((parent == &top_cpuset) &&
+			    !cpumask_empty(cs->exclusive_cpus) &&
+			    cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
+				deleting = false;
+			}
 		}
 
 		/*
@@ -2126,34 +2133,37 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		} else if (is_partition_invalid(cs)) {
 			struct cgroup_subsys_state *css;
 			struct cpuset *child;
-			bool exclusive = true;
 
 			/*
 			 * Convert invalid partition to valid has to pass the
-			 * cpu exclusivity test when cpuset.cpus.exclusive is
-			 * empty as well as the housekeeping check.
+			 * following tests:
+			 * 1) housekeeping CPU conflict check
+			 * 2) cpu exclusivity test when cpuset.cpus.exclusive is
+			 *    empty.
+			 * 3) if parent is top_cpuset, none of the effective_xcpus
+			 *    (tmp->delmask) are in subpartitions_cpus, if not a
+			 *    remote partition may be underneath.
 			 */
 			if (prstate_housekeeping_conflict(cs->partition_root_state,
 							  parent_prs, tmp->delmask, NULL)) {
 				part_error = PERR_HKEEPING;
-				goto write_error;
-			}
-			if (cpumask_empty(cs->exclusive_cpus)) {
+			} else if (cpumask_empty(cs->exclusive_cpus)) {
 				rcu_read_lock();
 				cpuset_for_each_child(child, css, parent) {
 					if (child == cs)
 						continue;
 					if (!cpusets_are_exclusive(cs, child)) {
-						exclusive = false;
+						part_error = PERR_NOTEXCL;
 						break;
 					}
 				}
 				rcu_read_unlock();
+			} else if ((parent == &top_cpuset) &&
+				   cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
 			}
-			if (exclusive)
+			if (!part_error)
 				deleting = !cpumask_empty(tmp->delmask);
-			else
-				part_error = PERR_NOTEXCL;
 		}
 	}
 
@@ -2174,7 +2184,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 			break;
 		case PRS_INVALID_ROOT:
 		case PRS_INVALID_ISOLATED:
-			if (!part_error)
+			if (!part_error && deleting)
 				new_prs = -old_prs;
 			break;
 		}
tools/testing/selftests/cgroup/test_cpuset_prs.sh +4 -0
index 1e9e6b4c6f7b..3d873804abc2 100755
@@ -563,6 +563,10 @@ TEST_MATRIX=(
 	" C1-6:P1 C3-6:P2 C5-6:P2  .       .    P1      .      .     0 A1:1-2|A2:3-4|A3:5-6 \
 								       A1:P1|A2:P1|A3:P2 5-6"
 
+	# A hotplug event will not incorrectly activate an invalid partition with
+	# a remote partition underneath.
+	"  CX1-4  C3-4:P1   .      .      P1   O1=0     .      .     0 A1:2|A2:3-4 A1:P-1|A2:P1"
+
 	#
 	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
 	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------
Signature · 1 line
2.56.0
#6 lore raw

Re: [PATCH-next v3 3/3] cgroup/cpuset: Check remote partition underneath before making an invalid partition valid

Quoted 97 lines · Waiman Long

When re-evaluating an invalid local partition via partcmd_update, update_parent_effective_cpumask() does not check whether a descendant remote partition is using its exclusive CPUs, promoting the cpuset to a valid partition root over the remote partition. This condition should only be possible if parent is the top_cpuset and exclusive_cpus isn't empty.

This leaves an active remote partition underneath a valid local partition root with overlapping exclusive CPUs, breaking scheduler domain partitioning and triggering warnings in compute_partition_effective_cpumask() and generate_sched_domains().

Fix that by checking the effective_xcpus against subpartitions_cpus when an underneath remote partition is possible to make sure that all the exclusive CPUs are available before allowing the invalid partition to be switched back to valid.

Also make sure that effective_xcpus is non-empty before swtiching an invalid partition back to valid.

Add a test case in test_cpuset_prs.sh to verify that an invalid partition with a remote partition underneath cannot be switched back to valid with a hotplug event.

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
 kernel/cgroup/cpuset.c                        | 42 ++++++++++++-------
 .../selftests/cgroup/test_cpuset_prs.sh       |  4 ++
 2 files changed, 30 insertions(+), 16 deletions(-)
kernel/cgroup/cpuset.c +20 -11
index 8b684361c94e..af5fd543fe88 100644
@@ -2034,11 +2034,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		 * tasks_nocpu_error() with this very mask, and a failing
 		 * write is routed to partcmd_invalidate instead.
 		 */
-		if (is_partition_invalid(cs)) {
-			adding = false;
-			deleting = cpumask_and(tmp->delmask,
-					newmask, parent->effective_xcpus);
-		} else {
+		if (is_partition_valid(cs)) {
 			cpumask_andnot(tmp->addmask, cs->effective_xcpus, newmask);
 			adding = cpumask_and(tmp->addmask, tmp->addmask,
 					     parent->effective_xcpus);
@@ -2046,6 +2042,17 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 			cpumask_andnot(tmp->delmask, newmask, cs->effective_xcpus);
 			deleting = cpumask_and(tmp->delmask, tmp->delmask,
 					       parent->effective_xcpus);
+		} else {
+			deleting = cpumask_and(tmp->delmask,
+					newmask, parent->effective_xcpus);
+
+			/* Check if a remote partition may be underneath */
+			if ((parent == &top_cpuset) &&
+			    !cpumask_empty(cs->exclusive_cpus) &&
+			    cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
+				deleting = false;
+			}
 		}
 
 		/*
@@ -2126,34 +2133,37 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 		} else if (is_partition_invalid(cs)) {
 			struct cgroup_subsys_state *css;
 			struct cpuset *child;
-			bool exclusive = true;
 
 			/*
 			 * Convert invalid partition to valid has to pass the
-			 * cpu exclusivity test when cpuset.cpus.exclusive is
-			 * empty as well as the housekeeping check.
+			 * following tests:
+			 * 1) housekeeping CPU conflict check
+			 * 2) cpu exclusivity test when cpuset.cpus.exclusive is
+			 *    empty.
+			 * 3) if parent is top_cpuset, none of the effective_xcpus
+			 *    (tmp->delmask) are in subpartitions_cpus, if not a
+			 *    remote partition may be underneath.
 			 */
 			if (prstate_housekeeping_conflict(cs->partition_root_state,
 							  parent_prs, tmp->delmask, NULL)) {
 				part_error = PERR_HKEEPING;
-				goto write_error;
-			}
-			if (cpumask_empty(cs->exclusive_cpus)) {
+			} else if (cpumask_empty(cs->exclusive_cpus)) {
 				rcu_read_lock();
 				cpuset_for_each_child(child, css, parent) {
 					if (child == cs)
 						continue;
 					if (!cpusets_are_exclusive(cs, child)) {

kernel/cgroup/cpuset.c static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, · line 2164

Quoted 10 lines
kernel/cgroup/cpuset.c +4 -1
-						exclusive = false;
+						part_error = PERR_NOTEXCL;
 						break;
 					}
 				}
 				rcu_read_unlock();
+			} else if ((parent == &top_cpuset) &&
+				   cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
 			}

Hi,Longman

I think this check can still miss a remote partition during hotplug. When CPU hotplug leaves no active CPU available to the top partition, cpuset_handle_hotplug() clears subpartitions_cpus before walking the descendants. The remote partitions are still valid at that point.

An invalid parent is visited before its remote descendant. Since the mask is empty, this check allows the parent to become valid and add its exclusive CPUs back to subpartitions_cpus. When the remote descendant is visited later, those CPUs are already back in the mask, so it passes the subset check and stays remote. We are then left with a valid remote partition below a valid local partition, and compute_partition_effective_cpumask() warns about the remote child.

Could we check whether there is still a valid remote partition below cs before allowing cs to become valid? subpartitions_cpus may have been cleared by hotplug, so the lack of overlap does not tell us that the remote partition has actually been disabled.

Thanks,
Guopeng

Quoted 32 lines
kernel/cgroup/cpuset.c +2 -4
-			if (exclusive)
+			if (!part_error)
 				deleting = !cpumask_empty(tmp->delmask);
-			else
-				part_error = PERR_NOTEXCL;
 		}
 	}
 
@@ -2174,7 +2184,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
 			break;
 		case PRS_INVALID_ROOT:
 		case PRS_INVALID_ISOLATED:
-			if (!part_error)
+			if (!part_error && deleting)
 				new_prs = -old_prs;
 			break;
 		}
tools/testing/selftests/cgroup/test_cpuset_prs.sh +4 -0
index 1e9e6b4c6f7b..3d873804abc2 100755
@@ -563,6 +563,10 @@ TEST_MATRIX=(
 	" C1-6:P1 C3-6:P2 C5-6:P2  .       .    P1      .      .     0 A1:1-2|A2:3-4|A3:5-6 \
 								       A1:P1|A2:P1|A3:P2 5-6"
 
+	# A hotplug event will not incorrectly activate an invalid partition with
+	# a remote partition underneath.
+	"  CX1-4  C3-4:P1   .      .      P1   O1=0     .      .     0 A1:2|A2:3-4 A1:P-1|A2:P1"
+
 	#
 	#  old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS
 	#  ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ --------
author maintainer
#7 lore raw

Re: [PATCH-next v3 3/3] cgroup/cpuset: Check remote partition underneath before making an invalid partition valid

Quoted 124 lines · Guopeng Zhang
Quoted 97 lines · Waiman Long

When re-evaluating an invalid local partition via partcmd_update, update_parent_effective_cpumask() does not check whether a descendant remote partition is using its exclusive CPUs, promoting the cpuset to a valid partition root over the remote partition. This condition should only be possible if parent is the top_cpuset and exclusive_cpus isn't empty.

This leaves an active remote partition underneath a valid local partition root with overlapping exclusive CPUs, breaking scheduler domain partitioning and triggering warnings in compute_partition_effective_cpumask() and generate_sched_domains().

Fix that by checking the effective_xcpus against subpartitions_cpus when an underneath remote partition is possible to make sure that all the exclusive CPUs are available before allowing the invalid partition to be switched back to valid.

Also make sure that effective_xcpus is non-empty before swtiching an invalid partition back to valid.

Add a test case in test_cpuset_prs.sh to verify that an invalid partition with a remote partition underneath cannot be switched back to valid with a hotplug event.

Fixes 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
---
  kernel/cgroup/cpuset.c                        | 42 ++++++++++++-------
  .../selftests/cgroup/test_cpuset_prs.sh       |  4 ++
  2 files changed, 30 insertions(+), 16 deletions(-)
kernel/cgroup/cpuset.c +20 -11
index 8b684361c94e..af5fd543fe88 100644
@@ -2034,11 +2034,7 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
  		 * tasks_nocpu_error() with this very mask, and a failing
  		 * write is routed to partcmd_invalidate instead.
  		 */
-		if (is_partition_invalid(cs)) {
-			adding = false;
-			deleting = cpumask_and(tmp->delmask,
-					newmask, parent->effective_xcpus);
-		} else {
+		if (is_partition_valid(cs)) {
  			cpumask_andnot(tmp->addmask, cs->effective_xcpus, newmask);
  			adding = cpumask_and(tmp->addmask, tmp->addmask,
  					     parent->effective_xcpus);
@@ -2046,6 +2042,17 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
  			cpumask_andnot(tmp->delmask, newmask, cs->effective_xcpus);
  			deleting = cpumask_and(tmp->delmask, tmp->delmask,
  					       parent->effective_xcpus);
+		} else {
+			deleting = cpumask_and(tmp->delmask,
+					newmask, parent->effective_xcpus);
+
+			/* Check if a remote partition may be underneath */
+			if ((parent == &top_cpuset) &&
+			    !cpumask_empty(cs->exclusive_cpus) &&
+			    cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
+				deleting = false;
+			}
  		}
 
  		/*
@@ -2126,34 +2133,37 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd,
  		} else if (is_partition_invalid(cs)) {
  			struct cgroup_subsys_state *css;
  			struct cpuset *child;
-			bool exclusive = true;
 
  			/*
  			 * Convert invalid partition to valid has to pass the
-			 * cpu exclusivity test when cpuset.cpus.exclusive is
-			 * empty as well as the housekeeping check.
+			 * following tests:
+			 * 1) housekeeping CPU conflict check
+			 * 2) cpu exclusivity test when cpuset.cpus.exclusive is
+			 *    empty.
+			 * 3) if parent is top_cpuset, none of the effective_xcpus
+			 *    (tmp->delmask) are in subpartitions_cpus, if not a
+			 *    remote partition may be underneath.
  			 */
  			if (prstate_housekeeping_conflict(cs->partition_root_state,
  							  parent_prs, tmp->delmask, NULL)) {
  				part_error = PERR_HKEEPING;
-				goto write_error;
-			}
-			if (cpumask_empty(cs->exclusive_cpus)) {
+			} else if (cpumask_empty(cs->exclusive_cpus)) {
  				rcu_read_lock();
  				cpuset_for_each_child(child, css, parent) {
  					if (child == cs)
  						continue;
  					if (!cpusets_are_exclusive(cs, child)) {

kernel/cgroup/cpuset.c static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, · line 2164

Quoted 10 lines
kernel/cgroup/cpuset.c +4 -1
-						exclusive = false;
+						part_error = PERR_NOTEXCL;
  						break;
  					}
  				}
  				rcu_read_unlock();
+			} else if ((parent == &top_cpuset) &&
+				   cpumask_intersects(tmp->delmask, subpartitions_cpus)) {
+				part_error = PERR_REMOTE;
  			}

Hi,Longman

I think this check can still miss a remote partition during hotplug. When CPU hotplug leaves no active CPU available to the top partition, cpuset_handle_hotplug() clears subpartitions_cpus before walking the descendants. The remote partitions are still valid at that point.

An invalid parent is visited before its remote descendant. Since the mask is empty, this check allows the parent to become valid and add its exclusive CPUs back to subpartitions_cpus. When the remote descendant is visited later, those CPUs are already back in the mask, so it passes the subset check and stays remote. We are then left with a valid remote partition below a valid local partition, and compute_partition_effective_cpumask() warns about the remote child.

Quoted 4 lines

Could we check whether there is still a valid remote partition below cs before allowing cs to become valid? subpartitions_cpus may have been cleared by hotplug, so the lack of overlap does not tell us that the remote partition has actually been disabled.

Yes, this is a corner case that I am aware of. Thanks for letting me know about that.

Right now, only invalid local partition has the ability to switch back to valid when the condition is right. Remote partition has no such capability at the moment. So a remote partition should always be invalidated when top_cpuset run out of active CPU. One way to handle this situation is to switch back to chain all remote partition in a linked list. We can then work the remote partition linked list when that happens and determined if we can their effective_xcpus back to subparttitions_cpus or has to disable it before going down to do hierarchical cpuset walk. This will avoid the corner you are talking about. It will be a separate patch. I will update the patch description to add this limitation and a todo item.

Cheers,
Longman