Skip to contentSkip to conversation

sched,topology: Update sched topology atomically

PATCH In review Huang Ying 2 participants 3 messages Apr 21, 2022 – Apr 26, 2022 Scheduler

lore
Share & tools
lore

Huang Ying

author
#0 lore raw

PATCH 1/1 sched,topology: Update sched topology atomically

When Peter Zijlstra reviewed commit 0fb3978b0aac ("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out that sched_domains_numa_distance and sched_domains_numa_masks are made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.

So we update sched_domains_numa_distance and sched_domains_numa_masks and some other related sched topology parameters atomically to address the potential issues.

[1] https://lkml.kernel.org/r/[email protected]

Cc Valentin Schneider <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Srikar Dronamraju <[email protected]>

Changelog:

v2:

- Addressed comments from Valentin Schneider, Thanks!

---
 kernel/sched/fair.c     |  33 +++---
 kernel/sched/sched.h    |   3 +-
 kernel/sched/topology.c | 255 ++++++++++++++++++++++------------------
 3 files changed, 158 insertions(+), 133 deletions(-)
kernel/sched/fair.c +17 -16
index d4bd299d67ab..6204efca2b56 100644
@@ -1294,17 +1294,17 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid,
 					int lim_dist, bool task)
 {
 	unsigned long score = 0;
-	int node, max_dist;
+	int node, max_dist, type;
+
+	sched_fetch_numa_topology(&type, &max_dist);
 
 	/*
 	 * All nodes are directly connected, and the same distance
 	 * from each other. No need for fancy placement algorithms.
 	 */
-	if (sched_numa_topology_type == NUMA_DIRECT)
+	if (type == NUMA_DIRECT)
 		return 0;
 
-	/* sched_max_numa_distance may be changed in parallel. */
-	max_dist = READ_ONCE(sched_max_numa_distance);
 	/*
 	 * This code is called for each node, introducing N^2 complexity,
 	 * which should be ok given the number of nodes rarely exceeds 8.
@@ -1327,7 +1327,7 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid,
 		 * "hoplimit", only nodes closer by than "hoplimit" are part
 		 * of each group. Skip other nodes.
 		 */
-		if (sched_numa_topology_type == NUMA_BACKPLANE && dist >= lim_dist)
+		if (type == NUMA_BACKPLANE && dist >= lim_dist)
 			continue;
 
 		/* Add up the faults from nearby nodes. */
@@ -1344,7 +1344,7 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid,
 		 * The further away a node is, the less the faults count.
 		 * This seems to result in good task placement.
 		 */
-		if (sched_numa_topology_type == NUMA_GLUELESS_MESH) {
+		if (type == NUMA_GLUELESS_MESH) {
 			faults *= (max_dist - dist);
 			faults /= (max_dist - LOCAL_DISTANCE);
 		}
@@ -1971,7 +1971,9 @@ static int task_numa_migrate(struct task_struct *p)
 	long taskimp, groupimp;
 	struct numa_group *ng;
 	struct rq *best_rq;
-	int nid, ret, dist;
+	int nid, ret, dist, type;
+
+	sched_fetch_numa_topology(&type, NULL);
 
 	/*
 	 * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -2026,8 +2028,7 @@ static int task_numa_migrate(struct task_struct *p)
 				continue;
 
 			dist = node_distance(env.src_nid, env.dst_nid);
-			if (sched_numa_topology_type == NUMA_BACKPLANE &&
-						dist != env.dist) {
+			if (type == NUMA_BACKPLANE && dist != env.dist) {
 				taskweight = task_weight(p, env.src_nid, dist);
 				groupweight = group_weight(p, env.src_nid, dist);
 			}
@@ -2262,10 +2263,12 @@ static u64 numa_get_avg_runtime(struct task_struct *p, u64 *period)
 static int preferred_group_nid(struct task_struct *p, int nid)
 {
 	nodemask_t nodes;
-	int dist;
+	int dist, max_dist, type;
+
+	sched_fetch_numa_topology(&type, &max_dist);
 
 	/* Direct connections between all NUMA nodes. */
-	if (sched_numa_topology_type == NUMA_DIRECT)
+	if (type == NUMA_DIRECT)
 		return nid;
 
 	/*
@@ -2273,14 +2276,12 @@ static int preferred_group_nid(struct task_struct *p, int nid)
 	 * scores nodes according to the number of NUMA hinting faults on
 	 * both the node itself, and on nearby nodes.
 	 */
-	if (sched_numa_topology_type == NUMA_GLUELESS_MESH) {
+	if (type == NUMA_GLUELESS_MESH) {
 		unsigned long score, max_score = 0;
 		int node, max_node = nid;
 
-		dist = sched_max_numa_distance;
-
 		for_each_node_state(node, N_CPU) {
-			score = group_weight(p, node, dist);
+			score = group_weight(p, node, max_dist);
 			if (score > max_score) {
 				max_score = score;
 				max_node = node;
@@ -2299,7 +2300,7 @@ static int preferred_group_nid(struct task_struct *p, int nid)
 	 * keep the complexity of the search down.
 	 */
 	nodes = node_states[N_CPU];
-	for (dist = sched_max_numa_distance; dist > LOCAL_DISTANCE; dist--) {
+	for (dist = max_dist; dist > LOCAL_DISTANCE; dist--) {
 		unsigned long max_faults = 0;
 		nodemask_t max_group = NODE_MASK_NONE;
 		int a, b;
kernel/sched/sched.h +1 -2
index 8dccb34eb190..547918e08939 100644
@@ -1638,9 +1638,8 @@ enum numa_topology_type {
 	NUMA_GLUELESS_MESH,
 	NUMA_BACKPLANE,
 };
-extern enum numa_topology_type sched_numa_topology_type;
-extern int sched_max_numa_distance;
 extern bool find_numa_distance(int distance);
+extern void sched_fetch_numa_topology(int *type, int *max_distance);
 extern void sched_init_numa(int offline_node);
 extern void sched_update_numa(int cpu, bool online);
 extern void sched_domains_numa_masks_set(unsigned int cpu);
kernel/sched/topology.c +140 -115

453 lines not on this page · raw message

Signature · 1 line
2.30.2
reviewer Reviewed-by on v1 1/1
#1 lore raw
Quoted 20 lines · Huang Ying

When Peter Zijlstra reviewed commit 0fb3978b0aac ("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out that sched_domains_numa_distance and sched_domains_numa_masks are made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.

So we update sched_domains_numa_distance and sched_domains_numa_masks and some other related sched topology parameters atomically to address the potential issues.

[1] https://lkml.kernel.org/r/[email protected]

Cc Valentin Schneider <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Srikar Dronamraju <[email protected]>
Quoted 5 lines

Changelog:

v2:

- Addressed comments from Valentin Schneider, Thanks!

One small bug and a whitespace nit below, with those fixed:

FWIW I briefly tested this vs hotplug on QEMU.

kernel/sched/topology.c void sched_init_numa(int offline_node) · line 1876

Quoted 7 lines
kernel/sched/topology.c +1 -2
@@ -1806,8 +1873,7 @@ void sched_init_numa(int offline_node)
 
                      if (distance < LOCAL_DISTANCE || distance >= NR_DISTANCE_VALUES) {
                              sched_numa_warn("Invalid distance value range");
-				bitmap_free(distance_map);
-				return;
+                                goto free_bitmap;

The indentation here is wrong (spaces vs tabs).

Quoted 6 lines
kernel/sched/topology.c +0 -0
                      }
 
                      bitmap_set(distance_map, distance, 1);
      /* Compute default topology size */
      for (i = 0; sched_domain_topology[i].mask; i++);

After the original boot this will now be the default topology with the NUMA bits on top, so we'll just keep growing the array every time we hotplug a node. This should use sched_domain_topology_default instead (ditto for the copy loop further down).

#2 lore raw

Hi, Valentin,

Quoted 23 lines · Valentin Schneider
Quoted 22 lines · Huang Ying

When Peter Zijlstra reviewed commit 0fb3978b0aac ("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out that sched_domains_numa_distance and sched_domains_numa_masks are made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.

So we update sched_domains_numa_distance and sched_domains_numa_masks and some other related sched topology parameters atomically to address the potential issues.

[1] https://lkml.kernel.org/r/[email protected]

Cc Valentin Schneider <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Rik van Riel <[email protected]>
Cc Srikar Dronamraju <[email protected]>

Changelog:

Quoted 7 lines
Quoted 3 lines

v2:

- Addressed comments from Valentin Schneider, Thanks!

One small bug and a whitespace nit below, with those fixed:

Thanks! Your review comments help me a lot!

Quoted 11 lines

FWIW I briefly tested this vs hotplug on QEMU.

kernel/sched/topology.c void sched_init_numa(int offline_node) · line 1876

Quoted 7 lines
kernel/sched/topology.c +1 -2
@@ -1806,8 +1873,7 @@ void sched_init_numa(int offline_node)
 
                       if (distance < LOCAL_DISTANCE || distance >= NR_DISTANCE_VALUES) {
                               sched_numa_warn("Invalid distance value range");
-				bitmap_free(distance_map);
-				return;
+                                goto free_bitmap;

The indentation here is wrong (spaces vs tabs).

Yes. Will fix in the next version.

Quoted 4 lines
Quoted 3 lines
kernel/sched/topology.c +0 -0
                       }
 
                       bitmap_set(distance_map, distance, 1);
Quoted 8 lines
Quoted 2 lines

      /* Compute default topology size */
      for (i = 0; sched_domain_topology[i].mask; i++);

After the original boot this will now be the default topology with the NUMA bits on top, so we'll just keep growing the array every time we hotplug a node. This should use sched_domain_topology_default instead (ditto for the copy loop further down).

Yes. You are right! Thanks for pointing this out. Will fix this in the next version.

Best Regards,
Huang, Ying