sched,topology: Update sched topology atomically
PATCH 1/1 sched,topology: Update sched topology atomically
When Peter Zijlstra reviewed commit 0fb3978b0aac ("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out that sched_domains_numa_distance and sched_domains_numa_masks are made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.
So we update sched_domains_numa_distance and sched_domains_numa_masks and some other related sched topology parameters atomically to address the potential issues.
[1] https://lkml.kernel.org/r/[email protected]
Changelog:
v2:
- Addressed comments from Valentin Schneider, Thanks!
--- kernel/sched/fair.c | 33 +++--- kernel/sched/sched.h | 3 +- kernel/sched/topology.c | 255 ++++++++++++++++++++++------------------ 3 files changed, 158 insertions(+), 133 deletions(-)
kernel/sched/fair.c +17 -16
@@ -1294,17 +1294,17 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid, int lim_dist, bool task) {unsigned long score =0 ; -int node, max_dist; +int node, max_dist, type; + +sched_fetch_numa_topology (&type, &max_dist);/* * All nodes are directly connected, and the same distance * from each other. No need for fancy placement algorithms. */ -if (sched_numa_topology_type ==NUMA_DIRECT ) +if (type ==NUMA_DIRECT )return 0 ; -/* sched_max_numa_distance may be changed in parallel. */ - max_dist =READ_ONCE (sched_max_numa_distance);/* * This code is called for each node, introducing N^2 complexity, * which should be ok given the number of nodes rarely exceeds 8.
@@ -1327,7 +1327,7 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid,* "hoplimit", only nodes closer by than "hoplimit" are part * of each group. Skip other nodes. */ -if (sched_numa_topology_type ==NUMA_BACKPLANE && dist >= lim_dist) +if (type ==NUMA_BACKPLANE && dist >= lim_dist)continue ;/* Add up the faults from nearby nodes. */
@@ -1344,7 +1344,7 @@ static unsigned long score_nearby_nodes(struct task_struct *p, int nid,* The further away a node is, the less the faults count. * This seems to result in good task placement. */ -if (sched_numa_topology_type ==NUMA_GLUELESS_MESH ) { +if (type ==NUMA_GLUELESS_MESH ) { faults *= (max_dist - dist); faults /= (max_dist -LOCAL_DISTANCE ); }
@@ -1971,7 +1971,9 @@ static int task_numa_migrate(struct task_struct *p)long taskimp, groupimp;struct numa_group *ng;struct rq *best_rq; -int nid, ret, dist; +int nid, ret, dist, type; + +sched_fetch_numa_topology (&type,NULL );/* * Pick the lowest SD_NUMA domain, as that would have the smallest
@@ -2026,8 +2028,7 @@ static int task_numa_migrate(struct task_struct *p)continue ; dist =node_distance (env.src_nid, env.dst_nid); -if (sched_numa_topology_type ==NUMA_BACKPLANE && - dist != env.dist) { +if (type ==NUMA_BACKPLANE && dist != env.dist) { taskweight =task_weight (p, env.src_nid, dist); groupweight =group_weight (p, env.src_nid, dist); }
@@ -2262,10 +2263,12 @@ static u64 numa_get_avg_runtime(struct task_struct *p, u64 *period)static int preferred_group_nid (struct task_struct *p,int nid) {nodemask_t nodes; -int dist; +int dist, max_dist, type; + +sched_fetch_numa_topology (&type, &max_dist);/* Direct connections between all NUMA nodes. */ -if (sched_numa_topology_type ==NUMA_DIRECT ) +if (type ==NUMA_DIRECT )return nid;/*
@@ -2273,14 +2276,12 @@ static int preferred_group_nid(struct task_struct *p, int nid)* scores nodes according to the number of NUMA hinting faults on * both the node itself, and on nearby nodes. */ -if (sched_numa_topology_type ==NUMA_GLUELESS_MESH ) { +if (type ==NUMA_GLUELESS_MESH ) {unsigned long score, max_score =0 ;int node, max_node = nid; - dist = sched_max_numa_distance; -for_each_node_state (node,N_CPU ) { - score =group_weight (p, node, dist); + score =group_weight (p, node, max_dist);if (score > max_score) { max_score = score; max_node = node;
@@ -2299,7 +2300,7 @@ static int preferred_group_nid(struct task_struct *p, int nid)* keep the complexity of the search down. */ nodes = node_states[N_CPU ]; -for (dist = sched_max_numa_distance; dist >LOCAL_DISTANCE ; dist--) { +for (dist = max_dist; dist >LOCAL_DISTANCE ; dist--) {unsigned long max_faults =0 ;nodemask_t max_group =NODE_MASK_NONE ;int a, b;
kernel/sched/sched.h +1 -2
@@ -1638,9 +1638,8 @@ enum numa_topology_type {NUMA_GLUELESS_MESH , NUMA_BACKPLANE, }; -extern enum numa_topology_type sched_numa_topology_type; -extern int sched_max_numa_distance;extern bool find_numa_distance (int distance); +extern void sched_fetch_numa_topology (int *type,int *max_distance);extern void sched_init_numa (int offline_node);extern void sched_update_numa (int cpu,bool online);extern void sched_domains_numa_masks_set (unsigned int cpu);
kernel/sched/topology.c +140 -115
453 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 20 lines · Huang Ying
When Peter Zijlstra reviewed commit
0fb3978b0aac("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out thatsched_domains_numa_distanceandsched_domains_numa_masksare made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.So we update
sched_domains_numa_distanceandsched_domains_numa_masksand some other related sched topology parameters atomically to address the potential issues.[1] https://lkml.kernel.org/r/[email protected]
Signed-off-by "Huang, Ying" <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Cc Valentin Schneider <[email protected]>Cc Ingo Molnar <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Srikar Dronamraju <[email protected]>
Quoted 5 lines
Changelog:
v2:
- Addressed comments from Valentin Schneider, Thanks!
One small bug and a whitespace nit below, with those fixed:
FWIW I briefly tested this vs hotplug on QEMU.
kernel/sched/topology.c void sched_init_numa(int offline_node) · line 1876
Quoted 7 lines
kernel/sched/topology.c +1
-2@@ -1806,8 +1873,7 @@ void sched_init_numa(int offline_node)if (distance <LOCAL_DISTANCE || distance >=NR_DISTANCE_VALUES ) {sched_numa_warn ("Invalid distance value range" ); -bitmap_free (distance_map); -return ; + goto free_bitmap;
The indentation here is wrong (spaces vs tabs).
Quoted 6 lines
kernel/sched/topology.c +0
-0}bitmap_set (distance_map, distance,1 );/* Compute default topology size */ for (i = 0; sched_domain_topology[i].mask; i++);
After the original boot this will now be the default topology with the NUMA bits on top, so we'll just keep growing the array every time we hotplug a node. This should use sched_domain_topology_default instead (ditto for the copy loop further down).
Hi, Valentin,
Quoted 23 lines · Valentin Schneider
Quoted 22 lines · Huang Ying
When Peter Zijlstra reviewed commit
0fb3978b0aac("sched/numa: Fix NUMA topology for systems with CPU-less nodes") [1], he pointed out thatsched_domains_numa_distanceandsched_domains_numa_masksare made separate RCU variables. That could go side-ways if there were a function using both, although there isn't for now.So we update
sched_domains_numa_distanceandsched_domains_numa_masksand some other related sched topology parameters atomically to address the potential issues.[1] https://lkml.kernel.org/r/[email protected]
Signed-off-by "Huang, Ying" <[email protected]>Suggested-by Peter Zijlstra <[email protected]>Cc Valentin Schneider <[email protected]>Cc Ingo Molnar <[email protected]>Cc Mel Gorman <[email protected]>Cc Rik van Riel <[email protected]>Cc Srikar Dronamraju <[email protected]>Changelog:
Quoted 7 lines
Quoted 3 lines
v2:
- Addressed comments from Valentin Schneider, Thanks!
One small bug and a whitespace nit below, with those fixed:
Reviewed-by Valentin Schneider <[email protected]>
Thanks! Your review comments help me a lot!
Quoted 11 lines
FWIW I briefly tested this vs hotplug on QEMU.
kernel/sched/topology.c void sched_init_numa(int offline_node) · line 1876
Quoted 7 lines
kernel/sched/topology.c +1
-2@@ -1806,8 +1873,7 @@ void sched_init_numa(int offline_node) if (distance < LOCAL_DISTANCE || distance >= NR_DISTANCE_VALUES) { sched_numa_warn("Invalid distance value range"); -bitmap_free (distance_map); -return ; + goto free_bitmap;The indentation here is wrong (spaces vs tabs).
Yes. Will fix in the next version.
Quoted 4 lines
Quoted 3 lines
kernel/sched/topology.c +0
-0} bitmap_set(distance_map, distance, 1);
Quoted 8 lines
Quoted 2 lines
/* Compute default topology size */
for (i = 0; sched_domain_topology[i].mask; i++);After the original boot this will now be the default topology with the NUMA bits on top, so we'll just keep growing the array every time we hotplug a node. This should use
sched_domain_topology_defaultinstead (ditto for the copy loop further down).
Yes. You are right! Thanks for pointing this out. Will fix this in the next version.
Best Regards,
Huang, Ying