Skip to contentSkip to conversation

mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

PATCH In review Wei Yang 2 participants 5 messages Feb 14, 2020 – Feb 18, 2020

Subsystems (3)
lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH v2 1/1 mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu affinity.

Current routine does like this:

a) Iterate all the numa node
b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to a particular numa node. So it is not necessary to iterate on every nodes on the system. This new onlined cpu just affect kswapd cpu affinity of this particular node.

For b) several cpumask operation is used to check whether the node has an online CPU. Since at this point we are sure one of our CPU onlined, we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected kswapd.

---
v2:
* rephrase the changelog

---
 mm/vmscan.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
mm/vmscan.c +10 -9
index 665f33258cd7..acc5af82b6ed 100644
@@ -4029,18 +4029,19 @@ unsigned long shrink_all_memory(unsigned long nr_to_reclaim)
    restore their cpu bindings. */
 static int kswapd_cpu_online(unsigned int cpu)
 {
-	int nid;
+	int nid = cpu_to_node(cpu);
+	pg_data_t *pgdat;
+	const struct cpumask *mask;
 
-	for_each_node_state(nid, N_MEMORY) {
-		pg_data_t *pgdat = NODE_DATA(nid);
-		const struct cpumask *mask;
+	if (!node_state(nid, N_MEMORY))
+		return 0;
 
-		mask = cpumask_of_node(pgdat->node_id);
+	pgdat = NODE_DATA(nid);
+	mask = cpumask_of_node(nid);
+
+	/* One of our CPUs online: restore mask */
+	set_cpus_allowed_ptr(pgdat->kswapd, mask);
 
-		if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
-			/* One of our CPUs online: restore mask */
-			set_cpus_allowed_ptr(pgdat->kswapd, mask);
-	}
 	return 0;
 }
 
Signature · 1 line
2.17.1
reviewer
#1 lore raw
Quoted 13 lines · Wei Yang

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu affinity.

Current routine does like this:

a) Iterate all the numa node
b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to a particular numa node. So it is not necessary to iterate on every nodes on the system. This new onlined cpu just affect kswapd cpu affinity of this particular node.

Quoted 6 lines

For b) several cpumask operation is used to check whether the node has an online CPU. Since at this point we are sure one of our CPU onlined, we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected kswapd.

How have you tested this patch?

Also this is an old code and quite convoluted but does it still work as inteded? I mean, I do not see any cpu offline callback to reduce the cpu mask as all the CPUs for the given node go offline? Wouldn't the scheduler simply go and fallback to no affinity if that happens? In other words what is the value of kswapd_cpu_online in the first place?

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#2 lore raw
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Wei Yang

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu affinity.

Current routine does like this:

a) Iterate all the numa node
b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to a particular numa node. So it is not necessary to iterate on every nodes on the system. This new onlined cpu just affect kswapd cpu affinity of this particular node.

Quoted 9 lines
Quoted 6 lines

For b) several cpumask operation is used to check whether the node has an online CPU. Since at this point we are sure one of our CPU onlined, we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected kswapd.

How have you tested this patch?

I online one cpu and confirm the "cpu" is the one we just onlined.

If my understanding is correct, this is the expected behavior.

Quoted 3 lines

Also this is an old code and quite convoluted but does it still work as inteded? I mean, I do not see any cpu offline callback to reduce the cpu mask as all the CPUs for the given node go offline? Wouldn't the

You are right, I didn't see the counterpart for cpu offline. This is the question I want to ask. Seems we didn't handle it at the very beginning.

Quoted 3 lines

scheduler simply go and fallback to no affinity if that happens? In other words what is the value of kswapd_cpu_online in the first place?

Some cases may this function be useful.

If we have a memory node which doesn't have any online cpu, the default cpumask is not set. After one of the cpu online, we want to change cpu affinity.

Or we want to add more cpu to the system, we could allow kswapd use more cpu resources. Otherwise, kswapd would be limited to those original cpus.

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Wei Yang
Help you, Help me
reviewer
#3 lore raw
Quoted 40 lines · Wei Yang
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Wei Yang

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu affinity.

Current routine does like this:

a) Iterate all the numa node
b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to a particular numa node. So it is not necessary to iterate on every nodes on the system. This new onlined cpu just affect kswapd cpu affinity of this particular node.

Quoted 9 lines
Quoted 6 lines

For b) several cpumask operation is used to check whether the node has an online CPU. Since at this point we are sure one of our CPU onlined, we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected kswapd.

How have you tested this patch?

I online one cpu and confirm the "cpu" is the one we just onlined.

If my understanding is correct, this is the expected behavior.

Quoted 3 lines

Also this is an old code and quite convoluted but does it still work as inteded? I mean, I do not see any cpu offline callback to reduce the cpu mask as all the CPUs for the given node go offline? Wouldn't the

You are right, I didn't see the counterpart for cpu offline. This is the question I want to ask. Seems we didn't handle it at the very beginning.

Quoted 3 lines

scheduler simply go and fallback to no affinity if that happens? In other words what is the value of kswapd_cpu_online in the first place?

Quoted 8 lines

Some cases may this function be useful.

If we have a memory node which doesn't have any online cpu, the default cpumask is not set. After one of the cpu online, we want to change cpu affinity.

Or we want to add more cpu to the system, we could allow kswapd use more cpu resources. Otherwise, kswapd would be limited to those original cpus.

OK, so the usecase is when a NUMA node gains a new CPU which wasn't there at the time when the node got onlined. Is this a scenario we really do care about? While not completely impossible I haven't seen a system which would allow such a runtime configurability. Maybe it would be simply easier to drop the callback for now until we have a real world usecase to support it and have it documented.

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#4 lore raw
Quoted 50 lines · Michal Hocko
Quoted 40 lines · Wei Yang
Quoted 14 lines · Michal Hocko
Quoted 13 lines · Wei Yang

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu affinity.

Current routine does like this:

a) Iterate all the numa node
b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to a particular numa node. So it is not necessary to iterate on every nodes on the system. This new onlined cpu just affect kswapd cpu affinity of this particular node.

Quoted 9 lines
Quoted 6 lines

For b) several cpumask operation is used to check whether the node has an online CPU. Since at this point we are sure one of our CPU onlined, we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected kswapd.

How have you tested this patch?

I online one cpu and confirm the "cpu" is the one we just onlined.

If my understanding is correct, this is the expected behavior.

Quoted 3 lines

Also this is an old code and quite convoluted but does it still work as inteded? I mean, I do not see any cpu offline callback to reduce the cpu mask as all the CPUs for the given node go offline? Wouldn't the

You are right, I didn't see the counterpart for cpu offline. This is the question I want to ask. Seems we didn't handle it at the very beginning.

Quoted 3 lines

scheduler simply go and fallback to no affinity if that happens? In other words what is the value of kswapd_cpu_online in the first place?

Quoted 8 lines

Some cases may this function be useful.

If we have a memory node which doesn't have any online cpu, the default cpumask is not set. After one of the cpu online, we want to change cpu affinity.

Or we want to add more cpu to the system, we could allow kswapd use more cpu resources. Otherwise, kswapd would be limited to those original cpus.

Quoted 6 lines

OK, so the usecase is when a NUMA node gains a new CPU which wasn't there at the time when the node got onlined. Is this a scenario we really do care about? While not completely impossible I haven't seen a system which would allow such a runtime configurability. Maybe it would be simply easier to drop the callback for now until we have a real world usecase to support it and have it documented.

I am fine with this suggestion. Let me prepare v3.

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Wei Yang
Help you, Help me