Skip to contentSkip to conversation

cgroup/cpuset: fix a memory binding failure for cgroup v2

RFC PATCH In review Feng Tang 3 participants 4 messages Apr 19, 2022 – Apr 24, 2022

Subsystems (3)
lore
Share & tools
lore

Feng Tang

author
#0 lore raw

RFC PATCH 1/1 cgroup/cpuset: fix a memory binding failure for cgroup v2

We got report that setting cpuset.mems failed when the nodemask contains a newly onlined memory node (not enumerated during boot) for cgroup v2, while the binding succeeded for cgroup v1.

The root cause is, for cgroup v2, when a new memory node is onlined, top_cpuset's 'mem_allowed' is not updated with the new nodemask of memory nodes, and the following setting memory nodemask will fail, if the nodemask contains a new node.

Fix it by updating top_cpuset.mems_allowed right after the new memory node is onlined, just like v1.

---
Very likely I missed some details here, but it looks strange that the top_cpuset.mem_allowed is not updatd even after we onlined several memory nodes after boot.

 kernel/cgroup/cpuset.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/cgroup/cpuset.c +1 -2
index 9390bfd9f1cd..b97caaf16374 100644
@@ -3314,8 +3314,7 @@ static void cpuset_hotplug_workfn(struct work_struct *work)
 	/* synchronize mems_allowed to N_MEMORY */
 	if (mems_updated) {
 		spin_lock_irq(&callback_lock);
-		if (!on_dfl)
-			top_cpuset.mems_allowed = new_mems;
+		top_cpuset.mems_allowed = new_mems;
 		top_cpuset.effective_mems = new_mems;
 		spin_unlock_irq(&callback_lock);
 		update_tasks_nodemask(&top_cpuset);
Signature · 1 line
2.27.0
maintainer
#1 lore raw

cc'ing Waiman and copying the whole body.

Waiman, can you please take a look?

Thanks.

Quoted 38 lines · Feng Tang

We got report that setting cpuset.mems failed when the nodemask contains a newly onlined memory node (not enumerated during boot) for cgroup v2, while the binding succeeded for cgroup v1.

The root cause is, for cgroup v2, when a new memory node is onlined, top_cpuset's 'mem_allowed' is not updated with the new nodemask of memory nodes, and the following setting memory nodemask will fail, if the nodemask contains a new node.

Fix it by updating top_cpuset.mems_allowed right after the new memory node is onlined, just like v1.

---
Very likely I missed some details here, but it looks strange that the top_cpuset.mem_allowed is not updatd even after we onlined several memory nodes after boot.

 kernel/cgroup/cpuset.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/cgroup/cpuset.c +1 -2
index 9390bfd9f1cd..b97caaf16374 100644
@@ -3314,8 +3314,7 @@ static void cpuset_hotplug_workfn(struct work_struct *work)
 	/* synchronize mems_allowed to N_MEMORY */
 	if (mems_updated) {
 		spin_lock_irq(&callback_lock);
-		if (!on_dfl)
-			top_cpuset.mems_allowed = new_mems;
+		top_cpuset.mems_allowed = new_mems;
 		top_cpuset.effective_mems = new_mems;
 		spin_unlock_irq(&callback_lock);
 		update_tasks_nodemask(&top_cpuset);
Signature · 1 line
2.27.0
Signature · 1 line
tejun
maintainer
#2 lore raw
Quoted 28 lines · Tejun Heo

cc'ing Waiman and copying the whole body.

Waiman, can you please take a look?

Thanks.

Quoted 21 lines · Feng Tang

We got report that setting cpuset.mems failed when the nodemask contains a newly onlined memory node (not enumerated during boot) for cgroup v2, while the binding succeeded for cgroup v1.

The root cause is, for cgroup v2, when a new memory node is onlined, top_cpuset's 'mem_allowed' is not updated with the new nodemask of memory nodes, and the following setting memory nodemask will fail, if the nodemask contains a new node.

Fix it by updating top_cpuset.mems_allowed right after the new memory node is onlined, just like v1.

---
Very likely I missed some details here, but it looks strange that the top_cpuset.mem_allowed is not updatd even after we onlined several memory nodes after boot.

  kernel/cgroup/cpuset.c | 3 +--
  1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 14 lines
Quoted 5 lines
kernel/cgroup/cpuset.c +0 -0
index 9390bfd9f1cd..b97caaf16374 100644

kernel/cgroup/cpuset.c static void cpuset_hotplug_workfn(struct work_struct *work) · line 3320

Quoted 9 lines
kernel/cgroup/cpuset.c +1 -2
@@ -3314,8 +3314,7 @@ static void cpuset_hotplug_workfn(struct work_struct *work)
  	/* synchronize mems_allowed to N_MEMORY */
  	if (mems_updated) {
  		spin_lock_irq(&callback_lock);
-		if (!on_dfl)
-			top_cpuset.mems_allowed = new_mems;
+		top_cpuset.mems_allowed = new_mems;
  		top_cpuset.effective_mems = new_mems;
  		spin_unlock_irq(&callback_lock);
  		update_tasks_nodemask(&top_cpuset);

The on_dfl check was added by commit 7e88291beefb ("cpuset: make cs->{cpus, mems}_allowed as user-configured masks"). This is the expected behavior for cgroup v2 as we don't want to remove a node because it is hot-removed. However, I do see a problem in case we are adding a node that is not originally in top_cpuset.mems_allowed. We should be allowed to add the extra memory node. So something like

        if (!on_dfl)
                top_cpuset.mems_allowed = new_mems;
        else if (!nodes_subset(new_mems, top_cpuset.mems_allowed)) nodes_or(top_cpuset.mems_allowed, top_cpuset.mems_allowed, new_mems);

For v2, top_cpuset.mems_allowed is set to node_possible_map in cpuset_bind(). Perhaps node_possible_map may not include all the nodes that are hot-pluggable.

I don't know if that is similar problem with cpu_possible_mask or not.

Cheers,
Longman

maintainer
#3 lore raw
Quoted 46 lines · Waiman Long
Quoted 28 lines · Tejun Heo

cc'ing Waiman and copying the whole body.

Waiman, can you please take a look?

Thanks.

Quoted 21 lines · Feng Tang

We got report that setting cpuset.mems failed when the nodemask contains a newly onlined memory node (not enumerated during boot) for cgroup v2, while the binding succeeded for cgroup v1.

The root cause is, for cgroup v2, when a new memory node is onlined, top_cpuset's 'mem_allowed' is not updated with the new nodemask of memory nodes, and the following setting memory nodemask will fail, if the nodemask contains a new node.

Fix it by updating top_cpuset.mems_allowed right after the new memory node is onlined, just like v1.

---
Very likely I missed some details here, but it looks strange that the top_cpuset.mem_allowed is not updatd even after we onlined several memory nodes after boot.

  kernel/cgroup/cpuset.c | 3 +--

  1 file changed, 1 insertion(+), 2 deletions(-)
Quoted 15 lines
Quoted 14 lines
kernel/cgroup/cpuset.c +1 -2
index 9390bfd9f1cd..b97caaf16374 100644
@@ -3314,8 +3314,7 @@ static void cpuset_hotplug_workfn(struct 
 work_struct *work)
       /* synchronize mems_allowed to N_MEMORY */
       if (mems_updated) {
           spin_lock_irq(&callback_lock);
-        if (!on_dfl)
-            top_cpuset.mems_allowed = new_mems;
+        top_cpuset.mems_allowed = new_mems;
           top_cpuset.effective_mems = new_mems;
           spin_unlock_irq(&callback_lock);
Quoted 1 line

          update_tasks_nodemask(&top_cpuset);

The on_dfl check was added by commit 7e88291beefb ("cpuset: make cs->{cpus, mems}_allowed as user-configured masks"). This is the expected behavior for cgroup v2 as we don't want to remove a node because it is hot-removed. However, I do see a problem in case we are adding a node that is not originally in top_cpuset.mems_allowed. We should be allowed to add the extra memory node. So something like

Quoted 8 lines

        if (!on_dfl)
                top_cpuset.mems_allowed = new_mems;
        else if (!nodes_subset(new_mems, top_cpuset.mems_allowed)) nodes_or(top_cpuset.mems_allowed, top_cpuset.mems_allowed, new_mems);

For v2, top_cpuset.mems_allowed is set to node_possible_map in cpuset_bind(). Perhaps node_possible_map may not include all the nodes that are hot-pluggable.

I don't know if that is similar problem with cpu_possible_mask or not.

Ah, I know why the top_cpuset.mems_allowed isn't correctly set. There are 2 places where it is set:

 - cpuset_bind(): top_cpuset.mems_allowed = node_possible_map
 - cpuset_init_smp(): top_cpuset.mems_allowed = node_states[N_MEMORY];

The first one is correct, but the second isn't. It turns out that cpuset_init_smp() can be called later than cpuset_bind().

[    2.611207] cpuset_bind called
[    2.631182] cblist_init_generic: Setting adjustable number of callback queues. [    3.082357] cpuset_init_smp called

So the proper fix may be to make sure that top_cpuset.mems_allowed is initialized correctly.

Cheers,
Longman