cgroup/cpuset: Support multiple destination cpusets for cpuset_*attach()
Subsystems (3)
Waiman Long
maintainerv2 cover cgroup/cpuset: Support multiple destination cpusets for cpuset_*attach()
v2:
- Make sure that attach_ctx.old_cs won't be set to a source cpuset that is also the destination cpuset.
v1: https://lore.kernel.org/lkml/[email protected]
This is a follow-up patch series to [1] to properly handle a special case for cpuset task migration operation where the source and destination cpusets are the same. Patch 1 handle this special cases by skipping tasks are not migrating with respect to cpuset. Patch 2 enables cpuset_*attach() to handle the case where thare many destination cpusets which when a cpuset controller is enabled. Patch 3 adds a new test case into test_cpuset to test proper handling of cpu affinity when cpuset controller is disabled.
[1] https://lore.kernel.org/lkml/[email protected]
Michal Koutný (1):
selftests/cgroup: Add test for cpuset affinity on controller disable
Waiman Long (2):
cgroup/cpuset: Handle the special case of non-moving tasks in
cpuset_can_attach()
cgroup/cpuset: Support multiple destination cpusets for
cpuset_*attach()
kernel/cgroup/cpuset.c | 131 ++++++---- tools/testing/selftests/cgroup/test_cpuset.c | 243 +++++++++++++++++++ 2 files changed, 321 insertions(+), 53 deletions(-)
Signature · 1 line
2.55.0
Waiman Long
maintainerPATCH v2 1/3 cgroup/cpuset: Handle the special case of non-moving tasks in cpuset_can_attach()
With cgroup v2 migration of a multithreaded process having threads in different cgroups of a threaded subtree, it is possible that cpuset_can_attach() can be called with tasks that are not migrating with respect to cpuset if cpuset controller is not enabled in some of the subtree cgroups. IOW, the old cpuset can be the same as the new one. This can cause problem when we need to track the set of old cpusets and the new cpusets in singly linked lists as a cpuset cannot be in both lists.
As reported by Tejun, the following is an example threaded subtree with partial cpuset delegation that can cause this issue to show up.
P (+cpuset)
|- R (cpuset) <- destination | `- C (no cpuset) -> effective cpuset == R
`- W (cpuset)
Group leader in R, thread_a in C, thread_b in W; migrate the whole process into R (echo $PID > R/cgroup.procs). thread_a moves C->R:
its cgroup changes so compare_css_sets() keeps it in the taskset, but its cpuset css is unchanged (C inherits R's), so task_cs() == cs == R. cpuset is in ss_mask because thread_b (W->R) changed. can_attach() then tags R as a source (thread_a) and the destination (thread_b):
Handle this special case by skipping tasks that are not migrating in cpuset_can_attach() and avoid calling cpuset_can_attach_check() in this case. By doing so, the destination cpuset will not be put into source cpuset linked list.
As the source cpuset cannot be easily determined in cpuset_attach(), unnecessary work can be performed if a task is not actually migrating. However, no harm will be done except wasting some CPU cycles. If it happens that none of the tasks is migrating, attach_ctx.old_cs will be NULL and task iteration won't be needed.
--- kernel/cgroup/cpuset.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-)
kernel/cgroup/cpuset.c +23 -10
@@ -3134,20 +3134,12 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)bool setsched_check;int ret; -/* used later by cpuset_attach() */ - attach_ctx.old_cs =task_cs (cgroup_taskset_first (tset, &css)); - oldcs = attach_ctx.old_cs; - cs =css_cs (css); - + oldcs =NULL ;mutex_lock (&cpuset_mutex); + attach_ctx.old_cs =NULL ;/* Used later in cpuset_attach_task() */ attach_ctx.cpus_updated = false; attach_ctx.mems_updated = false; -/* Check to see if task is allowed in the cpuset */ - ret =cpuset_can_attach_check (cs, oldcs, &setsched_check); -if (ret) - goto out_unlock; -/* * The attach_ctx.old_cs is used mainly by cpuset_migrate_mm() to get * the old_mems_allowed value. There are two ways that many-to-one
@@ -3164,17 +3156,30 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)* of child cpusets must always be a subset of the parent. So no real * page migration will be necessary no matter which child cpuset is * selected as attach_ctx.old_cs. +* +* For a v2 threaded subtree where cpuset isn't enabled in some of the +* cgroups, it is possible that oldcs == cs for some of the tasks. +* In this case, we can skip checking on those tasks as there is no +* actual migration wrt cpuset. */ cgroup_taskset_for_each (task, css, tset) {struct cpuset *new_oldcs =task_cs (task); + cs =css_cs (css);if (new_oldcs != oldcs) { oldcs = new_oldcs; +if (oldcs == cs) +continue ; +if (!attach_ctx.old_cs) + attach_ctx.old_cs = oldcs; ret =cpuset_can_attach_check (cs, oldcs, &setsched_check);if (ret) goto out_unlock; } +if (oldcs == cs) +continue ; + ret =task_can_attach (task);if (ret) goto out_unlock;
@@ -3312,6 +3317,14 @@ static void cpuset_attach(struct cgroup_taskset *tset) attach_ctx.task_work_queued = false;guarantee_online_mems (cs, &attach_ctx.nodemask_to); +/* +* attach_ctx.old_cs can only be NULL if no task is actually migrating. +* This is highly unlikely. If it happens at all, there is nothing we +* need to do here. +*/ +if (unlikely (!attach_ctx.old_cs)) + goto out; +/* * In the default hierarchy, enabling cpuset in the child cgroups * will trigger a cpuset_attach() call with no change in effective cpus
Signature · 1 line
2.55.0
Waiman Long
maintainerPATCH v2 2/3 cgroup/cpuset: Support multiple destination cpusets for cpuset_*attach()
In reply to Waiman LongThe only case where the cgroup_taskset structure requires task migration to multiple cpusets is when enabling a cpuset controller in cgroup v2 where the newly created child cpusets inherits the same effective CPUs and memory nodes from the parent. In that case, task migration can happen directly with no update to tasks' CPU and memory nodes assignment and no further work needed from the cpuset side except updating nr_deadline_tasks when DL tasks are involved and setting old_mems_allowed in the child cpusets.
Do that by tracking all the destination cpusets with a new dst_cs_head singly linked list. The reset_migrate_dl_data() function is integrated into clear_attach_data() so that it can be used for both source and destination cpusets.
A warning will be printed if there are multiple destination cpusets but it is not on default hierarchy or when the CPUs or memory nodes change.
--- kernel/cgroup/cpuset.c | 104 +++++++++++++++++++++++------------------ 1 file changed, 58 insertions(+), 46 deletions(-)
kernel/cgroup/cpuset.c +58 -46
@@ -366,10 +366,12 @@ static struct {bool cpus_updated;bool mems_updated;bool task_work_queued; +bool many_dest_cs;/* Have many destination cpusets */ struct cpuset *old_cs;/* Source cpuset */ nodemask_t nodemask_to; } attach_ctx;static LLIST_HEAD (src_cs_head); +static LLIST_HEAD (dst_cs_head);/* * Wait if task attach is in progress until it is done and then acquire
@@ -3052,6 +3054,9 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,if (!llist_on_list (&oldcs->attach_node))llist_add (&oldcs->attach_node, &src_cs_head); +if (!llist_on_list (&cs->attach_node)) +llist_add (&cs->attach_node, &dst_cs_head); + cpus_updated = !cpumask_equal (cs->effective_cpus, oldcs->effective_cpus); mems_updated = !nodes_equal (cs->effective_mems, oldcs->effective_mems);
@@ -3080,35 +3085,31 @@ static int cpuset_can_attach_check(struct cpuset *cs, struct cpuset *oldcs,return 0 ; } -static int cpuset_reserve_dl_bw (struct cpuset *cs) +static int cpuset_reserve_dl_bw (void ) { +struct cpuset *cs;int cpu, ret; -if (!cs->sum_migrate_dl_bw) -return 0 ; +llist_for_each_entry (cs, dst_cs_head.first, attach_node) { +if (!cs->sum_migrate_dl_bw) +continue ; - cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); -if (unlikely (cpu >= nr_cpu_ids)) -return -EINVAL ; + cpu =cpumask_any_and (cpu_active_mask, cs->effective_cpus); +if (unlikely (cpu >= nr_cpu_ids)) +return -EINVAL ; - ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); -if (ret) -return ret; + ret =dl_bw_alloc (cpu, cs->sum_migrate_dl_bw); +if (ret) +return ret; - cs->dl_bw_cpu = cpu; + cs->dl_bw_cpu = cpu; + }return 0 ; } -static void reset_migrate_dl_data (struct cpuset *cs) -{ - cs->nr_migrate_dl_tasks =0 ; - cs->sum_migrate_dl_bw =0 ; - cs->dl_bw_cpu =-1 ; -} -/* * Clear and optionally apply (@cancel is false) the attach related data in the -* source cpusets. +* source or destination cpuset. */ static void clear_attach_data (struct llist_head *head,bool cancel) {
@@ -3120,7 +3121,11 @@ static void clear_attach_data(struct llist_head *head, bool cancel)if (cs->nr_migrate_dl_tasks) {if (!cancel)atomic_add (cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks); +else if (cs->dl_bw_cpu >=0 )/* && cacnel */ +dl_bw_free (cs->dl_bw_cpu, cs->sum_migrate_dl_bw); cs->nr_migrate_dl_tasks =0 ; + cs->sum_migrate_dl_bw =0 ; + cs->dl_bw_cpu =-1 ; } } }
@@ -3134,11 +3139,12 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)bool setsched_check;int ret; - oldcs =NULL ; + cs = oldcs =NULL ;mutex_lock (&cpuset_mutex); attach_ctx.old_cs =NULL ;/* Used later in cpuset_attach_task() */ attach_ctx.cpus_updated = false; attach_ctx.mems_updated = false; + attach_ctx.many_dest_cs = false;/* * The attach_ctx.old_cs is used mainly by cpuset_migrate_mm() to get
@@ -3163,10 +3169,13 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)* actual migration wrt cpuset. */ cgroup_taskset_for_each (task, css, tset) { +struct cpuset *new_cs =css_cs (css);struct cpuset *new_oldcs =task_cs (task); - cs =css_cs (css); -if (new_oldcs != oldcs) { +if ((new_oldcs != oldcs) || (new_cs != cs)) { +if (cs && (new_cs != cs)) + attach_ctx.many_dest_cs = true; + cs = new_cs; oldcs = new_oldcs;if (oldcs == cs)continue ;
@@ -3207,12 +3216,28 @@ static int cpuset_can_attach(struct cgroup_taskset *tset) } } - ret =cpuset_reserve_dl_bw (cs); +/* +* The only case where there are multiple destination cpusets for +* task migration is when enabling a v2 cpuset controllers where +* tasks will be migrated to multiple child cpusets from a parent +* cpuset with the same effective CPUs and memory nodes. IOW, +* both attach_cpus_updated and attach_mems_updated should be false. +* If not, it is a condition that the current code cannot handle. +* Print a warning and abort the attach operation as further code +* change may be needed. +*/ +if (WARN_ON_ONCE (attach_ctx.many_dest_cs && (!cpuset_v2 () || + attach_ctx.cpus_updated || attach_ctx.mems_updated))) { + ret = -EINVAL ; + goto out_unlock; + } + + ret =cpuset_reserve_dl_bw (); out_unlock:if (ret) { -reset_migrate_dl_data (cs);/* Destination cpuset only */ clear_attach_data (&src_cs_head, true); +clear_attach_data (&dst_cs_head, true); }else { attach_ctx.in_progress++; }
@@ -3223,22 +3248,10 @@ static int cpuset_can_attach(struct cgroup_taskset *tset)static void cpuset_cancel_attach (struct cgroup_taskset *tset) { -struct cgroup_subsys_state *css; -struct cpuset *cs; - -cgroup_taskset_first (tset, &css); - cs =css_cs (css); -mutex_lock (&cpuset_mutex);dec_attach_in_progress_locked ();clear_attach_data (&src_cs_head, true); - -if (cs->dl_bw_cpu >=0 ) -dl_bw_free (cs->dl_bw_cpu, cs->sum_migrate_dl_bw); - -if (cs->nr_migrate_dl_tasks) -reset_migrate_dl_data (cs); - +clear_attach_data (&dst_cs_head, true);mutex_unlock (&cpuset_mutex); }
@@ -3320,7 +3333,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)/* * attach_ctx.old_cs can only be NULL if no task is actually migrating. * This is highly unlikely. If it happens at all, there is nothing we -* need to do here. +* need to do here not even setting old_mems_allowed. */ if (unlikely (!attach_ctx.old_cs)) goto out;
@@ -3329,25 +3342,24 @@ static void cpuset_attach(struct cgroup_taskset *tset)* In the default hierarchy, enabling cpuset in the child cgroups * will trigger a cpuset_attach() call with no change in effective cpus * and mems. In that case, we can optimize out by skipping the task -* iteration and update. +* iteration and the destination cpuset list is iterated to set +* old_mems_allowed. */ -if (cpuset_v2 () && !attach_ctx.cpus_updated && !attach_ctx.mems_updated) +if (cpuset_v2 () && !attach_ctx.cpus_updated && !attach_ctx.mems_updated) { +llist_for_each_entry (cs, dst_cs_head.first, attach_node) + cs->old_mems_allowed = attach_ctx.nodemask_to; goto out; + }cgroup_taskset_for_each (task, css, tset)cpuset_attach_task (cs, task); -out:if (attach_ctx.task_work_queued)schedule_flush_migrate_mm (); cs->old_mems_allowed = attach_ctx.nodemask_to; - -if (cs->nr_migrate_dl_tasks) { -atomic_add (cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks); -reset_migrate_dl_data (cs); - } - +out:clear_attach_data (&src_cs_head, false); +clear_attach_data (&dst_cs_head, false);dec_attach_in_progress_locked ();mutex_unlock (&cpuset_mutex);
Signature · 1 line
2.55.0
Waiman Long
maintainerPATCH v2 3/3 selftests/cgroup: Add test for cpuset affinity on controller disable
In reply to Waiman LongFrom: Michal Koutný <[email protected]>
Add a new selftest that exposes a bug in cpuset_attach() where thread CPU affinity is not properly updated when the cpuset controller is disabled in a threaded cgroup hierarchy.
The test creates a threaded cgroup hierarchy with two child cgroups (A and B) having different cpuset.cpus constraints:
- Parent: cpuset.cpus=0-1
- Child A: cpuset.cpus=0-1
- Child B: cpuset.cpus=1 (restricted to CPU 1 only)
A multithreaded process is created with threads placed in different cgroups. When the cpuset controller is disabled on the parent, thread affinities should be updated to match the parent's cpuset.
Expected behavior:
- thread_a affinity: {0-1} before and after (unchanged)
- thread_b affinity: {1} before, {0-1} after (expanded)
Current buggy behavior:
- thread_b affinity remains {1} after controller disable
--- tools/testing/selftests/cgroup/test_cpuset.c | 243 +++++++++++++++++++ 1 file changed, 243 insertions(+)
tools/testing/selftests/cgroup/test_cpuset.c +243 -0
@@ -1,7 +1,13 @@// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include <assert.h> #include <linux/limits.h> +#include <pthread.h> +#include <sched.h> #include <signal.h> +#include <sys/syscall.h> +#include <unistd.h> #include "kselftest.h" #include "cgroup_util.h"
@@ -232,6 +238,242 @@ static int test_cpuset_perms_subtree(const char *root)return ret; } +static int get_cpu_affinity (cpu_set_t *mask) +{ +CPU_ZERO (mask); +return sched_getaffinity (0 ,sizeof (*mask), mask); +} + +static int cpu_set_equal (cpu_set_t *dst,unsigned long mask) +{ +cpu_set_t expected; + +CPU_ZERO (&expected); +assert (sizeof (mask) <CPU_SETSIZE ); + +for (int cpu =0 ; cpu <sizeof (mask); ++cpu) +if ((1UL << cpu) & mask) +CPU_SET (cpu, &expected); + +return CPU_EQUAL (&expected, dst); +} + +enum test_phase { +AFFINITY_SETUP , +AFFINITY_THREAD_A_READY , +AFFINITY_THREADS_READY , +AFFINITY_CONTROLLER_DISABLED , +AFFINITY_COMPLETE , +AFFINITY_ERROR +}; + +struct thread_args { +const char *cgroup; +cpu_set_t *affinity_before; +cpu_set_t *affinity_after; +enum test_phase ready_phase; +}; + +static pthread_mutex_t test_mutex =PTHREAD_MUTEX_INITIALIZER ; +static pthread_cond_t test_cond =PTHREAD_COND_INITIALIZER ; +static enum test_phase test_phase; + +static void *affinity_thread_fn (void *arg) +{ +struct thread_args *args = (struct thread_args *)arg; + +if (cg_enter_current_thread (args->cgroup)) + goto fail; + +if (get_cpu_affinity (args->affinity_before) !=0 ) + goto fail; + +pthread_mutex_lock (&test_mutex); +if (test_phase < args->ready_phase) + test_phase = args->ready_phase; +pthread_cond_broadcast (&test_cond); + +while (test_phase <AFFINITY_CONTROLLER_DISABLED ) +pthread_cond_wait (&test_cond, &test_mutex); +pthread_mutex_unlock (&test_mutex); + +if (get_cpu_affinity (args->affinity_after) !=0 ) + goto fail; + + +return NULL ; + +fail: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_ERROR ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); +return NULL ; +} + +/* +* Test that disabling cpuset controller properly updates thread affinity. +* +* This test exposes a bug in cpuset_attach() where threads in child cgroups +* don't get their affinity updated when the cpuset controller is disabled. +* +* Setup: +* - Create parent cgroup with cpuset.cpus=0-1 +* - Create child A with cpuset.cpus=0-1 +* - Create child B with cpuset.cpus=1 +* - Place multithreaded process: group leader + thread_a in A, thread_b in B +* - Disable cpuset controller on parent +* +* Expected: thread_b's affinity should expand from {1} to {0-1} +* Buggy: thread_b's affinity remains {1} +*/ +static int test_cpuset_affinity_on_controller_disable (const char *root) +{ +char *parent =NULL , *child_a =NULL , *child_b =NULL ; +pthread_t thread_a, thread_b; +int thread_a_created =0 , thread_b_created =0 ; +cpu_set_t affinity_a_before, affinity_a_after; +cpu_set_t affinity_b_before, affinity_b_after; +int ret =KSFT_FAIL ; + + parent =cg_name (root,"cpuset_affinity_test" ); +if (!parent) + goto cleanup; +if (cg_create (parent)) + goto cleanup; +if (cg_write (parent,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_a =cg_name (parent,"A" ); +if (!child_a) + goto cleanup; +if (cg_create (child_a)) + goto cleanup; +if (cg_write (child_a,"cgroup.type" ,"threaded" )) + goto cleanup; + + child_b =cg_name (parent,"B" ); +if (!child_b) + goto cleanup; +if (cg_create (child_b)) + goto cleanup; +if (cg_write (child_b,"cgroup.type" ,"threaded" )) + goto cleanup; + +/* Now enable cpuset controller in parent */ +if (cg_write (parent,"cgroup.subtree_control" ,"+cpuset" )) { + ret =KSFT_SKIP ; + goto cleanup; + } + +/* Set CPU affinity constraints */ +if (cg_write (parent,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_a,"cpuset.cpus" ,"0-1" )) + goto cleanup; +if (cg_write (child_b,"cpuset.cpus" ,"1" )) + goto cleanup; + +/* Move group leader (main thread) to child A */ +if (cg_enter_current (child_a)) + goto cleanup; + +/* Create threads - they will move themselves to their respective cgroups */ + test_phase =AFFINITY_SETUP ; + +struct thread_args args_a = { + .cgroup = child_a, + .affinity_before = &affinity_a_before, + .affinity_after = &affinity_a_after, + .ready_phase =AFFINITY_THREAD_A_READY , + }; +if (pthread_create (&thread_a,NULL , affinity_thread_fn, &args_a)) + goto cleanup; + thread_a_created =1 ; + +struct thread_args args_b = { + .cgroup = child_b, + .affinity_before = &affinity_b_before, + .affinity_after = &affinity_b_after, + .ready_phase =AFFINITY_THREADS_READY , + }; +if (pthread_create (&thread_b,NULL , affinity_thread_fn, &args_b)) + goto cleanup_threads; + thread_b_created =1 ; + +pthread_mutex_lock (&test_mutex); +while (test_phase <AFFINITY_THREADS_READY ) +pthread_cond_wait (&test_cond, &test_mutex); + +/* If a thread failed during setup, bail out */ +if (test_phase ==AFFINITY_ERROR ) { +pthread_mutex_unlock (&test_mutex); + goto cleanup_threads; + } +pthread_mutex_unlock (&test_mutex); + +if (!cpu_set_equal (&affinity_a_before,0x3 )) { +ksft_print_msg ("FAIL: thread_a initial affinity incorrect\n" ); + goto cleanup_threads; + } + +if (!cpu_set_equal (&affinity_b_before,0x2 )) { +ksft_print_msg ("FAIL: thread_b initial affinity incorrect\n" ); + goto cleanup_threads; + } + +/* Disable cpuset controller - this should trigger affinity update */ +if (cg_write (parent,"cgroup.subtree_control" ,"-cpuset" )) + goto cleanup_threads; + +/* Signal threads to save their final affinity and exit */ +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_CONTROLLER_DISABLED ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +pthread_join (thread_a,NULL ); +pthread_join (thread_b,NULL ); + +/* Verify thread affinities AFTER disabling controller */ +if (!cpu_set_equal (&affinity_a_after,0x3 )) { +ksft_print_msg ("FAIL: thread_a final affinity incorrect\n" ); + goto cleanup; + } + +if (!cpu_set_equal (&affinity_b_after,0x3 )) { +ksft_print_msg ("FAIL: thread_b affinity did not expand to {0-1}\n" ); + goto cleanup; + } + + ret =KSFT_PASS ; + goto cleanup; + +cleanup_threads: +pthread_mutex_lock (&test_mutex); + test_phase =AFFINITY_COMPLETE ; +pthread_cond_broadcast (&test_cond); +pthread_mutex_unlock (&test_mutex); + +if (thread_a_created) +pthread_join (thread_a,NULL ); +if (thread_b_created) +pthread_join (thread_b,NULL ); + +cleanup: +/* Move back to root before cleanup */ +cg_enter_current (root); + +cg_destroy (child_b); +free (child_b); +cg_destroy (child_a); +free (child_a); +cg_destroy (parent); +free (parent); + +return ret; +} +#define T (x) { x, #x }struct cpuset_test {
@@ -241,6 +483,7 @@ struct cpuset_test {T (test_cpuset_perms_object_allow),T (test_cpuset_perms_object_deny),T (test_cpuset_perms_subtree), + T(test_cpuset_affinity_on_controller_disable), };#undef T
Signature · 1 line
2.55.0