cgroup: update task migration path
Tejun Heo
maintainerPATCH 1/1 cgroup: update task migration path
Hello,
Currently, when migrating a task or process from one cgroup to another, a flex_array is used to keep track of the target tasks and associated css_sets. This has a couple issues.
* flex_array size is limited. Given the current data structure, the limit is ~87k on 64bit, which is pretty high but not impossible to hit.
* If multiple targets are being migrated, as migrating each target involves memory allocation, it can fail at any point. cgroup core doesn't keep track of enough state to roll back partial migration either, so it ends up aborting with some targets migrated with no way of finding out which. While this isn't a big issue now, we're gonna be making more use of multi-target migration.
This patchset updates task migration path such that
* task->cg_list and css_sets are also used to keep track of targets during migration so that no extra memory allocation is necessary to keep track of migration targets.
* Migration is split into several stages so that all preparations which may fail can be performed for all targets before actually starting migrating tasks. Ignoring ->can_attach() failure, this can guarantee all-or-nothing semantics of multi-target migration.
This patchset contains the following five patches.
0001-cgroup-add-css_set-mg_tasks.patch 0002-cgroup-use-css_set-mg_tasks-to-track-target-tasks-du.patch 0003-cgroup-separate-out-cset_group_from_root-from-task_c.patch 0004-cgroup-split-process-task-migration-into-four-steps.patch 0005-cgroup-update-cgroup_transfer_tasks-to-either-succee.patch
0001-0002 update migration path so that it uses task->cg_list for keeping track of migration targets.
0003-0004 split migration into multiple steps so that preparation which may fail can be done up-front.
0005 updates cgroup_transfer_tasks() to use multi-step migration to guarantee all-or-nothing behavior as long as ->can_attach() doesn't fail.
This patch is on top of
cgroup/for-3.15 f7cef064aa01 ("Merge branch 'driver-core-next' into cgroup/for-3.15")
+ [1] [PATCHSET v2 cgroup/for-3.15] cgroup: convert to kernfs
+ [2] [PATCHSET v2 cgroup/for-3.15] cgroup: cleanups after kernfs conversion
+ [3] [PATCHSET cgroup/for-3.15] cgroup: more cleanups
and also available in the following git branch.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git review-migration-update
diffstat follows.
include/linux/cgroup.h | 25 ++ kernel/cgroup.c | 510 ++++++++++++++++++++++++++++++++----------------- 2 files changed, 363 insertions(+), 172 deletions(-)
Thanks.
--
tejun
[1] http://lkml.kernel.org/g/1391876127-7134-1-git-send-email-tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
[2] http://lkml.kernel.org/g/1391877509-10855-1-git-send-email-tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
[3] http://lkml.kernel.org/g/1391953964-22088-1-git-send-email-tj-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Tejun Heo
maintainerPATCH 1/5 cgroup: add css_set->mg_tasks
Currently, while migrating tasks from one cgroup to another, cgroup_attach_task() builds a flex array of all target tasks; unfortunately, this has a couple issues.
* Flex array has size limit. On 64bit, struct task_and_cgroup is 24bytes making the flex element limit around 87k. It is a high number but not impossible to hit. This means that the current cgroup implementation can't migrate a process with more than 87k threads.
* Process migration involves memory allocation whose size is dependent on the number of threads the process has. This means that cgroup core can't guarantee success or failure of multi-process migrations as memory allocation failure can happen in the middle. This is in part because cgroup can't grab threadgroup locks of multiple processes at the same time, so when there are multiple processes to migrate, it is imposible to tell how many tasks are to be migrated beforehand.
Note that this already affects cgroup_transfer_tasks(). cgroup currently cannot guarantee atomic success or failure of the operation. It may fail in the middle and after such failure cgroup doesn't have enough information to roll back properly. It just aborts with some tasks migrated and others not.
To resolve the situation, we're going to use task->cg_list during migration too. Instead of building a separate array, target tasks will be linked into a dedicated migration list_head on the owning css_set. Tasks on the migration list are treated the same as tasks on the usual tasks list; however, being on a separate list allows cgroup migration code path to keep track of the target tasks by simply keeping the list of css_sets with tasks being migrated, making unpredictable dynamic allocation unnecessary.
In prepartion of such migration path update, this patch introduces css_set->mg_tasks list and updates css_set task iterations so that they walk both css_set->tasks and ->mg_tasks. Note that ->mg_tasks isn't used yet.
--- include/linux/cgroup.h | 8 ++++++-- kernel/cgroup.c | 56 +++++++++++++++++++++++++++++++++----------------- 2 files changed, 43 insertions(+), 21 deletions(-)
include/linux/cgroup.h +6 -2
@@ -322,10 +322,14 @@ struct css_set {struct hlist_node hlist;/* -* List running through all tasks using this cgroup -* group. Protected by css_set_lock +* Lists running through all tasks using this cgroup group. +* mg_tasks lists tasks which belong to this cset but are in the +* process of being migrated out or in. Protected by +* css_set_rwsem, but, during migration, once tasks are moved to +* mg_tasks, it can be read safely while holding cgroup_mutex. */ struct list_head tasks; +struct list_head mg_tasks;/* * List of cgrp_cset_links pointing at cgroups referenced from this
kernel/cgroup.c +37 -19
92 lines not on this page · raw message
Signature · 1 line
1.8.5.3
Tejun Heo
maintainerPATCH 2/5 cgroup: use css_set->mg_tasks to track target tasks during migration
In reply to Tejun HeoCurrently, while migrating tasks from one cgroup to another, cgroup_attach_task() builds a flex array of all target tasks; unfortunately, this has a couple issues.
* Flex array has size limit. On 64bit, struct task_and_cgroup is 24bytes making the flex element limit around 87k. It is a high number but not impossible to hit. This means that the current cgroup implementation can't migrate a process with more than 87k threads.
* Process migration involves memory allocation whose size is dependent on the number of threads the process has. This means that cgroup core can't guarantee success or failure of multi-process migrations as memory allocation failure can happen in the middle. This is in part because cgroup can't grab threadgroup locks of multiple processes at the same time, so when there are multiple processes to migrate, it is imposible to tell how many tasks are to be migrated beforehand.
Note that this already affects cgroup_transfer_tasks(). cgroup currently cannot guarantee atomic success or failure of the operation. It may fail in the middle and after such failure cgroup doesn't have enough information to roll back properly. It just aborts with some tasks migrated and others not.
To resolve the situation, this patch updates the migration path to use task->cg_list to track target tasks. The previous patch already added css_set->mg_tasks and updated iterations in non-migration paths to include them during task migration. This patch updates migration path to actually make use of it.
Instead of putting onto a flex_array, each target task is moved from its css_set->tasks list to css_set->mg_tasks and the migration path keeps trace of all the source css_sets and the associated cgroups. Once all source css_sets are determined, the destination css_set for each is determined, linked to the matching source css_set and put on a separate list.
To iterate the target tasks, migration path just needs to iterat through either the source or target css_sets, depending on whether migration has been committed or not, and the tasks on their ->mg_tasks lists. cgroup_taskset is updated to contain the list_heads for source and target css_sets and the iteration cursor. cgroup_taskset_*() are accordingly updated to walk through css_sets and their ->mg_tasks.
This resolves the above listed issues with moderate additional complexity.
--- include/linux/cgroup.h | 16 ++++ kernel/cgroup.c | 223 +++++++++++++++++++++++++------------------------ 2 files changed, 131 insertions(+), 108 deletions(-)
include/linux/cgroup.h +16 -0
@@ -344,6 +344,22 @@ struct css_set {*/ struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT ]; +/* +* List of csets participating in the on-going migration either as +* source or destination. Protected by cgroup_mutex. +*/ +struct list_head mg_node; + +/* +* If this cset is acting as the source of migration the following +* two fields are set. mg_src_cgrp is the source cgroup of the +* on-going migration and mg_dst_cset is the destination cset the +* target tasks on this cset should be migrated to. Protected by +* cgroup_mutex. +*/ +struct cgroup *mg_src_cgrp; +struct css_set *mg_dst_cset; +/* For RCU-protected deletion */ struct rcu_head rcu_head; };
kernel/cgroup.c +115 -108
328 lines not on this page · raw message
Signature · 1 line
1.8.5.3
Tejun Heo
maintainerPATCH 3/5 cgroup: separate out cset_group_from_root() from task_cgroup_from_root()
In reply to Tejun HeoThis will be used by the planned migration path update.
--- kernel/cgroup.c | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-)
kernel/cgroup.c +17 -12
@@ -758,25 +758,15 @@ static void cgroup_destroy_root(struct cgroupfs_root *root)cgroup_free_root (root); } -/* -* Return the cgroup for "task" from the given hierarchy. Must be -* called with cgroup_mutex and css_set_rwsem held. -*/ -static struct cgroup *task_cgroup_from_root (struct task_struct *task, +/* look up cgroup associated with given css_set on the specified hierarchy */ +static struct cgroup *cset_cgroup_from_root (struct css_set *cset,struct cgroupfs_root *root) { -struct css_set *cset;struct cgroup *res =NULL ;lockdep_assert_held (&cgroup_mutex);lockdep_assert_held (&css_set_rwsem); -/* -* No need to lock the task - since we hold cgroup_mutex the -* task can't change groups, so the only thing that can happen -* is that it exits and its css is set back to init_css_set. -*/ - cset =task_css_set (task);if (cset == &init_css_set) { res = &root->top_cgroup; }else {
@@ -797,6 +787,21 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task, }/* +* Return the cgroup for "task" from the given hierarchy. Must be +* called with cgroup_mutex and css_set_rwsem held. +*/ +static struct cgroup *task_cgroup_from_root (struct task_struct *task, +struct cgroupfs_root *root) +{ +/* +* No need to lock the task - since we hold cgroup_mutex the +* task can't change groups, so the only thing that can happen +* is that it exits and its css is set back to init_css_set. +*/ +return cset_cgroup_from_root (task_css_set (task), root); +} + +/* * There is one global cgroup mutex. We also require taking * task_lock() when dereferencing a task's cgroup subsys pointers. * See "The task_lock() exception", at the end of this comment.
Signature · 1 line
1.8.5.3
Tejun Heo
maintainerPATCH 4/5 cgroup: split process / task migration into four steps
In reply to Tejun HeoCurrently, process / task migration is a single operation which may fail depending on memory pressure or the involved controllers' ->can_attach() callbacks. One problem with this approach is migration of multiple targets. It's impossible to tell whether a given target will be successfully migrated beforehand and cgroup core can't keep track of enough states to roll back after intermediate failure.
This is already an issue with cgroup_transfer_tasks(). Also, we're gonna need multiple target migration for unified hierarchy.
This patch splits migration into four stages - cgroup_migrate_add_src(), cgroup_migrate_prepare_dst(), cgroup_migrate() and cgroup_migrate_finish(), where cgroup_migrate_prepare_dst() performs all the operations which may fail due to allocation failure without actually migrating the target.
The four separate stages mean that, disregarding ->can_attach() failures, the success or failure of multi target migration can be determined before performing any actual migration. If preparations of all targets succeed, the whole thing will succeed. If not, the whole operation can fail without any side-effect.
Since the previous patch to use css_set->mg_tasks to keep track of migration targets, the only thing which may need memory allocation during migration is the target css_sets. cgroup_migrate_prepare() pins all source and target css_sets and link them up. Note that this can be performed without holding threadgroup_lock even if the target is a process. As long as cgroup_mutex is held, no new css_set can be put into play.
--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 240 +++++++++++++++++++++++++++++++++++++------------ 2 files changed, 182 insertions(+), 59 deletions(-)
include/linux/cgroup.h +1 -0
@@ -348,6 +348,7 @@ struct css_set {* List of csets participating in the on-going migration either as * source or destination. Protected by cgroup_mutex. */ +struct list_head mg_preload_node;struct list_head mg_node;/*
kernel/cgroup.c +181 -59
306 lines not on this page · raw message
Signature · 1 line
1.8.5.3
Tejun Heo
maintainerPATCH 5/5 cgroup: update cgroup_transfer_tasks() to either succeed or fail
In reply to Tejun Heocgroup_transfer_tasks() can currently fail in the middle due to memory allocation failure. When that happens, the function just aborts and returns error code and there's no way to tell how many actually got migrated at the point of failure and or to revert the partial migration.
Update it to use cgroup_migrate{_add_src|prepare_dst|migrate|finish}() so that the function either succeeds or fails as a whole as long as ->can_attach() doesn't fail.
--- kernel/cgroup.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-)
kernel/cgroup.c +23 -5
@@ -2809,10 +2809,28 @@ void css_task_iter_end(struct css_task_iter *it)*/ int cgroup_transfer_tasks (struct cgroup *to,struct cgroup *from) { +LIST_HEAD (preloaded_csets); +struct cgrp_cset_link *link;struct css_task_iter it;struct task_struct *task; -int ret =0 ; +int ret; + +mutex_lock (&cgroup_mutex); + +/* all tasks in @from are being moved, all csets are source */ +down_read (&css_set_rwsem); +list_for_each_entry (link, &from->cset_links, cset_link) +cgroup_migrate_add_src (link->cset, to, &preloaded_csets); +up_read (&css_set_rwsem); + ret =cgroup_migrate_prepare_dst (to, &preloaded_csets); +if (ret) + goto out_err; + +/* +* Migrate tasks one-by-one until @form is empty. This fails iff +* ->can_attach() fails. +*/ do { css_task_iter_start(&from->dummy_css, &it); task = css_task_iter_next(&it);
@@ -2821,13 +2839,13 @@ int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from)css_task_iter_end (&it);if (task) { -mutex_lock (&cgroup_mutex); - ret =cgroup_attach_task (to, task, false); -mutex_unlock (&cgroup_mutex); + ret =cgroup_migrate (to, task, false);put_task_struct (task); } }while (task && !ret); - +out_err: +cgroup_migrate_finish (&preloaded_csets); +mutex_unlock (&cgroup_mutex);return ret; }
Signature · 1 line
1.8.5.3
PATCH 5/5 cgroup: update cgroup_transfer_tasks() to either succeed or fail
In reply to a message not in the archivecgroup_transfer_tasks() can currently fail in the middle due to memory allocation failure. When that happens, the function just aborts and returns error code and there's no way to tell how many actually got migrated at the point of failure and or to revert the partial migration.
Update it to use cgroup_migrate{_add_src|prepare_dst|migrate|finish}() so that the function either succeeds or fails as a whole as long as ->can_attach() doesn't fail.
--- kernel/cgroup.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-)
kernel/cgroup.c +23 -5
@@ -2809,10 +2809,28 @@ void css_task_iter_end(struct css_task_iter *it)*/ int cgroup_transfer_tasks (struct cgroup *to,struct cgroup *from) { +LIST_HEAD (preloaded_csets); +struct cgrp_cset_link *link;struct css_task_iter it;struct task_struct *task; -int ret =0 ; +int ret; + +mutex_lock (&cgroup_mutex); + +/* all tasks in @from are being moved, all csets are source */ +down_read (&css_set_rwsem); +list_for_each_entry (link, &from->cset_links, cset_link) +cgroup_migrate_add_src (link->cset, to, &preloaded_csets); +up_read (&css_set_rwsem); + ret =cgroup_migrate_prepare_dst (to, &preloaded_csets); +if (ret) + goto out_err; + +/* +* Migrate tasks one-by-one until @form is empty. This fails iff +* ->can_attach() fails. +*/ do { css_task_iter_start(&from->dummy_css, &it); task = css_task_iter_next(&it);
@@ -2821,13 +2839,13 @@ int cgroup_transfer_tasks(struct cgroup *to, struct cgroup *from)css_task_iter_end (&it);if (task) { -mutex_lock (&cgroup_mutex); - ret =cgroup_attach_task (to, task, false); -mutex_unlock (&cgroup_mutex); + ret =cgroup_migrate (to, task, false);put_task_struct (task); } }while (task && !ret); - +out_err: +cgroup_migrate_finish (&preloaded_csets); +mutex_unlock (&cgroup_mutex);return ret; }
Signature · 1 line
1.8.5.3
Tejun Heo
maintainerRe: [PATCH 5/5] cgroup: update cgroup_transfer_tasks() to either succeed or fail
In reply to Tejun HeoQuoted 9 lines · Tejun Heo
cgroup_transfer_tasks()can currently fail in the middle due to memory allocation failure. When that happens, the function just aborts and returns error code and there's no way to tell how many actually got migrated at the point of failure and or to revert the partial migration.Update it to use cgroup_migrate{_add_src|prepare_dst|migrate|finish}() so that the function either succeeds or fails as a whole as long as ->can_attach() doesn't fail.
While this solves one aspect of the problem, it may still race with fork and may leave newly forked tasks behind. I'll think more about it.
Thanks.
Signature · 1 line
tejun