cgroup: Remove RCU from task->cgroups
PATCH 1/1 cgroup: Remove RCU from task->cgroups
The synchronize_rcu call in cgroup_attach_task can be very expensive. All fastpath accesses to task->cgroups already use task_lock() or cgroup_lock() to protect against updates, and only the CGROUP_DEBUG files have RCU read-side critical sections.
This patch replaces rcu_read_lock() with task_lock(current) around the debug file acceses to current->cgroups and removes the synchronize_rcu call in cgroup_attach_task.
--- kernel/cgroup.c | 22 ++++++++-------------- 1 files changed, 8 insertions(+), 14 deletions(-)
kernel/cgroup.c +8 -14
@@ -725,14 +725,11 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task,* cgroup_attach_task(), which overwrites one tasks cgroup pointer with * another. It does so using cgroup_mutex, however there are * several performance critical places that need to reference -* task->cgroup without the expense of grabbing a system global +* task->cgroups without the expense of grabbing a system global * mutex. Therefore except as noted below, when dereferencing or, as -* in cgroup_attach_task(), modifying a task'ss cgroup pointer we use +* in cgroup_attach_task(), modifying a task's cgroups pointer we use * task_lock(), which acts on a spinlock (task->alloc_lock) already in * the task_struct routinely used for such matters. -* -* P.S. One more locking exception. RCU is used to guard the -* update of a tasks cgroup pointer by cgroup_attach_task() */ /**
@@ -1786,7 +1783,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) retval = -ESRCH ; goto out; } -rcu_assign_pointer (tsk->cgroups, newcg); + tsk->cgroups = newcg;task_unlock (tsk);/* Update the css_set linked lists if we're using them */
@@ -1802,7 +1799,6 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) ss->attach (ss, cgrp, oldcgrp, tsk, false); }set_bit (CGRP_RELEASABLE , &oldcgrp->flags); -synchronize_rcu ();put_css_set (cg);/*
@@ -4827,9 +4823,9 @@ static u64 current_css_set_refcount_read(struct cgroup *cont, {u64 count; -rcu_read_lock (); +task_lock (current); count =atomic_read (¤t->cgroups->refcount); -rcu_read_unlock (); +task_unlock (current);return count; }
@@ -4838,12 +4834,10 @@ static int current_css_set_cg_links_read(struct cgroup *cont, struct seq_file *seq) {struct cg_cgroup_link *link; -struct css_set *cg;read_lock (&css_set_lock); -rcu_read_lock (); - cg =rcu_dereference (current->cgroups); -list_for_each_entry (link, &cg->cg_links, cg_link_list) { +task_lock (current); +list_for_each_entry (link, ¤t->cgroups->cg_links, cg_link_list) {struct cgroup *c = link->cgrp;const char *name;
@@ -4854,7 +4848,7 @@ static int current_css_set_cg_links_read(struct cgroup *cont,seq_printf (seq,"Root %d group %s\n" , c->root->hierarchy_id, name); } -rcu_read_unlock (); +task_unlock (current);read_unlock (&css_set_lock);return 0 ; }
Signature · 1 line
1.7.3.1
A message that is not in the archive (sent off-list or not yet received): 1290304824-22722-1-git-send-email-ccross-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsalready usetask_lock()orcgroup_lock()to protect against updates, and only theCGROUP_DEBUGfiles have RCU read-side critical sections.
Quoted 9 lines
This patch replaces
rcu_read_lock()withtask_lock(current)around the debug file acceses tocurrent->cgroupsand removes thesynchronize_rcucall incgroup_attach_task.Signed-off-by Colin Cross <[email protected]>---
kernel/cgroup.c | 22 ++++++++--------------1 files changed, 8 insertions(+), 14 deletions(-)
This patch isn't correct, there's an rcu_dereference I missed inside task_group(), and that's the important one.
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsalready usetask_lock()orcgroup_lock()to protect against updates, and only theCGROUP_DEBUGfiles have RCU read-side critical sections.
Quoted 9 lines
This patch replaces
rcu_read_lock()withtask_lock(current)around the debug file acceses tocurrent->cgroupsand removes thesynchronize_rcucall incgroup_attach_task.Signed-off-by Colin Cross <[email protected]>---
kernel/cgroup.c | 22 ++++++++--------------1 files changed, 8 insertions(+), 14 deletions(-)
This patch isn't correct, there's an rcu_dereference I missed inside task_group(), and that's the important one.
A message that is not in the archive (sent off-list or not yet received): AANLkTikx6d0_VFtZ4zWQucRCf=vFt7N2M6=0jpnKasEE-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
PATCH 1/1 cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
The synchronize_rcu call in cgroup_attach_task can be very expensive. All fastpath accesses to task->cgroups that expect task->cgroups not to change already use task_lock() or cgroup_lock() to protect against updates, and, in cgroup.c, only the CGROUP_DEBUG files have RCU read-side critical sections.
sched.c uses RCU read-side-critical sections on task->cgroups, but only to ensure that a dereference of task->cgroups does not become invalid, not that it doesn't change.
This patch adds a function put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferenced task->cgroups in sched.c have completed before the css_set is deleted. The synchronize_rcu()/put_css_set() combo in cgroup_attach_task() can then be replaced with put_css_set_rcu().
Also converts the CGROUP_DEBUG files that access current->cgroups to use task_lock(current) instead of rcu_read_lock().
---
This version fixes the problems with the previous patch by keeping the use of RCU in cgroup_attach_task, but allowing cgroup_attach_task to return immediately by deferring the final put_css_reg to an rcu callback.
include/linux/cgroup.h | 4 +++ kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 12 deletions(-)
include/linux/cgroup.h +4 -0
@@ -287,6 +287,10 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; + +/* For RCU-delayed puts */ +atomic_t delayed_put_count; +struct work_struct delayed_put_work; };/*
kernel/cgroup.c +46 -12
119 lines not on this page · raw message
Signature · 1 line
1.7.3.1
PATCH 1/1 cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Colin CrossThe synchronize_rcu call in cgroup_attach_task can be very expensive. All fastpath accesses to task->cgroups that expect task->cgroups not to change already use task_lock() or cgroup_lock() to protect against updates, and, in cgroup.c, only the CGROUP_DEBUG files have RCU read-side critical sections.
sched.c uses RCU read-side-critical sections on task->cgroups, but only to ensure that a dereference of task->cgroups does not become invalid, not that it doesn't change.
This patch adds a function put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferenced task->cgroups in sched.c have completed before the css_set is deleted. The synchronize_rcu()/put_css_set() combo in cgroup_attach_task() can then be replaced with put_css_set_rcu().
Also converts the CGROUP_DEBUG files that access current->cgroups to use task_lock(current) instead of rcu_read_lock().
---
This version fixes the problems with the previous patch by keeping the use of RCU in cgroup_attach_task, but allowing cgroup_attach_task to return immediately by deferring the final put_css_reg to an rcu callback.
include/linux/cgroup.h | 4 +++ kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 12 deletions(-)
include/linux/cgroup.h +4 -0
@@ -287,6 +287,10 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; + +/* For RCU-delayed puts */ +atomic_t delayed_put_count; +struct work_struct delayed_put_work; };/*
kernel/cgroup.c +46 -12
119 lines not on this page · raw message
Signature · 1 line
1.7.3.1
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 11 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.
Other cgroup subsystems also use rcu_read_lock to access task->cgroups, for example net_cls cgroup and device cgroup.
I don't think the performance of task attaching is so critically important that we have to use call_rcu() instead of synchronize_rcu()?
Quoted 9 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().
Quoted 4 lines
Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().
What for? What do we gain from doing this for those debug interfaces?
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++ kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 12 deletions(-)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 8 lines · Li Zefan
Quoted 7 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
Quoted 7 lines
Quoted 4 lines
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.Other cgroup subsystems also use
rcu_read_lockto accesstask->cgroups, for examplenet_clscgroup and device cgroup.
I believe the same comment applies as sched.c, I'll update the commit message.
Quoted 2 lines
I don't think the performance of task attaching is so critically important that we have to use
call_rcu()instead ofsynchronize_rcu()?
On my desktop, moving a task between cgroups averages 100 ms, and on an Tegra2 SMP ARM platform it takes 20 ms. Moving a task with many threads can take hundreds of milliseconds or more. With this patch it takes 50 microseconds to move one task, a 400x improvement.
Quoted 9 lines
Quoted 8 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().
Quoted 7 lines
Quoted 4 lines
Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().What for? What do we gain from doing this for those debug interfaces?
Left over from the previous patch that incorrectly dropped RCU completely. I'll put the rcu_read_locks back.
Quoted 13 lines
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++
kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++----------2 files changed, 50 insertions(+), 12 deletions(-)
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Li ZefanQuoted 8 lines · Li Zefan
Quoted 7 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
Quoted 7 lines
Quoted 4 lines
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.Other cgroup subsystems also use
rcu_read_lockto accesstask->cgroups, for examplenet_clscgroup and device cgroup.
I believe the same comment applies as sched.c, I'll update the commit message.
Quoted 2 lines
I don't think the performance of task attaching is so critically important that we have to use
call_rcu()instead ofsynchronize_rcu()?
On my desktop, moving a task between cgroups averages 100 ms, and on an Tegra2 SMP ARM platform it takes 20 ms. Moving a task with many threads can take hundreds of milliseconds or more. With this patch it takes 50 microseconds to move one task, a 400x improvement.
Quoted 9 lines
Quoted 8 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().
Quoted 7 lines
Quoted 4 lines
Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().What for? What do we gain from doing this for those debug interfaces?
Left over from the previous patch that incorrectly dropped RCU completely. I'll put the rcu_read_locks back.
Quoted 13 lines
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++
kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++----------2 files changed, 50 insertions(+), 12 deletions(-)
A message that is not in the archive (sent off-list or not yet received): AANLkTimjpW6NZ6fEiVi0VzjkpQGVob4=VHsohXUiDQkJ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 39 lines · Colin Cross
Quoted 8 lines · Li Zefan
Quoted 7 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.Quoted 7 lines
Quoted 4 lines
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.Other cgroup subsystems also use
rcu_read_lockto accesstask->cgroups, for examplenet_clscgroup and device cgroup.I believe the same comment applies as
sched.c, I'll update the commit message.Quoted 2 lines
I don't think the performance of task attaching is so critically important that we have to use
call_rcu()instead ofsynchronize_rcu()?On my desktop, moving a task between cgroups averages 100 ms, and on an Tegra2 SMP ARM platform it takes 20 ms. Moving a task with many threads can take hundreds of milliseconds or more. With this patch it takes 50 microseconds to move one task, a 400x improvement.
Quoted 14 lines
Quoted 13 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().
Quoted 19 lines
Quoted 2 lines
What for? What do we gain from doing this for those debug interfaces?
Left over from the previous patch that incorrectly dropped RCU completely. I'll put the
rcu_read_locksback.Quoted 13 lines
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++
kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++----------2 files changed, 50 insertions(+), 12 deletions(-)
This patch has another problem - calling put_css_set_rcu twice before an rcu grace period has elapsed would not guarantee the appropriate rcu grace period for the second call. I'll try a new approach, moving the parts of put_css_set that need to be protected by rcu into free_css_set_rcu.
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 39 lines · Colin Cross
Quoted 8 lines · Li Zefan
Quoted 7 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.Quoted 7 lines
Quoted 4 lines
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.Other cgroup subsystems also use
rcu_read_lockto accesstask->cgroups, for examplenet_clscgroup and device cgroup.I believe the same comment applies as
sched.c, I'll update the commit message.Quoted 2 lines
I don't think the performance of task attaching is so critically important that we have to use
call_rcu()instead ofsynchronize_rcu()?On my desktop, moving a task between cgroups averages 100 ms, and on an Tegra2 SMP ARM platform it takes 20 ms. Moving a task with many threads can take hundreds of milliseconds or more. With this patch it takes 50 microseconds to move one task, a 400x improvement.
Quoted 14 lines
Quoted 13 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().
Quoted 19 lines
Quoted 2 lines
What for? What do we gain from doing this for those debug interfaces?
Left over from the previous patch that incorrectly dropped RCU completely. I'll put the
rcu_read_locksback.Quoted 13 lines
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++
kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++----------2 files changed, 50 insertions(+), 12 deletions(-)
This patch has another problem - calling put_css_set_rcu twice before an rcu grace period has elapsed would not guarantee the appropriate rcu grace period for the second call. I'll try a new approach, moving the parts of put_css_set that need to be protected by rcu into free_css_set_rcu.
A message that is not in the archive (sent off-list or not yet received): 1290398767-15230-1-git-send-email-ccross-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 11 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
sched.cuses RCU read-side-critical sections ontask->cgroups, but only to ensure that a dereference oftask->cgroupsdoes not become invalid, not that it doesn't change.
Other cgroup subsystems also use rcu_read_lock to access task->cgroups, for example net_cls cgroup and device cgroup.
I don't think the performance of task attaching is so critically important that we have to use call_rcu() instead of synchronize_rcu()?
Quoted 9 lines
This patch adds a function
put_css_set_rcu, which delays the put until after a grace period has elapsed. This ensures that any RCU read-side critical sections that dereferencedtask->cgroupsinsched.chave completed before thecss_setis deleted. The synchronize_rcu()/put_css_set() combo incgroup_attach_task()can then be replaced withput_css_set_rcu().
Quoted 4 lines
Also converts the
CGROUP_DEBUGfiles that accesscurrent->cgroupsto usetask_lock(current)instead ofrcu_read_lock().
What for? What do we gain from doing this for those debug interfaces?
Quoted 12 lines
Signed-off-by Colin Cross <[email protected]>---
This version fixes the problems with the previous patch by keeping the use of RCU in
cgroup_attach_task, but allowingcgroup_attach_taskto return immediately by deferring the finalput_css_regto an rcu callback.include/linux/cgroup.h | 4 +++ kernel/cgroup.c | 58 ++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 50 insertions(+), 12 deletions(-)
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to a message not in the archiveQuoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
I definitely agree with the goal of using lighter-weight synchronization than the current synchronize_rcu() call. However, there are definitely some subtleties to worry about in this code.
One of the reasons originally for the current synchronization was to avoid the case of calling subsystem destroy() callbacks while there could still be threads with RCU references to the subsystem state. The fact that synchronize_rcu() was called within a cgroup_mutex critical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.
I suspect that when we moved a lot of the cgroup teardown code from cgroup_rmdir() to cgroup_diput() (which also has a synchronize_rcu() call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.
I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
1) make find_existing_css_set ignore css_set objects with a zero refcount
2) change __put_css_set to be simply
if (atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}
3) move the rest of __put_css_set into a delayed work struct that's scheduled by free_css_set_rcu
4) Get rid of the taskexit parameter - I think we can do that via a simple flag that indicates whether any task has ever been moved into the cgroup.
5) Put extra checks in cgroup_rmdir() such that if it tries to remove a cgroup that has a non-zero refcount, it scans the cgroup's css_sets list - if it finds only zero-refcount entries, then wait (via synchronize_rcu() or some other appropriate means, maybe reusing the CGRP_WAIT_ON_RMDIR mechanism?) until the css_set objects have been fully cleaned up and the cgroup's refcounts have been released. Otherwise the operation of moving the last thread out of a cgroup and immediately deleting the cgroup would very likely fail with an EBUSY
Paul
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.
I definitely agree with the goal of using lighter-weight synchronization than the current synchronize_rcu() call. However, there are definitely some subtleties to worry about in this code.
One of the reasons originally for the current synchronization was to avoid the case of calling subsystem destroy() callbacks while there could still be threads with RCU references to the subsystem state. The fact that synchronize_rcu() was called within a cgroup_mutex critical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.
I suspect that when we moved a lot of the cgroup teardown code from cgroup_rmdir() to cgroup_diput() (which also has a synchronize_rcu() call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.
I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
1) make find_existing_css_set ignore css_set objects with a zero refcount
2) change __put_css_set to be simply
if (atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}
3) move the rest of __put_css_set into a delayed work struct that's scheduled by free_css_set_rcu
4) Get rid of the taskexit parameter - I think we can do that via a simple flag that indicates whether any task has ever been moved into the cgroup.
5) Put extra checks in cgroup_rmdir() such that if it tries to remove a cgroup that has a non-zero refcount, it scans the cgroup's css_sets list - if it finds only zero-refcount entries, then wait (via synchronize_rcu() or some other appropriate means, maybe reusing the CGRP_WAIT_ON_RMDIR mechanism?) until the css_set objects have been fully cleaned up and the cgroup's refcounts have been released. Otherwise the operation of moving the last thread out of a cgroup and immediately deleting the cgroup would very likely fail with an EBUSY
Paul
PATCH 1/1 cgroup: Remove call to synchronize_rcu in cgroup_attach_task
synchronize_rcu can be very expensive, averaging 100 ms in some cases. In cgroup_attach_task, it is used to prevent a task->cgroups pointer dereferenced in an RCU read side critical section from being invalidated by delaying the call to put_css_set until after an RCU grace period.
To avoid the call to synchronize_rcu, make the put_css_set call rcu-safe by moving the deletion of the css_set links into rcu-protected free_css_set_rcu.
The calls to check_for_release in free_css_set_rcu now occur in softirq context, so convert all uses of the release_list_lock spinlock to irq safe versions.
The decrement of the cgroup refcount is no longer synchronous with the call to put_css_set, which can result in the cgroup refcount staying positive after the last call to cgroup_attach_task returns. To allow the cgroup to be deleted with cgroup_rmdir synchronously after cgroup_attach_task, introduce a second refcount, rmdir_count, that is decremented synchronously in put_css_set. If cgroup_rmdir is called on a cgroup for hich rmdir_count is zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.
---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted check_for_release to be callable in softirq context rather than schedule work in free_css_set_rcu. Your css_set scanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs with spin_lock_irqsave in check_for_release?
include/linux/cgroup.h | 6 ++ kernel/cgroup.c | 124 ++++++++++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 52 deletions(-)
include/linux/cgroup.h +6 -0
@@ -202,6 +202,12 @@ struct cgroup {atomic_t count;/* +* separate refcount for rmdir on a cgroup. When rmdir_count is 0, +* rmdir should block until count is 0. +*/ +atomic_t rmdir_count; + +/* * We link our 'sibling' struct into our parent's 'children'. * Our children link their 'sibling' into our 'children'. */
kernel/cgroup.c +72 -52
241 lines not on this page · raw message
Signature · 1 line
1.7.3.1
A message that is not in the archive (sent off-list or not yet received): 1290563018-2804-1-git-send-email-ccross-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
Quoted 6 lines · Colin Cross
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?
free_css_set_rcu needs to take a write lock on css_set_lock to protect the list_del(&link->cgrp_link_list). I'll convert it to schedule work, and change the spin_lock_irqsave back to spin_lock.
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to a message not in the archiveQuoted 36 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?
Quoted 4 lines
include/linux/cgroup.h | 6 ++ kernel/cgroup.c | 124 ++++++++++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 52 deletions(-)
Colin, what became of this patch? I see this in your Tegra tree for Android.
I looked in linux-next but didn't see it there. This resolves a performance issue on MSM SMP so I'm curious if this is going upstream.
Thanks.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Bryan Huntsman
Tested-by Mike BohanRe: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to a message not in the archiveQuoted 307 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?include/linux/cgroup.h | 6 ++ kernel/cgroup.c | 124 ++++++++++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 52 deletions(-)include/linux/cgroup.h +6
-0@@ -202,6 +202,12 @@ struct cgroup {atomic_t count;/* +* separate refcount for rmdir on a cgroup. When rmdir_count is 0, +* rmdir should block until count is 0. +*/ +atomic_t rmdir_count; + +/* * We link our 'sibling' struct into our parent's 'children'. * Our children link their 'sibling' into our 'children'. */ kernel/cgroup.c +70
-50@@ -267,6 +267,33 @@ static void cgroup_release_agent(struct work_struct *work); static DECLARE_WORK(release_agent_work, cgroup_release_agent);static void check_for_release (struct cgroup *cgrp); +/* +* A queue for waiters to do rmdir() cgroup. A tasks will sleep when +* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some +* reference to css->refcnt. In general, this refcnt is expected to goes down +* to zero, soon. +* +* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; +*/ +DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); + +static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) +{ +if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) +wake_up_all (&cgroup_rmdir_waitq); +} + +void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) +{ +css_get (css); +} + +void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) +{ +cgroup_wakeup_rmdir_waiter (css->cgroup); +css_put (css); +} +/* Link structure for associating css_set objects with cgroups */ struct cg_cgroup_link {/* @@ -329,6 +356,22 @@ static struct hlist_head *css_set_hash(struct cgroup_subsys_state *css[])static void free_css_set_rcu (struct rcu_head *obj) { struct css_set *cg = container_of(obj, struct css_set, rcu_head); +struct cg_cgroup_link *link; +struct cg_cgroup_link *saved_link; + +/* Nothing else can have a reference to cg, no need for css_set_lock */ +list_for_each_entry_safe (link, saved_link, &cg->cg_links, + cg_link_list) { +struct cgroup *cgrp = link->cgrp; +list_del (&link->cg_link_list); +list_del (&link->cgrp_link_list); +if (atomic_dec_and_test (&cgrp->count)) { +check_for_release (cgrp); +cgroup_wakeup_rmdir_waiter (cgrp); + } +kfree (link); + } +kfree (cg); }@@ -355,23 +398,20 @@ static void __put_css_set(struct css_set *cg, int taskexit)return ; } -/* This css_set is dead. unlink it and release cgroup refcounts */ hlist_del (&cg->hlist); css_set_count--; +/* This css_set is now unreachable from the css_set_table, but RCU +* read-side critical sections may still have a reference to it. +* Decrement the cgroup rmdir_count so that rmdir's on an empty +* cgroup can block until the free_css_set_rcu callback */ list_for_each_entry_safe (link, saved_link, &cg->cg_links, cg_link_list) {struct cgroup *cgrp = link->cgrp; -list_del (&link->cg_link_list); -list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count) && -notify_on_release (cgrp)) { -if (taskexit) -set_bit (CGRP_RELEASABLE , &cgrp->flags); -check_for_release (cgrp); - } - -kfree (link); +if (taskexit) +set_bit (CGRP_RELEASABLE , &cgrp->flags); +atomic_dec (&cgrp->rmdir_count); +smp_mb (); }write_unlock (&css_set_lock);@@ -571,6 +611,8 @@ static void link_css_set(struct list_head *tmp_cg_links, cgrp_link_list); link->cg = cg; link->cgrp = cgrp; +atomic_inc (&cgrp->rmdir_count); +smp_mb ();/* make sure rmdir_count increments first */ atomic_inc (&cgrp->count);list_move (&link->cgrp_link_list, &cgrp->css_sets);/* @@ -725,9 +767,9 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task,* cgroup_attach_task(), which overwrites one tasks cgroup pointer with * another. It does so using cgroup_mutex, however there are * several performance critical places that need to reference -* task->cgroup without the expense of grabbing a system global +* task->cgroups without the expense of grabbing a system global * mutex. Therefore except as noted below, when dereferencing or, as -* in cgroup_attach_task(), modifying a task'ss cgroup pointer we use +* in cgroup_attach_task(), modifying a task's cgroups pointer we use * task_lock(), which acts on a spinlock (task->alloc_lock) already in * the task_struct routinely used for such matters. * @@ -909,33 +951,6 @@ static void cgroup_d_remove_dir(struct dentry *dentry) }/* -* A queue for waiters to do rmdir() cgroup. A tasks will sleep when -* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some -* reference to css->refcnt. In general, this refcnt is expected to goes down -* to zero, soon. -* -* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; -*/ -DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); - -static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) -{ -if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) -wake_up_all (&cgroup_rmdir_waitq); -} - -void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) -{ -css_get (css); -} - -void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) -{ -cgroup_wakeup_rmdir_waiter (css->cgroup); -css_put (css); -} - -/* * Call with cgroup_mutex held. Drops reference counts on modules, including * any duplicate ones that parse_cgroupfs_options took. If this function * returns an error, no reference counts are touched. @@ -1802,7 +1817,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) ss->attach (ss, cgrp, oldcgrp, tsk, false); }set_bit (CGRP_RELEASABLE , &oldcgrp->flags); -synchronize_rcu (); +/* put_css_set will not destroy cg until after an RCU grace period */ put_css_set (cg);/* @@ -3566,11 +3581,12 @@ static int cgroup_rmdir(struct inode *unused_dir, struct dentry *dentry)DEFINE_WAIT (wait);struct cgroup_event *event, *tmp;int ret; +unsigned long flags;/* the vfs holds both inode->i_mutex already */ again:mutex_lock (&cgroup_mutex); -if (atomic_read (&cgrp->count) !=0 ) { +if (atomic_read (&cgrp->rmdir_count) !=0 ) {mutex_unlock (&cgroup_mutex);return -EBUSY ; }@@ -3603,13 +3619,13 @@ again:mutex_lock (&cgroup_mutex); parent = cgrp->parent; -if (atomic_read (&cgrp->count) || !list_empty (&cgrp->children)) { +if (atomic_read (&cgrp->rmdir_count) || !list_empty (&cgrp->children)) {clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags);mutex_unlock (&cgroup_mutex);return -EBUSY ; }prepare_to_wait (&cgroup_rmdir_waitq, &wait,TASK_INTERRUPTIBLE ); -if (!cgroup_clear_css_refs (cgrp)) { +if (atomic_read (&cgrp->count) !=0 || !cgroup_clear_css_refs (cgrp)) {mutex_unlock (&cgroup_mutex);/* * Because someone may call cgroup_wakeup_rmdir_waiter() before @@ -3627,11 +3643,11 @@ again:finish_wait (&cgroup_rmdir_waitq, &wait);clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);set_bit (CGRP_REMOVED , &cgrp->flags);if (!list_empty (&cgrp->release_list))list_del (&cgrp->release_list); -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);cgroup_lock_hierarchy (cgrp->root);/* delete this cgroup from parent->children */ @@ -4389,6 +4405,8 @@ int cgroup_is_descendant(const struct cgroup *cgrp, struct task_struct *task)static void check_for_release (struct cgroup *cgrp) { +unsigned long flags; +/* All of these checks rely on RCU to keep the cgroup * structure alive */ if (cgroup_is_releasable(cgrp) && !atomic_read(&cgrp->count)@@ -4397,13 +4415,13 @@ static void check_for_release(struct cgroup *cgrp)* already queued for a userspace notification, queue * it now */ int need_schedule_work =0 ; -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);if (!cgroup_is_removed (cgrp) &&list_empty (&cgrp->release_list)) {list_add (&cgrp->release_list, &release_list); need_schedule_work =1 ; } -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);if (need_schedule_work)schedule_work (&release_agent_work); }@@ -4453,9 +4471,11 @@ EXPORT_SYMBOL_GPL(__css_put);*/ static void cgroup_release_agent (struct work_struct *work) { +unsigned long flags; +BUG_ON (work != &release_agent_work);mutex_lock (&cgroup_mutex); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);while (!list_empty (&release_list)) {char *argv[3 ], *envp[3 ];int i;@@ -4464,7 +4484,7 @@ static void cgroup_release_agent(struct work_struct *work) struct cgroup, release_list);list_del_init (&cgrp->release_list); -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags); pathbuf =kmalloc (PAGE_SIZE ,GFP_KERNEL );if (!pathbuf) goto continue_free;@@ -4494,9 +4514,9 @@ static void cgroup_release_agent(struct work_struct *work) continue_free:
kernel/cgroup.c static void cgroup_release_agent(struct work_struct *work) · line 4522
Quoted 10 lines
kernel/cgroup.c +2
-2kfree (pathbuf);kfree (agentbuf); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags); } -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);mutex_unlock (&cgroup_mutex); }
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 6 lines · Colin Cross
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?
free_css_set_rcu needs to take a write lock on css_set_lock to protect the list_del(&link->cgrp_link_list). I'll convert it to schedule work, and change the spin_lock_irqsave back to spin_lock.
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 36 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?
Quoted 4 lines
include/linux/cgroup.h | 6 ++ kernel/cgroup.c | 124 ++++++++++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 52 deletions(-)
Colin, what became of this patch? I see this in your Tegra tree for Android.
I looked in linux-next but didn't see it there. This resolves a performance issue on MSM SMP so I'm curious if this is going upstream.
Thanks.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
Quoted 42 lines · Bryan Huntsman
Quoted 36 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?Quoted 4 lines
include/linux/cgroup.h | 6 ++
kernel/cgroup.c | 124 ++++++++++++++++++++++++++++--------------------2 files changed, 78 insertions(+), 52 deletions(-)
Quoted 9 lines
Colin, what became of this patch? I see this in your Tegra tree for Android.
I looked in linux-next but didn't see it there. This resolves a performance issue on MSM SMP so I'm curious if this is going upstream. Thanks.
It's been posted, there are no outstanding comments I am working on, but they haven't been picked up.
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Bryan HuntsmanQuoted 42 lines · Bryan Huntsman
Quoted 36 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?Quoted 4 lines
include/linux/cgroup.h | 6 ++
kernel/cgroup.c | 124 ++++++++++++++++++++++++++++--------------------2 files changed, 78 insertions(+), 52 deletions(-)
Quoted 9 lines
Colin, what became of this patch? I see this in your Tegra tree for Android.
I looked in linux-next but didn't see it there. This resolves a performance issue on MSM SMP so I'm curious if this is going upstream. Thanks.
It's been posted, there are no outstanding comments I am working on, but they haven't been picked up.
Re: [PATCH] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 307 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks into rcu-protectedfree_css_set_rcu.The calls to
check_for_releaseinfree_css_set_rcunow occur in softirq context, so convert all uses of therelease_list_lockspinlock to irq safe versions.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, introduce a second refcount,rmdir_count, that is decremented synchronously input_css_set. Ifcgroup_rmdiris called on a cgroup for hichrmdir_countis zero but count is nonzero, reuse the rmdir waitqueue to block the rmdir until the rcu callback is called.Signed-off-by Colin Cross <[email protected]>---
This patch is similar to what you described. The main differences are that I used a new atomic to handle the rmdir case, and I converted
check_for_releaseto be callable in softirq context rather than schedule work infree_css_set_rcu. Yourcss_setscanning in rmdir sounds better, I'll take another look at that. Is there any problem with disabling irqs withspin_lock_irqsaveincheck_for_release?include/linux/cgroup.h | 6 ++ kernel/cgroup.c | 124 ++++++++++++++++++++++++++++-------------------- 2 files changed, 78 insertions(+), 52 deletions(-)include/linux/cgroup.h +6
-0@@ -202,6 +202,12 @@ struct cgroup {atomic_t count;/* +* separate refcount for rmdir on a cgroup. When rmdir_count is 0, +* rmdir should block until count is 0. +*/ +atomic_t rmdir_count; + +/* * We link our 'sibling' struct into our parent's 'children'. * Our children link their 'sibling' into our 'children'. */ kernel/cgroup.c +70
-50@@ -267,6 +267,33 @@ static void cgroup_release_agent(struct work_struct *work); static DECLARE_WORK(release_agent_work, cgroup_release_agent);static void check_for_release (struct cgroup *cgrp); +/* +* A queue for waiters to do rmdir() cgroup. A tasks will sleep when +* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some +* reference to css->refcnt. In general, this refcnt is expected to goes down +* to zero, soon. +* +* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; +*/ +DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); + +static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) +{ +if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) +wake_up_all (&cgroup_rmdir_waitq); +} + +void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) +{ +css_get (css); +} + +void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) +{ +cgroup_wakeup_rmdir_waiter (css->cgroup); +css_put (css); +} +/* Link structure for associating css_set objects with cgroups */ struct cg_cgroup_link {/* @@ -329,6 +356,22 @@ static struct hlist_head *css_set_hash(struct cgroup_subsys_state *css[])static void free_css_set_rcu (struct rcu_head *obj) { struct css_set *cg = container_of(obj, struct css_set, rcu_head); +struct cg_cgroup_link *link; +struct cg_cgroup_link *saved_link; + +/* Nothing else can have a reference to cg, no need for css_set_lock */ +list_for_each_entry_safe (link, saved_link, &cg->cg_links, + cg_link_list) { +struct cgroup *cgrp = link->cgrp; +list_del (&link->cg_link_list); +list_del (&link->cgrp_link_list); +if (atomic_dec_and_test (&cgrp->count)) { +check_for_release (cgrp); +cgroup_wakeup_rmdir_waiter (cgrp); + } +kfree (link); + } +kfree (cg); }@@ -355,23 +398,20 @@ static void __put_css_set(struct css_set *cg, int taskexit)return ; } -/* This css_set is dead. unlink it and release cgroup refcounts */ hlist_del (&cg->hlist); css_set_count--; +/* This css_set is now unreachable from the css_set_table, but RCU +* read-side critical sections may still have a reference to it. +* Decrement the cgroup rmdir_count so that rmdir's on an empty +* cgroup can block until the free_css_set_rcu callback */ list_for_each_entry_safe (link, saved_link, &cg->cg_links, cg_link_list) {struct cgroup *cgrp = link->cgrp; -list_del (&link->cg_link_list); -list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count) && -notify_on_release (cgrp)) { -if (taskexit) -set_bit (CGRP_RELEASABLE , &cgrp->flags); -check_for_release (cgrp); - } - -kfree (link); +if (taskexit) +set_bit (CGRP_RELEASABLE , &cgrp->flags); +atomic_dec (&cgrp->rmdir_count); +smp_mb (); }write_unlock (&css_set_lock);@@ -571,6 +611,8 @@ static void link_css_set(struct list_head *tmp_cg_links, cgrp_link_list); link->cg = cg; link->cgrp = cgrp; +atomic_inc (&cgrp->rmdir_count); +smp_mb ();/* make sure rmdir_count increments first */ atomic_inc (&cgrp->count);list_move (&link->cgrp_link_list, &cgrp->css_sets);/* @@ -725,9 +767,9 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task,* cgroup_attach_task(), which overwrites one tasks cgroup pointer with * another. It does so using cgroup_mutex, however there are * several performance critical places that need to reference -* task->cgroup without the expense of grabbing a system global +* task->cgroups without the expense of grabbing a system global * mutex. Therefore except as noted below, when dereferencing or, as -* in cgroup_attach_task(), modifying a task'ss cgroup pointer we use +* in cgroup_attach_task(), modifying a task's cgroups pointer we use * task_lock(), which acts on a spinlock (task->alloc_lock) already in * the task_struct routinely used for such matters. * @@ -909,33 +951,6 @@ static void cgroup_d_remove_dir(struct dentry *dentry) }/* -* A queue for waiters to do rmdir() cgroup. A tasks will sleep when -* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some -* reference to css->refcnt. In general, this refcnt is expected to goes down -* to zero, soon. -* -* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; -*/ -DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); - -static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) -{ -if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) -wake_up_all (&cgroup_rmdir_waitq); -} - -void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) -{ -css_get (css); -} - -void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) -{ -cgroup_wakeup_rmdir_waiter (css->cgroup); -css_put (css); -} - -/* * Call with cgroup_mutex held. Drops reference counts on modules, including * any duplicate ones that parse_cgroupfs_options took. If this function * returns an error, no reference counts are touched. @@ -1802,7 +1817,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) ss->attach (ss, cgrp, oldcgrp, tsk, false); }set_bit (CGRP_RELEASABLE , &oldcgrp->flags); -synchronize_rcu (); +/* put_css_set will not destroy cg until after an RCU grace period */ put_css_set (cg);/* @@ -3566,11 +3581,12 @@ static int cgroup_rmdir(struct inode *unused_dir, struct dentry *dentry)DEFINE_WAIT (wait);struct cgroup_event *event, *tmp;int ret; +unsigned long flags;/* the vfs holds both inode->i_mutex already */ again:mutex_lock (&cgroup_mutex); -if (atomic_read (&cgrp->count) !=0 ) { +if (atomic_read (&cgrp->rmdir_count) !=0 ) {mutex_unlock (&cgroup_mutex);return -EBUSY ; }@@ -3603,13 +3619,13 @@ again:mutex_lock (&cgroup_mutex); parent = cgrp->parent; -if (atomic_read (&cgrp->count) || !list_empty (&cgrp->children)) { +if (atomic_read (&cgrp->rmdir_count) || !list_empty (&cgrp->children)) {clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags);mutex_unlock (&cgroup_mutex);return -EBUSY ; }prepare_to_wait (&cgroup_rmdir_waitq, &wait,TASK_INTERRUPTIBLE ); -if (!cgroup_clear_css_refs (cgrp)) { +if (atomic_read (&cgrp->count) !=0 || !cgroup_clear_css_refs (cgrp)) {mutex_unlock (&cgroup_mutex);/* * Because someone may call cgroup_wakeup_rmdir_waiter() before @@ -3627,11 +3643,11 @@ again:finish_wait (&cgroup_rmdir_waitq, &wait);clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);set_bit (CGRP_REMOVED , &cgrp->flags);if (!list_empty (&cgrp->release_list))list_del (&cgrp->release_list); -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);cgroup_lock_hierarchy (cgrp->root);/* delete this cgroup from parent->children */ @@ -4389,6 +4405,8 @@ int cgroup_is_descendant(const struct cgroup *cgrp, struct task_struct *task)static void check_for_release (struct cgroup *cgrp) { +unsigned long flags; +/* All of these checks rely on RCU to keep the cgroup * structure alive */ if (cgroup_is_releasable(cgrp) && !atomic_read(&cgrp->count)@@ -4397,13 +4415,13 @@ static void check_for_release(struct cgroup *cgrp)* already queued for a userspace notification, queue * it now */ int need_schedule_work =0 ; -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);if (!cgroup_is_removed (cgrp) &&list_empty (&cgrp->release_list)) {list_add (&cgrp->release_list, &release_list); need_schedule_work =1 ; } -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);if (need_schedule_work)schedule_work (&release_agent_work); }@@ -4453,9 +4471,11 @@ EXPORT_SYMBOL_GPL(__css_put);*/ static void cgroup_release_agent (struct work_struct *work) { +unsigned long flags; +BUG_ON (work != &release_agent_work);mutex_lock (&cgroup_mutex); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags);while (!list_empty (&release_list)) {char *argv[3 ], *envp[3 ];int i;@@ -4464,7 +4484,7 @@ static void cgroup_release_agent(struct work_struct *work) struct cgroup, release_list);list_del_init (&cgrp->release_list); -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags); pathbuf =kmalloc (PAGE_SIZE ,GFP_KERNEL );if (!pathbuf) goto continue_free;@@ -4494,9 +4514,9 @@ static void cgroup_release_agent(struct work_struct *work) continue_free:
kernel/cgroup.c static void cgroup_release_agent(struct work_struct *work) · line 4522
Quoted 10 lines
kernel/cgroup.c +2
-2kfree (pathbuf);kfree (agentbuf); -spin_lock (&release_list_lock); +spin_lock_irqsave (&release_list_lock, flags); } -spin_unlock (&release_list_lock); +spin_unlock_irqrestore (&release_list_lock, flags);mutex_unlock (&cgroup_mutex); }
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Paul MenageQuoted 29 lines · Paul Menage
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.I definitely agree with the goal of using lighter-weight synchronization than the current
synchronize_rcu()call. However, there are definitely some subtleties to worry about in this code.One of the reasons originally for the current synchronization was to avoid the case of calling subsystem
destroy()callbacks while there could still be threads with RCU references to the subsystem state. The fact thatsynchronize_rcu()was called within acgroup_mutexcritical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.I suspect that when we moved a lot of the cgroup teardown code from
cgroup_rmdir()tocgroup_diput()(which also has asynchronize_rcu()call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
Quoted 6 lines
1) make
find_existing_css_setignorecss_setobjects with a zero refcount
2) change__put_css_setto be simplyif (
atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}
If we do this, it's not anymore safe to use get_css_set(), which just increments the refcount without checking if it's zero.
Quoted 17 lines
3) move the rest of
__put_css_setinto a delayed work struct that's scheduled byfree_css_set_rcu4) Get rid of the taskexit parameter - I think we can do that via a simple flag that indicates whether any task has ever been moved into the cgroup.
5) Put extra checks in
cgroup_rmdir()such that if it tries to remove a cgroup that has a non-zero refcount, it scans the cgroup'scss_setslist - if it finds only zero-refcount entries, then wait (viasynchronize_rcu()or some other appropriate means, maybe reusing theCGRP_WAIT_ON_RMDIRmechanism?) until thecss_setobjects have been fully cleaned up and the cgroup's refcounts have been released. Otherwise the operation of moving the last thread out of a cgroup and immediately deleting the cgroup would very likely fail with an EBUSY
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 33 lines · Li Zefan
Quoted 32 lines · Paul Menage
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.I definitely agree with the goal of using lighter-weight synchronization than the current
synchronize_rcu()call. However, there are definitely some subtleties to worry about in this code.One of the reasons originally for the current synchronization was to avoid the case of calling subsystem
destroy()callbacks while there could still be threads with RCU references to the subsystem state. The fact thatsynchronize_rcu()was called within acgroup_mutexcritical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.I suspect that when we moved a lot of the cgroup teardown code from
cgroup_rmdir()tocgroup_diput()(which also has asynchronize_rcu()call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
1) make
find_existing_css_setignorecss_setobjects with a zero refcount
2) change__put_css_setto be simply
Quoted 6 lines
Quoted 3 lines
if (
atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}If we do this, it's not anymore safe to use
get_css_set(), which just increments the refcount without checking if it's zero.
I used an alternate approach, removing the css_set from the hash table in put_css_set, but delaying the deletion to free_css_set_rcu. That way, nothing can get another reference to the css_set to call get_css_set on.
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to a message not in the archiveQuoted 3 lines · Li Zefan
If we do this, it's not anymore safe to use
get_css_set(), which just increments the refcount without checking if it's zero.
I don't believe that it's currently safe to use get_css_set() on a zero-refcount css_set anyway - a css_set starts with refcount 1 and can be freed the moment that its refcount reaches 0. You can only use get_css_set() on a stable reference to an existing css_set, which by definition has a refcount of at least 1. Were you thinking of any particular existing uses?
Paul
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Li ZefanQuoted 33 lines · Li Zefan
Quoted 32 lines · Paul Menage
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.I definitely agree with the goal of using lighter-weight synchronization than the current
synchronize_rcu()call. However, there are definitely some subtleties to worry about in this code.One of the reasons originally for the current synchronization was to avoid the case of calling subsystem
destroy()callbacks while there could still be threads with RCU references to the subsystem state. The fact thatsynchronize_rcu()was called within acgroup_mutexcritical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.I suspect that when we moved a lot of the cgroup teardown code from
cgroup_rmdir()tocgroup_diput()(which also has asynchronize_rcu()call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
1) make
find_existing_css_setignorecss_setobjects with a zero refcount
2) change__put_css_setto be simply
Quoted 6 lines
Quoted 3 lines
if (
atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}If we do this, it's not anymore safe to use
get_css_set(), which just increments the refcount without checking if it's zero.
I used an alternate approach, removing the css_set from the hash table in put_css_set, but delaying the deletion to free_css_set_rcu. That way, nothing can get another reference to the css_set to call get_css_set on.
A message that is not in the archive (sent off-list or not yet received): AANLkTi=6nwDCdzDz7E2EaAw2pf3KUVjmKMRqGfz5zVhP-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
PATCH 1/2 cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Changes the meaning of CGRP_RELEASABLE to be set on any cgroup that has ever had a task or cgroup in it, or had css_get called on it. The bit is set in cgroup_attach_task, cgroup_create, and __css_get. It is not necessary to set the bit in cgroup_fork, as the task is either in the root cgroup, in which can never be released, or the task it was forked from already set the bit in croup_attach_task.
--- include/linux/cgroup.h | 12 +-------- kernel/cgroup.c | 54 ++++++++++++++++++++--------------------------- 2 files changed, 25 insertions(+), 41 deletions(-)
include/linux/cgroup.h +2 -10
@@ -84,12 +84,6 @@ enum { CSS_REMOVED, /* This CSS is dead */ }; -/* Caller must verify that the css is not for root cgroup */ -static inline void __css_get (struct cgroup_subsys_state *css,int count) -{ -atomic_add (count, &css->refcnt); -} -/* * Call css_get() to hold a reference on the css; it can be used * for a reference obtained via:
@@ -97,6 +91,7 @@ static inline void __css_get(struct cgroup_subsys_state *css, int count)* - task->cgroups for a locked task */ +extern void __css_get (struct cgroup_subsys_state *css,int count);static inline void css_get (struct cgroup_subsys_state *css) {/* We don't need to reference count the root state */
@@ -143,10 +138,7 @@ static inline void css_put(struct cgroup_subsys_state *css)enum {/* Control Group is dead */ CGRP_REMOVED , -/* -* Control Group has previously had a child cgroup or a task, -* but no longer (only if CGRP_NOTIFY_ON_RELEASE is set) -*/ +/* Control Group has ever had a child cgroup or a task */ CGRP_RELEASABLE ,/* Control Group requires release notifications to userspace */ CGRP_NOTIFY_ON_RELEASE ,
kernel/cgroup.c +23 -31
109 lines not on this page · raw message
Signature · 1 line
1.7.3.1
PATCH 2/2 cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to a message not in the archivesynchronize_rcu can be very expensive, averaging 100 ms in some cases. In cgroup_attach_task, it is used to prevent a task->cgroups pointer dereferenced in an RCU read side critical section from being invalidated, by delaying the call to put_css_set until after an RCU grace period.
To avoid the call to synchronize_rcu, make the put_css_set call rcu-safe by moving the deletion of the css_set links into free_css_set_work, scheduled by the rcu callback free_css_set_rcu.
The decrement of the cgroup refcount is no longer synchronous with the call to put_css_set, which can result in the cgroup refcount staying positive after the last call to cgroup_attach_task returns. To allow the cgroup to be deleted with cgroup_rmdir synchronously after cgroup_attach_task, have rmdir check the refcount of all associated css_sets. If cgroup_rmdir is called on a cgroup for which the css_sets all have refcount zero but the cgroup refcount is nonzero, reuse the rmdir waitqueue to block the rmdir until free_css_set_work is called.
--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 120 +++++++++++++++++++++++++++++------------------- 2 files changed, 74 insertions(+), 47 deletions(-)
include/linux/cgroup.h +1 -0
@@ -279,6 +279,7 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; +struct work_struct work; };/*
kernel/cgroup.c +73 -47
187 lines not on this page · raw message
Signature · 1 line
1.7.3.1
PATCH 1/2 cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin CrossChanges the meaning of CGRP_RELEASABLE to be set on any cgroup that has ever had a task or cgroup in it, or had css_get called on it. The bit is set in cgroup_attach_task, cgroup_create, and __css_get. It is not necessary to set the bit in cgroup_fork, as the task is either in the root cgroup, in which can never be released, or the task it was forked from already set the bit in croup_attach_task.
--- include/linux/cgroup.h | 12 +-------- kernel/cgroup.c | 54 ++++++++++++++++++++--------------------------- 2 files changed, 25 insertions(+), 41 deletions(-)
include/linux/cgroup.h +2 -10
@@ -84,12 +84,6 @@ enum { CSS_REMOVED, /* This CSS is dead */ }; -/* Caller must verify that the css is not for root cgroup */ -static inline void __css_get (struct cgroup_subsys_state *css,int count) -{ -atomic_add (count, &css->refcnt); -} -/* * Call css_get() to hold a reference on the css; it can be used * for a reference obtained via:
@@ -97,6 +91,7 @@ static inline void __css_get(struct cgroup_subsys_state *css, int count)* - task->cgroups for a locked task */ +extern void __css_get (struct cgroup_subsys_state *css,int count);static inline void css_get (struct cgroup_subsys_state *css) {/* We don't need to reference count the root state */
@@ -143,10 +138,7 @@ static inline void css_put(struct cgroup_subsys_state *css)enum {/* Control Group is dead */ CGRP_REMOVED , -/* -* Control Group has previously had a child cgroup or a task, -* but no longer (only if CGRP_NOTIFY_ON_RELEASE is set) -*/ +/* Control Group has ever had a child cgroup or a task */ CGRP_RELEASABLE ,/* Control Group requires release notifications to userspace */ CGRP_NOTIFY_ON_RELEASE ,
kernel/cgroup.c +23 -31
109 lines not on this page · raw message
Signature · 1 line
1.7.3.1
A message that is not in the archive (sent off-list or not yet received): 1290577024-12347-1-git-send-email-ccross-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
kernel/cgroup.c static void __put_css_set(struct css_set *cg, int taskexit) · line 372
Quoted 11 lines · Colin Cross
kernel/cgroup.c +1
-5@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit) struct cgroup *cgrp = link->cgrp; list_del(&link->cg_link_list); list_del(&link->cgrp_link_list); - if (atomic_dec_and_test(&cgrp->count) && - notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); + if (atomic_dec_and_test(&cgrp->count)) check_for_release(cgrp); - }
We seem to have lost some notify_on_release() checks - maybe move that to check_for_release()?
kernel/cgroup.c static void check_for_release(struct cgroup *cgrp) · line 4404
Quoted 6 lines
kernel/cgroup.c +5
-0/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ + atomic_add(count, &css->refcnt); + set_bit(CGRP_RELEASABLE, &css->cgroup->flags); +}
Is css_get() the right place to be putting this? It's not clear to me why a subsystem taking a refcount on a cgroup's state should render it releasable when it drops that refcount.
Should we maybe clear the CGRP_RELEASABLE flag right before doing the userspace callback?
Paul
Bryan Huntsman
Tested-by Mike BohanRe: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to a message not in the archiveQuoted 163 lines · Colin Cross
Changes the meaning of
CGRP_RELEASABLEto be set on any cgroup that has ever had a task or cgroup in it, or hadcss_getcalled on it. The bit is set incgroup_attach_task,cgroup_create, and__css_get. It is not necessary to set the bit incgroup_fork, as the task is either in the root cgroup, in which can never be released, or the task it was forked from already set the bit incroup_attach_task.Signed-off-by Colin Cross <[email protected]>--- include/linux/cgroup.h | 12 +-------- kernel/cgroup.c | 54 ++++++++++++++++++++--------------------------- 2 files changed, 25 insertions(+), 41 deletions(-)include/linux/cgroup.h +2
-10@@ -84,12 +84,6 @@ enum { CSS_REMOVED, /* This CSS is dead */ }; -/* Caller must verify that the css is not for root cgroup */ -static inline void __css_get (struct cgroup_subsys_state *css,int count) -{ -atomic_add (count, &css->refcnt); -} -/* * Call css_get() to hold a reference on the css; it can be used * for a reference obtained via: @@ -97,6 +91,7 @@ static inline void __css_get(struct cgroup_subsys_state *css, int count)* - task->cgroups for a locked task */ +extern void __css_get (struct cgroup_subsys_state *css,int count);static inline void css_get (struct cgroup_subsys_state *css) {/* We don't need to reference count the root state */ @@ -143,10 +138,7 @@ static inline void css_put(struct cgroup_subsys_state *css)enum {/* Control Group is dead */ CGRP_REMOVED , -/* -* Control Group has previously had a child cgroup or a task, -* but no longer (only if CGRP_NOTIFY_ON_RELEASE is set) -*/ +/* Control Group has ever had a child cgroup or a task */ CGRP_RELEASABLE ,/* Control Group requires release notifications to userspace */ CGRP_NOTIFY_ON_RELEASE ,kernel/cgroup.c +22
-27@@ -338,7 +338,15 @@ static void free_css_set_rcu(struct rcu_head *obj)* compiled into their kernel but not actually in use */ static int use_task_css_set_links__read_mostly ; -static void __put_css_set (struct css_set *cg,int taskexit) +/* +* refcounted get/put for css_set objects +*/ +static inline void get_css_set (struct css_set *cg) +{ +atomic_inc (&cg->refcount); +} + +static void put_css_set (struct css_set *cg) {struct cg_cgroup_link *link;struct cg_cgroup_link *saved_link;@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit)struct cgroup *cgrp = link->cgrp;list_del (&link->cg_link_list);list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count) && -notify_on_release (cgrp)) { -if (taskexit) -set_bit (CGRP_RELEASABLE , &cgrp->flags); +if (atomic_dec_and_test (&cgrp->count))check_for_release (cgrp); - }kfree (link); }@@ -379,24 +383,6 @@ static void __put_css_set(struct css_set *cg, int taskexit) }/* -* refcounted get/put for css_set objects -*/ -static inline void get_css_set (struct css_set *cg) -{ -atomic_inc (&cg->refcount); -} - -static inline void put_css_set (struct css_set *cg) -{ -__put_css_set (cg,0 ); -} - -static inline void put_css_set_taskexit (struct css_set *cg) -{ -__put_css_set (cg,1 ); -} - -/* * compare_css_sets - helper function for find_existing_css_set(). * @cg: candidate css_set being tested * @old_cg: existing css_set for a task @@ -1801,7 +1787,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)if (ss->attach) ss->attach (ss, cgrp, oldcgrp, tsk, false); } -set_bit (CGRP_RELEASABLE , &oldcgrp->flags); +set_bit (CGRP_RELEASABLE , &cgrp->flags);synchronize_rcu ();put_css_set (cg);@@ -3427,6 +3413,8 @@ static long cgroup_create(struct cgroup *parent, struct dentry *dentry,if (err <0 ) goto err_remove; +set_bit (CGRP_RELEASABLE , &parent->flags); +/* The cgroup directory was pre-locked for us */ BUG_ON (!mutex_is_locked (&cgrp->dentry->d_inode->i_mutex));@@ -3645,7 +3633,6 @@ again:cgroup_d_remove_dir (d);dput (d); -set_bit (CGRP_RELEASABLE , &parent->flags);check_for_release (parent);/* @@ -4240,7 +4227,7 @@ void cgroup_exit(struct task_struct *tsk, int run_callbacks) tsk->cgroups = &init_css_set;task_unlock (tsk);if (cg) -put_css_set_taskexit (cg); +put_css_set (cg); }/** @@ -4410,6 +4397,14 @@ static void check_for_release(struct cgroup *cgrp) }/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ +atomic_add (count, &css->refcnt); +set_bit (CGRP_RELEASABLE , &css->cgroup->flags); +} +EXPORT_SYMBOL_GPL (__css_get); + +/* Caller must verify that the css is not for root cgroup */ void __css_put (struct cgroup_subsys_state *css,int count) {struct cgroup *cgrp = css->cgroup;@@ -4417,10 +4412,7 @@ void __css_put(struct cgroup_subsys_state *css, int count)rcu_read_lock ();
kernel/cgroup.c void __css_put(struct cgroup_subsys_state *css, int count) · line 4418
Quoted 10 lines
kernel/cgroup.c +1
-4val =atomic_sub_return (count, &css->refcnt);if (val ==1 ) { -if (notify_on_release (cgrp)) { -set_bit (CGRP_RELEASABLE , &cgrp->flags); -check_for_release (cgrp); - } +check_for_release (cgrp);cgroup_wakeup_rmdir_waiter (cgrp); }rcu_read_unlock ();
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin Crosskernel/cgroup.c static void __put_css_set(struct css_set *cg, int taskexit) · line 372
Quoted 11 lines · Colin Cross
kernel/cgroup.c +1
-5@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit) struct cgroup *cgrp = link->cgrp; list_del(&link->cg_link_list); list_del(&link->cgrp_link_list); - if (atomic_dec_and_test(&cgrp->count) && - notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); + if (atomic_dec_and_test(&cgrp->count)) check_for_release(cgrp); - }
We seem to have lost some notify_on_release() checks - maybe move that to check_for_release()?
kernel/cgroup.c static void check_for_release(struct cgroup *cgrp) · line 4404
Quoted 6 lines
kernel/cgroup.c +5
-0/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ + atomic_add(count, &css->refcnt); + set_bit(CGRP_RELEASABLE, &css->cgroup->flags); +}
Is css_get() the right place to be putting this? It's not clear to me why a subsystem taking a refcount on a cgroup's state should render it releasable when it drops that refcount.
Should we maybe clear the CGRP_RELEASABLE flag right before doing the userspace callback?
Paul
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 15 lines · Paul Menage
kernel/cgroup.c static void __put_css_set(struct css_set *cg, int taskexit) · line 372
Quoted 11 lines · Colin Cross
kernel/cgroup.c +1
-5@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit) struct cgroup *cgrp = link->cgrp; list_del(&link->cg_link_list); list_del(&link->cgrp_link_list); - if (atomic_dec_and_test(&cgrp->count) && - notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); + if (atomic_dec_and_test(&cgrp->count)) check_for_release(cgrp); - }We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_release immediately calls cgroup_is_releasable, which checks for the same bit as notify_on_release. There's no need for CGRP_RELEASABLE to depend on notify_on_release, or to check notify_on_release before calling check_for_release.
Quoted 10 lines
kernel/cgroup.c static void check_for_release(struct cgroup *cgrp) · line 4404
Quoted 6 lines
kernel/cgroup.c +5
-0/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ + atomic_add(count, &css->refcnt); + set_bit(CGRP_RELEASABLE, &css->cgroup->flags); +}Is
css_get()the right place to be putting this? It's not clear to me why a subsystem taking a refcount on a cgroup's state should render it releasable when it drops that refcount.
I matched the existing behavior, __css_put sets CGRP_RELEASABLE when refcnt goes to 0.
Quoted 2 lines
Should we maybe clear the
CGRP_RELEASABLEflag right before doing the userspace callback?
Actually, I think CGRP_RELEASABLE can be dropped entirely. check_for_release is only called from __css_put, cgroup_rmdir, and __put_css_set (or free_css_set_work after my second patch). Those all imply that __css_get, get_css_set, or cgroup_create have been previously called, which are the functions that set CGRP_RELEASABLE.
A message that is not in the archive (sent off-list or not yet received): AANLkTimwvP2Ey1gJ6AbbFNtDKjGZt4cwqL=08nGBa_PT-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 34 lines · Colin Cross
Quoted 15 lines · Paul Menage
kernel/cgroup.c static void __put_css_set(struct css_set *cg, int taskexit) · line 372
Quoted 11 lines · Colin Cross
kernel/cgroup.c +1
-5@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit) struct cgroup *cgrp = link->cgrp; list_del(&link->cg_link_list); list_del(&link->cgrp_link_list); - if (atomic_dec_and_test(&cgrp->count) && - notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); + if (atomic_dec_and_test(&cgrp->count)) check_for_release(cgrp); - }We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.Quoted 10 lines
kernel/cgroup.c static void check_for_release(struct cgroup *cgrp) · line 4404
Quoted 6 lines
kernel/cgroup.c +5
-0/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ + atomic_add(count, &css->refcnt); + set_bit(CGRP_RELEASABLE, &css->cgroup->flags); +}Is
css_get()the right place to be putting this? It's not clear to me why a subsystem taking a refcount on a cgroup's state should render it releasable when it drops that refcount.I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.
Quoted 7 lines
Quoted 2 lines
Should we maybe clear the
CGRP_RELEASABLEflag right before doing the userspace callback?Actually, I think
CGRP_RELEASABLEcan be dropped entirely.check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Nevermind, that's not true - get_css_set does not set CGRP_RELEASABLE, cgroup_attach_task does.
If CGRP_RELEASABLE is not cleared before the callback, the release_agent would be run once when the last task was removed from the cgroup, and then again if a task failed to be added to the empty cgroup because the task was exiting, so clearing the flag sounds like a good idea.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to a message not in the archiveQuoted 7 lines · Colin Cross
Quoted 3 lines
We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.
OK.
Quoted 3 lines
I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.
Ah, we do appear to have had that behaviour for a while. I don't remember the justification for it at this point :-)
Quoted 4 lines
check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call put_css_set() on the new css_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.
Paul
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin CrossQuoted 34 lines · Colin Cross
Quoted 15 lines · Paul Menage
kernel/cgroup.c static void __put_css_set(struct css_set *cg, int taskexit) · line 372
Quoted 11 lines · Colin Cross
kernel/cgroup.c +1
-5@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit) struct cgroup *cgrp = link->cgrp; list_del(&link->cg_link_list); list_del(&link->cgrp_link_list); - if (atomic_dec_and_test(&cgrp->count) && - notify_on_release(cgrp)) { - if (taskexit) - set_bit(CGRP_RELEASABLE, &cgrp->flags); + if (atomic_dec_and_test(&cgrp->count)) check_for_release(cgrp); - }We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.Quoted 10 lines
kernel/cgroup.c static void check_for_release(struct cgroup *cgrp) · line 4404
Quoted 6 lines
kernel/cgroup.c +5
-0/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ + atomic_add(count, &css->refcnt); + set_bit(CGRP_RELEASABLE, &css->cgroup->flags); +}Is
css_get()the right place to be putting this? It's not clear to me why a subsystem taking a refcount on a cgroup's state should render it releasable when it drops that refcount.I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.
Quoted 7 lines
Quoted 2 lines
Should we maybe clear the
CGRP_RELEASABLEflag right before doing the userspace callback?Actually, I think
CGRP_RELEASABLEcan be dropped entirely.check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Nevermind, that's not true - get_css_set does not set CGRP_RELEASABLE, cgroup_attach_task does.
If CGRP_RELEASABLE is not cleared before the callback, the release_agent would be run once when the last task was removed from the cgroup, and then again if a task failed to be added to the empty cgroup because the task was exiting, so clearing the flag sounds like a good idea.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin CrossQuoted 7 lines · Colin Cross
Quoted 3 lines
We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.
OK.
Quoted 3 lines
I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.
Ah, we do appear to have had that behaviour for a while. I don't remember the justification for it at this point :-)
Quoted 4 lines
check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call put_css_set() on the new css_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.
Paul
A message that is not in the archive (sent off-list or not yet received): AANLkTimJA52-GTM=AzS+tkOugrsi6Keh0_j87vK1BkGv-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 23 lines · Paul Menage
Quoted 7 lines · Colin Cross
Quoted 3 lines
We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.OK.
Quoted 3 lines
I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.Ah, we do appear to have had that behaviour for a while. I don't remember the justification for it at this point :-)
Quoted 4 lines
check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.
Clearing the CGRP_RELEASABLE bit any time after the tests in check_for_release introduces a race if __css_get is called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in __css_get, I don't see any way to safely clear CGRP_RELEASABLE.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Paul MenageQuoted 23 lines · Paul Menage
Quoted 7 lines · Colin Cross
Quoted 3 lines
We seem to have lost some
notify_on_release()checks - maybe move that tocheck_for_release()?
check_for_releaseimmediately callscgroup_is_releasable, which checks for the same bit asnotify_on_release. There's no need forCGRP_RELEASABLEto depend onnotify_on_release, or to checknotify_on_releasebefore callingcheck_for_release.OK.
Quoted 3 lines
I matched the existing behavior,
__css_putsetsCGRP_RELEASABLEwhen refcnt goes to 0.Ah, we do appear to have had that behaviour for a while. I don't remember the justification for it at this point :-)
Quoted 4 lines
check_for_releaseis only called from__css_put,cgroup_rmdir, and__put_css_set(orfree_css_set_workafter my second patch). Those all imply that__css_get,get_css_set, orcgroup_createhave been previously called, which are the functions that setCGRP_RELEASABLE.
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.
Clearing the CGRP_RELEASABLE bit any time after the tests in check_for_release introduces a race if __css_get is called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in __css_get, I don't see any way to safely clear CGRP_RELEASABLE.
A message that is not in the archive (sent off-list or not yet received): AANLkTim67fLN+PYz-P0TM0QRmvQKP80tyXSNKNSZhFZ2-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 11 lines · Colin Cross
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.Clearing the
CGRP_RELEASABLEbit any time after the tests incheck_for_releaseintroduces a race if__css_getis called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in__css_get, I don't see any way to safely clearCGRP_RELEASABLE.
I don't quite follow your argument here. Are you saying that the problem is that you could end up spawning a release agent for a cgroup that was no longer releasable since it now had a process in it again? If so, then I don't think that's a problem - spurious release agent invocations for non-empty cgroups will always happen occasionally due to races between the kernel and userspace. But a failed move of a task into a previously-empty cgroup shouldn't trigger the agent.
Paul
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin CrossQuoted 11 lines · Colin Cross
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.Clearing the
CGRP_RELEASABLEbit any time after the tests incheck_for_releaseintroduces a race if__css_getis called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in__css_get, I don't see any way to safely clearCGRP_RELEASABLE.
I don't quite follow your argument here. Are you saying that the problem is that you could end up spawning a release agent for a cgroup that was no longer releasable since it now had a process in it again? If so, then I don't think that's a problem - spurious release agent invocations for non-empty cgroups will always happen occasionally due to races between the kernel and userspace. But a failed move of a task into a previously-empty cgroup shouldn't trigger the agent.
Paul
A message that is not in the archive (sent off-list or not yet received): AANLkTinZarXbEyb1xfJWjG4gN2qhTVTXTdso4Cym5M9T-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 13 lines · Paul Menage
Quoted 11 lines · Colin Cross
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.Clearing the
CGRP_RELEASABLEbit any time after the tests incheck_for_releaseintroduces a race if__css_getis called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in__css_get, I don't see any way to safely clearCGRP_RELEASABLE.
Quoted 7 lines
I don't quite follow your argument here. Are you saying that the problem is that you could end up spawning a release agent for a cgroup that was no longer releasable since it now had a process in it again? If so, then I don't think that's a problem - spurious release agent invocations for non-empty cgroups will always happen occasionally due to races between the kernel and userspace. But a failed move of a task into a previously-empty cgroup shouldn't trigger the agent.
No, if you add a new process to the group while check_for_release, the bit could get set by the add for the new process, then cleared by the concurrently running check_for_release. The release agent would be spawned with a process in the group, which is fine, but when RELEASABLE bit would be clear. When the new process was removed, check_for_release would not call the release agent at all.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Paul MenageQuoted 13 lines · Paul Menage
Quoted 11 lines · Colin Cross
Quoted 5 lines
Not in one case - if we create a new cgroup and try to move a thread into it, but the thread is exiting as we move it, we'll call
put_css_set()on the newcss_set, which will drop the refcount on the target cgroup back to 0. We wouldn't want the auto-release notification to kick in in that situation, I think.Clearing the
CGRP_RELEASABLEbit any time after the tests incheck_for_releaseintroduces a race if__css_getis called between the check and clearing the bit - the cgroup will have an entry, but the bit will not be set. Without additional locking in__css_get, I don't see any way to safely clearCGRP_RELEASABLE.
Quoted 7 lines
I don't quite follow your argument here. Are you saying that the problem is that you could end up spawning a release agent for a cgroup that was no longer releasable since it now had a process in it again? If so, then I don't think that's a problem - spurious release agent invocations for non-empty cgroups will always happen occasionally due to races between the kernel and userspace. But a failed move of a task into a previously-empty cgroup shouldn't trigger the agent.
No, if you add a new process to the group while check_for_release, the bit could get set by the add for the new process, then cleared by the concurrently running check_for_release. The release agent would be spawned with a process in the group, which is fine, but when RELEASABLE bit would be clear. When the new process was removed, check_for_release would not call the release agent at all.
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Colin CrossQuoted 163 lines · Colin Cross
Changes the meaning of
CGRP_RELEASABLEto be set on any cgroup that has ever had a task or cgroup in it, or hadcss_getcalled on it. The bit is set incgroup_attach_task,cgroup_create, and__css_get. It is not necessary to set the bit incgroup_fork, as the task is either in the root cgroup, in which can never be released, or the task it was forked from already set the bit incroup_attach_task.Signed-off-by Colin Cross <[email protected]>--- include/linux/cgroup.h | 12 +-------- kernel/cgroup.c | 54 ++++++++++++++++++++--------------------------- 2 files changed, 25 insertions(+), 41 deletions(-)include/linux/cgroup.h +2
-10@@ -84,12 +84,6 @@ enum { CSS_REMOVED, /* This CSS is dead */ }; -/* Caller must verify that the css is not for root cgroup */ -static inline void __css_get (struct cgroup_subsys_state *css,int count) -{ -atomic_add (count, &css->refcnt); -} -/* * Call css_get() to hold a reference on the css; it can be used * for a reference obtained via: @@ -97,6 +91,7 @@ static inline void __css_get(struct cgroup_subsys_state *css, int count)* - task->cgroups for a locked task */ +extern void __css_get (struct cgroup_subsys_state *css,int count);static inline void css_get (struct cgroup_subsys_state *css) {/* We don't need to reference count the root state */ @@ -143,10 +138,7 @@ static inline void css_put(struct cgroup_subsys_state *css)enum {/* Control Group is dead */ CGRP_REMOVED , -/* -* Control Group has previously had a child cgroup or a task, -* but no longer (only if CGRP_NOTIFY_ON_RELEASE is set) -*/ +/* Control Group has ever had a child cgroup or a task */ CGRP_RELEASABLE ,/* Control Group requires release notifications to userspace */ CGRP_NOTIFY_ON_RELEASE ,kernel/cgroup.c +22
-27@@ -338,7 +338,15 @@ static void free_css_set_rcu(struct rcu_head *obj)* compiled into their kernel but not actually in use */ static int use_task_css_set_links__read_mostly ; -static void __put_css_set (struct css_set *cg,int taskexit) +/* +* refcounted get/put for css_set objects +*/ +static inline void get_css_set (struct css_set *cg) +{ +atomic_inc (&cg->refcount); +} + +static void put_css_set (struct css_set *cg) {struct cg_cgroup_link *link;struct cg_cgroup_link *saved_link;@@ -364,12 +372,8 @@ static void __put_css_set(struct css_set *cg, int taskexit)struct cgroup *cgrp = link->cgrp;list_del (&link->cg_link_list);list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count) && -notify_on_release (cgrp)) { -if (taskexit) -set_bit (CGRP_RELEASABLE , &cgrp->flags); +if (atomic_dec_and_test (&cgrp->count))check_for_release (cgrp); - }kfree (link); }@@ -379,24 +383,6 @@ static void __put_css_set(struct css_set *cg, int taskexit) }/* -* refcounted get/put for css_set objects -*/ -static inline void get_css_set (struct css_set *cg) -{ -atomic_inc (&cg->refcount); -} - -static inline void put_css_set (struct css_set *cg) -{ -__put_css_set (cg,0 ); -} - -static inline void put_css_set_taskexit (struct css_set *cg) -{ -__put_css_set (cg,1 ); -} - -/* * compare_css_sets - helper function for find_existing_css_set(). * @cg: candidate css_set being tested * @old_cg: existing css_set for a task @@ -1801,7 +1787,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)if (ss->attach) ss->attach (ss, cgrp, oldcgrp, tsk, false); } -set_bit (CGRP_RELEASABLE , &oldcgrp->flags); +set_bit (CGRP_RELEASABLE , &cgrp->flags);synchronize_rcu ();put_css_set (cg);@@ -3427,6 +3413,8 @@ static long cgroup_create(struct cgroup *parent, struct dentry *dentry,if (err <0 ) goto err_remove; +set_bit (CGRP_RELEASABLE , &parent->flags); +/* The cgroup directory was pre-locked for us */ BUG_ON (!mutex_is_locked (&cgrp->dentry->d_inode->i_mutex));@@ -3645,7 +3633,6 @@ again:cgroup_d_remove_dir (d);dput (d); -set_bit (CGRP_RELEASABLE , &parent->flags);check_for_release (parent);/* @@ -4240,7 +4227,7 @@ void cgroup_exit(struct task_struct *tsk, int run_callbacks) tsk->cgroups = &init_css_set;task_unlock (tsk);if (cg) -put_css_set_taskexit (cg); +put_css_set (cg); }/** @@ -4410,6 +4397,14 @@ static void check_for_release(struct cgroup *cgrp) }/* Caller must verify that the css is not for root cgroup */ +void __css_get (struct cgroup_subsys_state *css,int count) +{ +atomic_add (count, &css->refcnt); +set_bit (CGRP_RELEASABLE , &css->cgroup->flags); +} +EXPORT_SYMBOL_GPL (__css_get); + +/* Caller must verify that the css is not for root cgroup */ void __css_put (struct cgroup_subsys_state *css,int count) {struct cgroup *cgrp = css->cgroup;@@ -4417,10 +4412,7 @@ void __css_put(struct cgroup_subsys_state *css, int count)rcu_read_lock ();
kernel/cgroup.c void __css_put(struct cgroup_subsys_state *css, int count) · line 4418
Quoted 10 lines
kernel/cgroup.c +1
-4val =atomic_sub_return (count, &css->refcnt);if (val ==1 ) { -if (notify_on_release (cgrp)) { -set_bit (CGRP_RELEASABLE , &cgrp->flags); -check_for_release (cgrp); - } +check_for_release (cgrp);cgroup_wakeup_rmdir_waiter (cgrp); }rcu_read_unlock ();
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 8 lines · Bryan Huntsman
Tested-by Mike Bohan <[email protected]>I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
Is this attached to the wrong patch? I'd thought that it was the other patch (removing the rcu_synchronize()) that's the performance booster. This one is more about preserving the semantics of the notification API.
Paul
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Bryan HuntsmanQuoted 8 lines · Bryan Huntsman
Tested-by Mike Bohan <[email protected]>I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
Is this attached to the wrong patch? I'd thought that it was the other patch (removing the rcu_synchronize()) that's the performance booster. This one is more about preserving the semantics of the notification API.
Paul
A message that is not in the archive (sent off-list or not yet received): AANLkTin7B51maXHRH+FNmZ14bmWmEp9P2=2QTNqgq_Fi-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
Quoted 10 lines · Paul Menage
Quoted 8 lines · Bryan Huntsman
Tested-by Mike Bohan<[email protected]>I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
Quoted 4 lines
Is this attached to the wrong patch? I'd thought that it was the other patch (removing the
rcu_synchronize()) that's the performance booster. This one is more about preserving the semantics of the notification API.
You are correct. "[PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task" improved the performance.
To be more correct, I tested this patch (eg. "cgroup: Set CGRP_RELEASABLE when adding to a cgroup") to the degree that it didn't appear to cause any stability or functional regressions when performing the simple benchmark procedure described above. I did also test "[PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task" independently of this patch to verify that it alone improved the performance.
Thanks,
Mike
Signature · 2 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
Re: [PATCH 1/2] cgroup: Set CGRP_RELEASABLE when adding to a cgroup
In reply to Paul MenageQuoted 10 lines · Paul Menage
Quoted 8 lines · Bryan Huntsman
Tested-by Mike Bohan<[email protected]>I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
Quoted 4 lines
Is this attached to the wrong patch? I'd thought that it was the other patch (removing the
rcu_synchronize()) that's the performance booster. This one is more about preserving the semantics of the notification API.
You are correct. "[PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task" improved the performance.
To be more correct, I tested this patch (eg. "cgroup: Set CGRP_RELEASABLE when adding to a cgroup") to the degree that it didn't appear to cause any stability or functional regressions when performing the simple benchmark procedure described above. I did also test "[PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task" independently of this patch to verify that it alone improved the performance.
Thanks,
Mike
Signature · 2 lines
Employee of Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum
PATCH 2/2 cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Colin Crosssynchronize_rcu can be very expensive, averaging 100 ms in some cases. In cgroup_attach_task, it is used to prevent a task->cgroups pointer dereferenced in an RCU read side critical section from being invalidated, by delaying the call to put_css_set until after an RCU grace period.
To avoid the call to synchronize_rcu, make the put_css_set call rcu-safe by moving the deletion of the css_set links into free_css_set_work, scheduled by the rcu callback free_css_set_rcu.
The decrement of the cgroup refcount is no longer synchronous with the call to put_css_set, which can result in the cgroup refcount staying positive after the last call to cgroup_attach_task returns. To allow the cgroup to be deleted with cgroup_rmdir synchronously after cgroup_attach_task, have rmdir check the refcount of all associated css_sets. If cgroup_rmdir is called on a cgroup for which the css_sets all have refcount zero but the cgroup refcount is nonzero, reuse the rmdir waitqueue to block the rmdir until free_css_set_work is called.
--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 120 +++++++++++++++++++++++++++++------------------- 2 files changed, 74 insertions(+), 47 deletions(-)
include/linux/cgroup.h +1 -0
@@ -279,6 +279,7 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; +struct work_struct work; };/*
kernel/cgroup.c +73 -47
187 lines not on this page · raw message
Signature · 1 line
1.7.3.1
A message that is not in the archive (sent off-list or not yet received): 1290577024-12347-2-git-send-email-ccross-z5hGa2qSFaRBDgjK7y7TUQ@public.gmane.org
Bryan Huntsman
Tested-by Mike BohanRe: [PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
Quoted 233 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated, by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks intofree_css_set_work, scheduled by the rcu callbackfree_css_set_rcu.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, have rmdir check the refcount of all associatedcss_sets. Ifcgroup_rmdiris called on a cgroup for which thecss_setsall have refcount zero but the cgroup refcount is nonzero, reuse the rmdir waitqueue to block the rmdir untilfree_css_set_workis called.Signed-off-by Colin Cross <[email protected]>--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 120 +++++++++++++++++++++++++++++------------------- 2 files changed, 74 insertions(+), 47 deletions(-)include/linux/cgroup.h +1
-0@@ -279,6 +279,7 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; +struct work_struct work; };/* kernel/cgroup.c +72
-46@@ -267,6 +267,33 @@ static void cgroup_release_agent(struct work_struct *work); static DECLARE_WORK(release_agent_work, cgroup_release_agent);static void check_for_release (struct cgroup *cgrp); +/* +* A queue for waiters to do rmdir() cgroup. A tasks will sleep when +* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some +* reference to css->refcnt. In general, this refcnt is expected to goes down +* to zero, soon. +* +* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; +*/ +DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); + +static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) +{ +if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) +wake_up_all (&cgroup_rmdir_waitq); +} + +void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) +{ +css_get (css); +} + +void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) +{ +cgroup_wakeup_rmdir_waiter (css->cgroup); +css_put (css); +} +/* Link structure for associating css_set objects with cgroups */ struct cg_cgroup_link {/* @@ -326,10 +353,35 @@ static struct hlist_head *css_set_hash(struct cgroup_subsys_state *css[])return &css_set_table[index]; } +static void free_css_set_work (struct work_struct *work) +{ + struct css_set *cg = container_of(work, struct css_set, work); +struct cg_cgroup_link *link; +struct cg_cgroup_link *saved_link; + +write_lock (&css_set_lock); +list_for_each_entry_safe (link, saved_link, &cg->cg_links, + cg_link_list) { +struct cgroup *cgrp = link->cgrp; +list_del (&link->cg_link_list); +list_del (&link->cgrp_link_list); +if (atomic_dec_and_test (&cgrp->count)) { +check_for_release (cgrp); +cgroup_wakeup_rmdir_waiter (cgrp); + } +kfree (link); + } +write_unlock (&css_set_lock); + +kfree (cg); +} +static void free_css_set_rcu (struct rcu_head *obj) { struct css_set *cg = container_of(obj, struct css_set, rcu_head); -kfree (cg); + +INIT_WORK (&cg->work, free_css_set_work); +schedule_work (&cg->work); }/* We don't maintain the lists running through each css_set to its @@ -348,8 +400,6 @@ static inline void get_css_set(struct css_set *cg)static void put_css_set (struct css_set *cg) { -struct cg_cgroup_link *link; -struct cg_cgroup_link *saved_link;/* * Ensure that the refcount doesn't hit zero while any readers * can see it. Similar to atomic_dec_and_lock(), but for an @@ -363,21 +413,9 @@ static void put_css_set(struct css_set *cg)return ; } -/* This css_set is dead. unlink it and release cgroup refcounts */ hlist_del (&cg->hlist); css_set_count--; -list_for_each_entry_safe (link, saved_link, &cg->cg_links, - cg_link_list) { -struct cgroup *cgrp = link->cgrp; -list_del (&link->cg_link_list); -list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count)) -check_for_release (cgrp); - -kfree (link); - } -write_unlock (&css_set_lock);call_rcu (&cg->rcu_head, free_css_set_rcu); }@@ -711,9 +749,9 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task,* cgroup_attach_task(), which overwrites one tasks cgroup pointer with * another. It does so using cgroup_mutex, however there are * several performance critical places that need to reference -* task->cgroup without the expense of grabbing a system global +* task->cgroups without the expense of grabbing a system global * mutex. Therefore except as noted below, when dereferencing or, as -* in cgroup_attach_task(), modifying a task'ss cgroup pointer we use +* in cgroup_attach_task(), modifying a task's cgroups pointer we use * task_lock(), which acts on a spinlock (task->alloc_lock) already in * the task_struct routinely used for such matters. * @@ -895,33 +933,6 @@ static void cgroup_d_remove_dir(struct dentry *dentry) }/* -* A queue for waiters to do rmdir() cgroup. A tasks will sleep when -* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some -* reference to css->refcnt. In general, this refcnt is expected to goes down -* to zero, soon. -* -* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; -*/ -DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); - -static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) -{ -if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) -wake_up_all (&cgroup_rmdir_waitq); -} - -void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) -{ -css_get (css); -} - -void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) -{ -cgroup_wakeup_rmdir_waiter (css->cgroup); -css_put (css); -} - -/* * Call with cgroup_mutex held. Drops reference counts on modules, including * any duplicate ones that parse_cgroupfs_options took. If this function * returns an error, no reference counts are touched. @@ -1788,7 +1799,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) ss->attach (ss, cgrp, oldcgrp, tsk, false); }set_bit (CGRP_RELEASABLE , &cgrp->flags); -synchronize_rcu (); +/* put_css_set will not destroy cg until after an RCU grace period */ put_css_set (cg);/* @@ -3546,6 +3557,21 @@ static int cgroup_clear_css_refs(struct cgroup *cgrp)return !failed; } +/* checks if all of the css_sets attached to a cgroup have a refcount of 0. +* Must be called with css_set_lock held */ +static int cgroup_css_sets_empty (struct cgroup *cgrp) +{ +struct cg_cgroup_link *link; + +list_for_each_entry (link, &cgrp->css_sets, cgrp_link_list) { +struct css_set *cg = link->cg; +if (atomic_read (&cg->refcount) >0 ) +return 0 ; + } + +return 1 ; +} +static int cgroup_rmdir (struct inode *unused_dir,struct dentry *dentry) {struct cgroup *cgrp = dentry->d_fsdata;@@ -3558,7 +3584,7 @@ static int cgroup_rmdir(struct inode *unused_dir, struct dentry *dentry)/* the vfs holds both inode->i_mutex already */ again:mutex_lock (&cgroup_mutex); -if (atomic_read (&cgrp->count) !=0 ) { +if (!cgroup_css_sets_empty (cgrp)) {mutex_unlock (&cgroup_mutex);return -EBUSY ; }
kernel/cgroup.c again: · line 3623
Quoted 8 lines
kernel/cgroup.c +1
-1@@ -3591,7 +3617,7 @@ again:mutex_lock (&cgroup_mutex); parent = cgrp->parent; -if (atomic_read (&cgrp->count) || !list_empty (&cgrp->children)) { +if (!cgroup_css_sets_empty (cgrp) || !list_empty (&cgrp->children)) {clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags);mutex_unlock (&cgroup_mutex);return -EBUSY ;
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCH 2/2] cgroup: Remove call to synchronize_rcu in cgroup_attach_task
In reply to Colin CrossQuoted 233 lines · Colin Cross
synchronize_rcucan be very expensive, averaging 100 ms in some cases. Incgroup_attach_task, it is used to prevent atask->cgroupspointer dereferenced in an RCU read side critical section from being invalidated, by delaying the call toput_css_setuntil after an RCU grace period.To avoid the call to
synchronize_rcu, make theput_css_setcall rcu-safe by moving the deletion of thecss_setlinks intofree_css_set_work, scheduled by the rcu callbackfree_css_set_rcu.The decrement of the cgroup refcount is no longer synchronous with the call to
put_css_set, which can result in the cgroup refcount staying positive after the last call tocgroup_attach_taskreturns. To allow the cgroup to be deleted withcgroup_rmdirsynchronously aftercgroup_attach_task, have rmdir check the refcount of all associatedcss_sets. Ifcgroup_rmdiris called on a cgroup for which thecss_setsall have refcount zero but the cgroup refcount is nonzero, reuse the rmdir waitqueue to block the rmdir untilfree_css_set_workis called.Signed-off-by Colin Cross <[email protected]>--- include/linux/cgroup.h | 1 + kernel/cgroup.c | 120 +++++++++++++++++++++++++++++------------------- 2 files changed, 74 insertions(+), 47 deletions(-)include/linux/cgroup.h +1
-0@@ -279,6 +279,7 @@ struct css_set {/* For RCU-protected deletion */ struct rcu_head rcu_head; +struct work_struct work; };/* kernel/cgroup.c +72
-46@@ -267,6 +267,33 @@ static void cgroup_release_agent(struct work_struct *work); static DECLARE_WORK(release_agent_work, cgroup_release_agent);static void check_for_release (struct cgroup *cgrp); +/* +* A queue for waiters to do rmdir() cgroup. A tasks will sleep when +* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some +* reference to css->refcnt. In general, this refcnt is expected to goes down +* to zero, soon. +* +* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; +*/ +DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); + +static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) +{ +if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) +wake_up_all (&cgroup_rmdir_waitq); +} + +void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) +{ +css_get (css); +} + +void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) +{ +cgroup_wakeup_rmdir_waiter (css->cgroup); +css_put (css); +} +/* Link structure for associating css_set objects with cgroups */ struct cg_cgroup_link {/* @@ -326,10 +353,35 @@ static struct hlist_head *css_set_hash(struct cgroup_subsys_state *css[])return &css_set_table[index]; } +static void free_css_set_work (struct work_struct *work) +{ + struct css_set *cg = container_of(work, struct css_set, work); +struct cg_cgroup_link *link; +struct cg_cgroup_link *saved_link; + +write_lock (&css_set_lock); +list_for_each_entry_safe (link, saved_link, &cg->cg_links, + cg_link_list) { +struct cgroup *cgrp = link->cgrp; +list_del (&link->cg_link_list); +list_del (&link->cgrp_link_list); +if (atomic_dec_and_test (&cgrp->count)) { +check_for_release (cgrp); +cgroup_wakeup_rmdir_waiter (cgrp); + } +kfree (link); + } +write_unlock (&css_set_lock); + +kfree (cg); +} +static void free_css_set_rcu (struct rcu_head *obj) { struct css_set *cg = container_of(obj, struct css_set, rcu_head); -kfree (cg); + +INIT_WORK (&cg->work, free_css_set_work); +schedule_work (&cg->work); }/* We don't maintain the lists running through each css_set to its @@ -348,8 +400,6 @@ static inline void get_css_set(struct css_set *cg)static void put_css_set (struct css_set *cg) { -struct cg_cgroup_link *link; -struct cg_cgroup_link *saved_link;/* * Ensure that the refcount doesn't hit zero while any readers * can see it. Similar to atomic_dec_and_lock(), but for an @@ -363,21 +413,9 @@ static void put_css_set(struct css_set *cg)return ; } -/* This css_set is dead. unlink it and release cgroup refcounts */ hlist_del (&cg->hlist); css_set_count--; -list_for_each_entry_safe (link, saved_link, &cg->cg_links, - cg_link_list) { -struct cgroup *cgrp = link->cgrp; -list_del (&link->cg_link_list); -list_del (&link->cgrp_link_list); -if (atomic_dec_and_test (&cgrp->count)) -check_for_release (cgrp); - -kfree (link); - } -write_unlock (&css_set_lock);call_rcu (&cg->rcu_head, free_css_set_rcu); }@@ -711,9 +749,9 @@ static struct cgroup *task_cgroup_from_root(struct task_struct *task,* cgroup_attach_task(), which overwrites one tasks cgroup pointer with * another. It does so using cgroup_mutex, however there are * several performance critical places that need to reference -* task->cgroup without the expense of grabbing a system global +* task->cgroups without the expense of grabbing a system global * mutex. Therefore except as noted below, when dereferencing or, as -* in cgroup_attach_task(), modifying a task'ss cgroup pointer we use +* in cgroup_attach_task(), modifying a task's cgroups pointer we use * task_lock(), which acts on a spinlock (task->alloc_lock) already in * the task_struct routinely used for such matters. * @@ -895,33 +933,6 @@ static void cgroup_d_remove_dir(struct dentry *dentry) }/* -* A queue for waiters to do rmdir() cgroup. A tasks will sleep when -* cgroup->count == 0 && list_empty(&cgroup->children) && subsys has some -* reference to css->refcnt. In general, this refcnt is expected to goes down -* to zero, soon. -* -* CGRP_WAIT_ON_RMDIR flag is set under cgroup's inode->i_mutex; -*/ -DECLARE_WAIT_QUEUE_HEAD (cgroup_rmdir_waitq); - -static void cgroup_wakeup_rmdir_waiter (struct cgroup *cgrp) -{ -if (unlikely (test_and_clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags))) -wake_up_all (&cgroup_rmdir_waitq); -} - -void cgroup_exclude_rmdir (struct cgroup_subsys_state *css) -{ -css_get (css); -} - -void cgroup_release_and_wakeup_rmdir (struct cgroup_subsys_state *css) -{ -cgroup_wakeup_rmdir_waiter (css->cgroup); -css_put (css); -} - -/* * Call with cgroup_mutex held. Drops reference counts on modules, including * any duplicate ones that parse_cgroupfs_options took. If this function * returns an error, no reference counts are touched. @@ -1788,7 +1799,7 @@ int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk) ss->attach (ss, cgrp, oldcgrp, tsk, false); }set_bit (CGRP_RELEASABLE , &cgrp->flags); -synchronize_rcu (); +/* put_css_set will not destroy cg until after an RCU grace period */ put_css_set (cg);/* @@ -3546,6 +3557,21 @@ static int cgroup_clear_css_refs(struct cgroup *cgrp)return !failed; } +/* checks if all of the css_sets attached to a cgroup have a refcount of 0. +* Must be called with css_set_lock held */ +static int cgroup_css_sets_empty (struct cgroup *cgrp) +{ +struct cg_cgroup_link *link; + +list_for_each_entry (link, &cgrp->css_sets, cgrp_link_list) { +struct css_set *cg = link->cg; +if (atomic_read (&cg->refcount) >0 ) +return 0 ; + } + +return 1 ; +} +static int cgroup_rmdir (struct inode *unused_dir,struct dentry *dentry) {struct cgroup *cgrp = dentry->d_fsdata;@@ -3558,7 +3584,7 @@ static int cgroup_rmdir(struct inode *unused_dir, struct dentry *dentry)/* the vfs holds both inode->i_mutex already */ again:mutex_lock (&cgroup_mutex); -if (atomic_read (&cgrp->count) !=0 ) { +if (!cgroup_css_sets_empty (cgrp)) {mutex_unlock (&cgroup_mutex);return -EBUSY ; }
kernel/cgroup.c again: · line 3623
Quoted 8 lines
kernel/cgroup.c +1
-1@@ -3591,7 +3617,7 @@ again:mutex_lock (&cgroup_mutex); parent = cgrp->parent; -if (atomic_read (&cgrp->count) || !list_empty (&cgrp->children)) { +if (!cgroup_css_sets_empty (cgrp) || !list_empty (&cgrp->children)) {clear_bit (CGRP_WAIT_ON_RMDIR , &cgrp->flags);mutex_unlock (&cgroup_mutex);return -EBUSY ;
I'm responding on Mike's behalf and adding him to this thread. This patch improves launch time of a test app from ~700ms to ~250ms on MSM, with much lower variance across tests. We also see UI latency improvements, but have not quantified the gains.
- Bryan
Signature · 2 lines
Sent by an employee of the Qualcomm Innovation Center, Inc. The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
In reply to Li ZefanQuoted 3 lines · Li Zefan
If we do this, it's not anymore safe to use
get_css_set(), which just increments the refcount without checking if it's zero.
I don't believe that it's currently safe to use get_css_set() on a zero-refcount css_set anyway - a css_set starts with refcount 1 and can be freed the moment that its refcount reaches 0. You can only use get_css_set() on a stable reference to an existing css_set, which by definition has a refcount of at least 1. Were you thinking of any particular existing uses?
Paul
A message that is not in the archive (sent off-list or not yet received): AANLkTi=4-OgPUugnUBaqSU3oC=3wxTjAsOB_Ais3Or+i-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] cgroup: Convert synchronize_rcu to call_rcu in cgroup_attach_task
Quoted 29 lines · Paul Menage
Quoted 6 lines · Colin Cross
The
synchronize_rcucall incgroup_attach_taskcan be very expensive. All fastpath accesses totask->cgroupsthat expecttask->cgroupsnot to change already usetask_lock()orcgroup_lock()to protect against updates, and, incgroup.c, only theCGROUP_DEBUGfiles have RCU read-side critical sections.I definitely agree with the goal of using lighter-weight synchronization than the current
synchronize_rcu()call. However, there are definitely some subtleties to worry about in this code.One of the reasons originally for the current synchronization was to avoid the case of calling subsystem
destroy()callbacks while there could still be threads with RCU references to the subsystem state. The fact thatsynchronize_rcu()was called within acgroup_mutexcritical section meant that an rmdir (or any other significant cgrooup management action) couldn't possibly start until any RCU read sections were done.I suspect that when we moved a lot of the cgroup teardown code from
cgroup_rmdir()tocgroup_diput()(which also has asynchronize_rcu()call in it) this restriction could have been eased, but I think I left it as it was mostly out of paranoia that I was missing/forgetting some crucial reason for keeping it in place.I'd suggest trying the following approach, which I suspect is similar to what you were suggesting in your last email
Quoted 6 lines
1) make
find_existing_css_setignorecss_setobjects with a zero refcount
2) change__put_css_setto be simplyif (
atomic_dec_and_test(&cg->refcount)) {
call_rcu(&cg->rcu_head, free_css_set_rcu);
}
If we do this, it's not anymore safe to use get_css_set(), which just increments the refcount without checking if it's zero.
Quoted 17 lines
3) move the rest of
__put_css_setinto a delayed work struct that's scheduled byfree_css_set_rcu4) Get rid of the taskexit parameter - I think we can do that via a simple flag that indicates whether any task has ever been moved into the cgroup.
5) Put extra checks in
cgroup_rmdir()such that if it tries to remove a cgroup that has a non-zero refcount, it scans the cgroup'scss_setslist - if it finds only zero-refcount entries, then wait (viasynchronize_rcu()or some other appropriate means, maybe reusing theCGRP_WAIT_ON_RMDIRmechanism?) until thecss_setobjects have been fully cleaned up and the cgroup's refcounts have been released. Otherwise the operation of moving the last thread out of a cgroup and immediately deleting the cgroup would very likely fail with an EBUSY