cgroups: support for module-loadable subsystems
Subsystems (5)
cover cgroups: support for module-loadable subsystems
[This is a revision of http://lkml.org/lkml/2009/11/2/442 ]
This patch series implements support for building, loading, and unloading subsystems as modules, both within and outside the kernel source tree. It provides an interface cgroup_load_subsys() and cgroup_unload_subsys() which modular subsystems can use to register and depart during runtime. The net_cls classifier subsystem serves as the example for a subsystem which can be converted into a module using these changes. Additionally, support for subsystem dependencies is implemented with the new modular subsystem scenario in mind.
Patch #1 sets up the subsys[] array so its contents can be dynamic as modules appear and (eventually) disappear. I introduce an rwsem called subsys_mutex to protect against concurrent loads/unloads/reads, and modify iterations over the array to handle when subsystems are absent.
Patch #2 implements an interface for modules to load subsystems, called cgroup_load_subsys, similar to cgroup_init_subsys, and adds a module pointer in struct cgroup_subsys.
Patch #3 modifies the net_cls subsystem, which already had some module declarations, to be configurable as a module, which also serves as a simple proof-of-concept.
Patch #4 adds a mechanism for unloading modular subsystems, which includes a more advanced rework of the rudimentary reference counting introduced in patch 2. It also modifies the exitcall of net_cls to use this interface.
Patch #5 implements support for subsystems to specify dependencies on other subsystems. It requires the subsystem to give a list of its dependencies in the form of a NULL-terminated array of strings, each being the name of a depended-upon subsystem, for reasons explained in the patch itself.
Part of implementing patches 2 and 4 involved updating css pointers in each css_set when the module appears or leaves. In doing this, it was discovered that css_sets always remain linked to the dummy cgroup, regardless of whether or not any subsystems are actually bound to it (i.e., not mounted on an actual hierarchy). The subsystem loading and unloading code therefore should keep in mind the special cases where the added subsystem is the only one in the dummy cgroup (and therefore all css_sets need to be linked back into it) and where the removed subsys was the only one in the dummy cgroup (and therefore all css_sets should be unlinked from it) - however, as all css_sets always stay attached to the dummy cgroup anyway, these cases are ignored. Any fix that addresses this issue should also make sure these cases are addressed in the subsystem loading and unloading code.
-- bblum
A message that is not in the archive (sent off-list or not yet received): [email protected]
RFC PATCH 1/5 cgroups: revamp subsys array
Make subsys[] able to be dynamically populated to support modular subsystems
From: Ben Blum <[email protected]>
This patch reworks the way the subsys[] array is used so that subsystems can register themselves after boot time, and enables the internals of cgroups to be able to handle when subsystems are not present or may appear/disappear.
---
include/linux/cgroup.h | 8 +++- kernel/cgroup.c | 92 ++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 82 insertions(+), 18 deletions(-)
include/linux/cgroup.h +6 -2
@@ -41,13 +41,17 @@ extern int cgroupstats_build(struct cgroupstats *stats,extern struct file_operations proc_cgroup_operations; -/* Define the enumeration of all cgroup subsystems */ +/* Define the enumeration of all builtin cgroup subsystems */ #define SUBSYS (_x) _x ## _subsys_id,enum cgroup_subsys_id {#include <linux/cgroup_subsys.h> - CGROUP_SUBSYS_COUNT + CGROUP_BUILTIN_SUBSYS_COUNT };#undef SUBSYS +/* This define indicates the maximum number of subsystems that can be loaded +* at once. We limit to this many since cgroupfs_root has subsys_bits to keep +* track of all of them. */ +#define CGROUP_SUBSYS_COUNT (BITS_PER_BYTE*sizeof(unsigned long))/* Per-subsystem/per-cgroup state maintained by the system. */ struct cgroup_subsys_state {
kernel/cgroup.c +76 -16
279 lines not on this page · raw message
RFC PATCH 2/5 cgroups: subsystem module loading interface
In reply to a message not in the archiveAdd interface between cgroups subsystem management and module loading
From: Ben Blum <[email protected]>
This patch implements rudimentary module-loading support for cgroups - namely, a cgroup_load_subsys (similar to cgroup_init_subsys) for use as a module initcall, and a struct module pointer in struct cgroup_subsys.
Several functions that might be wanted by modules have had EXPORT_SYMBOL added to them, but it's unclear exactly which functions want it and which won't.
---
Documentation/cgroups/cgroups.txt | 3 + include/linux/cgroup.h | 4 + kernel/cgroup.c | 119 +++++++++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+), 0 deletions(-)
Documentation/cgroups/cgroups.txt +3 -0
@@ -492,6 +492,9 @@ Each subsystem should: - add an entry in linux/cgroup_subsys.h - define a cgroup_subsys object called <name>_subsys +If a subsystem can be compiled as a module, it should also have in its +module initcall a call to cgroup_load_subsys(). + Each subsystem may export the following methods. The only mandatory methods are create/destroy. Any others that are null are presumed to be successful no-ops.
include/linux/cgroup.h +4 -0
@@ -38,6 +38,7 @@ extern void cgroup_fork_failed(struct task_struct *p, int run_callbacks, unsigned long clone_flags);extern int cgroupstats_build (struct cgroupstats *stats,struct dentry *dentry); +extern int cgroup_load_subsys (struct cgroup_subsys *ss);extern struct file_operations proc_cgroup_operations;
@@ -477,6 +478,9 @@ struct cgroup_subsys {/* used when use_id == true */ struct idr idr;spinlock_t id_lock; + +/* should be defined only by modular subsystems */ +struct module *module; };#define SUBSYS (_x) extern struct cgroup_subsys _x ## _subsys;
kernel/cgroup.c +119 -0
138 lines not on this page · raw message
RFC PATCH 4/5 cgroups: subsystem module unloading
In reply to a message not in the archiveProvides support for unloading modular subsystems.
From: Ben Blum <[email protected]>
This patch adds a new function cgroup_unload_subsys which is to be used for removing a loaded subsystem during module deletion. Reference counting of the subsystems' modules is moved from once (at load time) to once per attached hierarchy (in rebind_subsystems) (i.e., 0 or 1).
It also adds a proper module_delete call in net/sched/cls_cgroup.c.
---
Documentation/cgroups/cgroups.txt | 3 + include/linux/cgroup.h | 4 +- kernel/cgroup.c | 92 ++++++++++++++++++++++++++++++++----- net/sched/cls_cgroup.c | 3 - 4 files changed, 86 insertions(+), 16 deletions(-)
Documentation/cgroups/cgroups.txt +2 -1
@@ -493,7 +493,8 @@ Each subsystem should: - define a cgroup_subsys object called <name>_subsys If a subsystem can be compiled as a module, it should also have in its -module initcall a call to cgroup_load_subsys(). +module initcall a call to cgroup_load_subsys(), and in its exitcall a +call to cgroup_unload_subsys(). Each subsystem may export the following methods. The only mandatory methods are create/destroy. Any others that are null are presumed to
include/linux/cgroup.h +3 -1
@@ -39,6 +39,7 @@ extern void cgroup_fork_failed(struct task_struct *p, int run_callbacks,extern int cgroupstats_build (struct cgroupstats *stats,struct dentry *dentry);extern int cgroup_load_subsys (struct cgroup_subsys *ss); +extern void cgroup_unload_subsys (struct cgroup_subsys *ss);extern struct file_operations proc_cgroup_operations;
@@ -264,7 +265,8 @@ struct css_set {/* * Set of subsystem states, one for each subsystem. This array * is immutable after creation apart from the init_css_set -* during subsystem registration (at boot time). +* during subsystem registration (at boot time) and modular subsystem +* loading/unloading. */ struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT ];
kernel/cgroup.c +80 -12
140 lines not on this page · raw message
net/sched/cls_cgroup.c +1 -2
@@ -300,8 +300,7 @@ static int __init init_cgroup_cls(void) static void __exit exit_cgroup_cls(void) {unregister_tcf_proto_ops (&cls_cgroup_ops); -/* TODO: unload subsystem. for now, the try_module_get in load_subsys -* prevents us from getting here. */ +cgroup_unload_subsys (&net_cls_subsys); }module_init (init_cgroup_cls);
RFC PATCH 5/5 cgroups: subsystem dependencies
In reply to a message not in the archiveAdds support for cgroup subsystems to specify dependencies on other subsystems
From: Ben Blum <[email protected]>
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end of parse_cgroupfs_options.
---
Documentation/cgroups/cgroups.txt | 4 ++++ include/linux/cgroup.h | 8 ++++++++ kernel/cgroup.c | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 0 deletions(-)
Documentation/cgroups/cgroups.txt +4 -0
@@ -458,6 +458,10 @@ Other fields in the cgroup_subsys object include: - name: should be initialized to a unique subsystem name. Should be no longer than MAX_CGROUP_TYPE_NAMELEN. +- depends: allows subsystems to specify dependencies in the form of + a list (NULL-terminated) of names of other subsystems. A subsystem + cannot be mounted unless all of its dependencies are also mounted. + - early_init: indicate if the subsystem needs early initialization at system boot.
include/linux/cgroup.h +8 -0
@@ -457,6 +457,14 @@ struct cgroup_subsys {bool use_id;#define MAX_CGROUP_TYPE_NAMELEN 32const char *name; +/* +* A subsystem can specify dependencies in the form of a list of names +* of other subsystems. Such a subsystem can only be mounted on a +* hierarchy if all subsystems named in this list are also to be +* mounted. If this list of strings is initialized, it must be null +* terminated. +*/ +const char **depends;/* * Protects sibling/children links of cgroups in this
kernel/cgroup.c +36 -0
@@ -1015,6 +1015,38 @@ struct cgroup_sb_opts { };/* +* Given a mask of subsystems to be mounted, ensure all dependencies are met. +*/ +static bool check_subsys_dependencies (unsigned long subsys_bits) +{ +unsigned long i, j, k; +bool matched; + +for_each_bit (i, &subsys_bits,CGROUP_SUBSYS_COUNT ) { +BUG_ON (!subsys[i]); +if (!subsys[i]->depends) +continue ; +/* For each specified dependency, make sure there's a matching +* subsystem among the ones intended for mounting. */ +for (j =0 ; subsys[i]->depends[j]; j++) { + matched = false; +for_each_bit (k, &subsys_bits,CGROUP_SUBSYS_COUNT ) { +BUG_ON (!subsys[k]); +BUG_ON (!subsys[k]->name); +if (!strcmp (subsys[i]->depends[j], + subsys[k]->name)) { + matched = true; +break ; + } + } +if (!matched) +return false; + } + } +return true; +} + +/* * Convert a hierarchy specifier into a bitmask of subsystems and flags. Call * with subsys_mutex held to guard subsys[] between us and rebind_subsystems. */
@@ -1123,6 +1155,10 @@ static int parse_cgroupfs_options(char *data,if (!opts->subsys_bits && !opts->name)return -EINVAL ; +/* Check subsystem dependencies */ +if (opts->subsys_bits && !check_subsys_dependencies (opts->subsys_bits)) +return -ESRCH ; +return 0 ; }
RFC PATCH 1/5 cgroups: revamp subsys array
In reply to Ben BlumMake subsys[] able to be dynamically populated to support modular subsystems
From: Ben Blum <[email protected]>
This patch reworks the way the subsys[] array is used so that subsystems can register themselves after boot time, and enables the internals of cgroups to be able to handle when subsystems are not present or may appear/disappear.
---
include/linux/cgroup.h | 8 +++- kernel/cgroup.c | 92 ++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 82 insertions(+), 18 deletions(-)
include/linux/cgroup.h +6 -2
@@ -41,13 +41,17 @@ extern int cgroupstats_build(struct cgroupstats *stats,extern struct file_operations proc_cgroup_operations; -/* Define the enumeration of all cgroup subsystems */ +/* Define the enumeration of all builtin cgroup subsystems */ #define SUBSYS (_x) _x ## _subsys_id,enum cgroup_subsys_id {#include <linux/cgroup_subsys.h> - CGROUP_SUBSYS_COUNT + CGROUP_BUILTIN_SUBSYS_COUNT };#undef SUBSYS +/* This define indicates the maximum number of subsystems that can be loaded +* at once. We limit to this many since cgroupfs_root has subsys_bits to keep +* track of all of them. */ +#define CGROUP_SUBSYS_COUNT (BITS_PER_BYTE*sizeof(unsigned long))/* Per-subsystem/per-cgroup state maintained by the system. */ struct cgroup_subsys_state {
kernel/cgroup.c +76 -16
279 lines not on this page · raw message
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);
Hmm.. this can lead to deadlock. sget() returns success with sb->s_umount held, so here we have:
down_read(&subsys_mutex); down_write(&sb->s_umount);
On the other hand, sb->s_umount is held before calling kill_sb(), so when umounting we have:
down_write(&sb->s_umount); down_read(&subsys_mutex);
Quoted 33 lines
kernel/cgroup.c +5
-0ret =parse_cgroupfs_options (data, &opts);if (ret) goto out_err;@@ -1367,6 +1401,8 @@ static int cgroup_get_sb(struct file_system_type *fs_type,free_cg_links (&tmp_cg_links); goto drop_new_super; } +/* done with subsys stuff and no other failure case */ +up_read (&subsys_mutex);/* EBUSY should be the only error here */ BUG_ON (ret);@@ -1415,6 +1451,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type, drop_new_super:deactivate_locked_super (sb); out_err: +up_read (&subsys_mutex);kfree (opts.release_agent);kfree (opts.name);@@ -1434,10 +1471,12 @@ static void cgroup_kill_sb(struct super_block *sb) {BUG_ON (!list_empty (&cgrp->children));BUG_ON (!list_empty (&cgrp->sibling)); +down_read (&subsys_mutex);mutex_lock (&cgroup_mutex);/* Rebind all subsystems back to the default hierarchy */ ret =rebind_subsystems (root,0 ); +up_read (&subsys_mutex);/* Shouldn't be able to fail ... */ BUG_ON (ret);
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Ben Blumkernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);
Hmm.. this can lead to deadlock. sget() returns success with sb->s_umount held, so here we have:
down_read(&subsys_mutex); down_write(&sb->s_umount);
On the other hand, sb->s_umount is held before calling kill_sb(), so when umounting we have:
down_write(&sb->s_umount); down_read(&subsys_mutex);
Quoted 33 lines
kernel/cgroup.c +5
-0ret =parse_cgroupfs_options (data, &opts);if (ret) goto out_err;@@ -1367,6 +1401,8 @@ static int cgroup_get_sb(struct file_system_type *fs_type,free_cg_links (&tmp_cg_links); goto drop_new_super; } +/* done with subsys stuff and no other failure case */ +up_read (&subsys_mutex);/* EBUSY should be the only error here */ BUG_ON (ret);@@ -1415,6 +1451,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type, drop_new_super:deactivate_locked_super (sb); out_err: +up_read (&subsys_mutex);kfree (opts.release_agent);kfree (opts.name);@@ -1434,10 +1471,12 @@ static void cgroup_kill_sb(struct super_block *sb) {BUG_ON (!list_empty (&cgrp->children));BUG_ON (!list_empty (&cgrp->sibling)); +down_read (&subsys_mutex);mutex_lock (&cgroup_mutex);/* Rebind all subsystems back to the default hierarchy */ ret =rebind_subsystems (root,0 ); +up_read (&subsys_mutex);/* Shouldn't be able to fail ... */ BUG_ON (ret);
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 13 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);
Quoted 6 lines
On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount); down_read(&subsys_mutex);
Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being down_write.
In fairness to readability, perhaps subsys_mutex should instead be subsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
-- bblum
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to a message not in the archiveQuoted 30 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount); down_read(&subsys_mutex);Quoted 10 lines
kernel/cgroup.c +2
-0ret =parse_cgroupfs_options (data, &opts);if (ret) goto out_err;@@ -1367,6 +1401,8 @@ static int cgroup_get_sb(struct file_system_type *fs_type,free_cg_links (&tmp_cg_links); goto drop_new_super; } +/* done with subsys stuff and no other failure case */ +up_read (&subsys_mutex);
Quoted 9 lines
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1456
Quoted 9 lines
kernel/cgroup.c +1
-0/* EBUSY should be the only error here */ BUG_ON (ret);@@ -1415,6 +1451,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type, drop_new_super:deactivate_locked_super (sb); out_err: +up_read (&subsys_mutex);kfree (opts.release_agent);kfree (opts.name);
There's another problem here. deactivate_locked_super() calls kill_sb(), which tries to down_read(&subsys_mutex), and rwsems can't recurse. This can be fixed by just moving the up_read(&subsys_mutex); at the end to before each 'goto' in each error case, though, so not a big deal.
-- bblum
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanQuoted 13 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);
Quoted 6 lines
On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount); down_read(&subsys_mutex);
Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being down_write.
In fairness to readability, perhaps subsys_mutex should instead be subsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
-- bblum
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 16 lines · Ben Blum
Quoted 15 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:
Quoted 8 lines
Quoted 3 lines
down_write(&sb->s_umount); down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.
lockdep will warn on this..
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------
| read(A) write(B) | | write(A) | | read(A) | | write(B) |
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Quoted 4 lines
In fairness to readability, perhaps
subsys_mutexshould instead besubsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Ben BlumQuoted 16 lines · Ben Blum
Quoted 15 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:
Quoted 8 lines
Quoted 3 lines
down_write(&sb->s_umount); down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.
lockdep will warn on this..
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------
| read(A) write(B) | | write(A) | | read(A) | | write(B) |
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Quoted 4 lines
In fairness to readability, perhaps
subsys_mutexshould instead besubsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Fix "To" and "Cc"..
Quoted 51 lines · Li Zefan
Quoted 16 lines · Ben Blum
Quoted 15 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:Quoted 7 lines
Quoted 3 lines
down_write(&sb->s_umount); down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Quoted 4 lines
In fairness to readability, perhaps
subsys_mutexshould instead besubsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to a message not in the archiveQuoted 19 lines · Li Zefan
Quoted 18 lines · Ben Blum
Quoted 17 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount);
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
Hm. Why did I not see this warning...?
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |
Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in cgroup_get_sb:
1) Take subsys_mutex
2) Call parse_cgroupfs_options()
3) Drop subsys_mutex
4) Call sget(), which gets sb->s_umount without subsys_mutex held
5) Take subsys_mutex
6) Call verify_cgroupfs_options()
7) Proceed as normal
In which verify_cgroupfs_options will be a new function that ensures the invariants that rebind_subsystems expects are still there; if not, bail out by jumping to drop_new_super just as if parse_cgroupfs_options had failed in the first place.
Another question: What's the justification for having an interface of seemingly symmetrical "initialize" and "destroy" functions, one of which has to take a lock and the other gets called with the lock already held? Seems like it's asking for trouble.
-- bblum
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanFix "To" and "Cc"..
Quoted 51 lines · Li Zefan
Quoted 16 lines · Ben Blum
Quoted 15 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:Quoted 7 lines
Quoted 3 lines
down_write(&sb->s_umount); down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Quoted 4 lines
In fairness to readability, perhaps
subsys_mutexshould instead besubsys_rwsem? It seemed to me to be that calling it "mutex" was conventional anyway.
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanQuoted 19 lines · Li Zefan
Quoted 18 lines · Ben Blum
Quoted 17 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount);
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
Hm. Why did I not see this warning...?
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |
Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in cgroup_get_sb:
1) Take subsys_mutex
2) Call parse_cgroupfs_options()
3) Drop subsys_mutex
4) Call sget(), which gets sb->s_umount without subsys_mutex held
5) Take subsys_mutex
6) Call verify_cgroupfs_options()
7) Proceed as normal
In which verify_cgroupfs_options will be a new function that ensures the invariants that rebind_subsystems expects are still there; if not, bail out by jumping to drop_new_super just as if parse_cgroupfs_options had failed in the first place.
Another question: What's the justification for having an interface of seemingly symmetrical "initialize" and "destroy" functions, one of which has to take a lock and the other gets called with the lock already held? Seems like it's asking for trouble.
-- bblum
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount);
Quoted 9 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
Hm. Why did I not see this warning...?
Because you haven't triggered it. ;)
The scripts below can trigger the warning (at least for me):
# cat test1.sh
#! /bin/sh
for ((; ;))
{
mount -t cgroup -o devices xxx /cgroup1
umount /cgroup1
}# cat test2.sh
#! /bin/sh
for ((; ;))
{
mount -t cgroup -o devices xxx /cgroup2
umount /cgroup2
}Quoted 33 lines
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in
cgroup_get_sb:1) Take
subsys_mutex
2) Callparse_cgroupfs_options()
3) Dropsubsys_mutex
4) Callsget(), which getssb->s_umountwithoutsubsys_mutexheld
5) Takesubsys_mutex
6) Callverify_cgroupfs_options()
7) Proceed as normal
Quoted 5 lines
In which
verify_cgroupfs_optionswill be a new function that ensures the invariants thatrebind_subsystemsexpects are still there; if not, bail out by jumping todrop_new_superjust as ifparse_cgroupfs_optionshad failed in the first place.
The current code doesn't need this verify_cgroupfs_options, so why it will become necessary? I think what we need is grab module refcnt in parse_cgroupfs_options, and then we can drop subsys_mutex.
But why you are using a rw semaphore? I think a mutex is fine. And why not just use cgroup_mutex to protect the subsys[] array? The adding and spreading of subsys_mutex looks ugly to me.
Quoted 5 lines
Another question: What's the justification for having an interface of seemingly symmetrical "initialize" and "destroy" functions, one of which has to take a lock and the other gets called with the lock already held? Seems like it's asking for trouble.
Are you refering to get_sb() and kill_sb()? VFS is not my area, so I'm not going to judge it. ;)
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Ben BlumQuoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount);
Quoted 9 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
down_read(&subsys_mutex);Unless I'm gravely mistaken, you can't have deadlock on an rwsem when it's being taken for reading in both cases? You would have to have at least one of the cases being
down_write.lockdep will warn on this..
Hm. Why did I not see this warning...?
Because you haven't triggered it. ;)
The scripts below can trigger the warning (at least for me):
# cat test1.sh
#! /bin/sh
for ((; ;))
{
mount -t cgroup -o devices xxx /cgroup1
umount /cgroup1
}# cat test2.sh
#! /bin/sh
for ((; ;))
{
mount -t cgroup -o devices xxx /cgroup2
umount /cgroup2
}Quoted 33 lines
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in
cgroup_get_sb:1) Take
subsys_mutex
2) Callparse_cgroupfs_options()
3) Dropsubsys_mutex
4) Callsget(), which getssb->s_umountwithoutsubsys_mutexheld
5) Takesubsys_mutex
6) Callverify_cgroupfs_options()
7) Proceed as normal
Quoted 5 lines
In which
verify_cgroupfs_optionswill be a new function that ensures the invariants thatrebind_subsystemsexpects are still there; if not, bail out by jumping todrop_new_superjust as ifparse_cgroupfs_optionshad failed in the first place.
The current code doesn't need this verify_cgroupfs_options, so why it will become necessary? I think what we need is grab module refcnt in parse_cgroupfs_options, and then we can drop subsys_mutex.
But why you are using a rw semaphore? I think a mutex is fine. And why not just use cgroup_mutex to protect the subsys[] array? The adding and spreading of subsys_mutex looks ugly to me.
Quoted 5 lines
Another question: What's the justification for having an interface of seemingly symmetrical "initialize" and "destroy" functions, one of which has to take a lock and the other gets called with the lock already held? Seems like it's asking for trouble.
Are you refering to get_sb() and kill_sb()? VFS is not my area, so I'm not going to judge it. ;)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 39 lines · Li Zefan
Quoted 33 lines
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in
cgroup_get_sb:1) Take
subsys_mutex
2) Callparse_cgroupfs_options()
3) Dropsubsys_mutex
4) Callsget(), which getssb->s_umountwithoutsubsys_mutexheld
5) Takesubsys_mutex
6) Callverify_cgroupfs_options()
7) Proceed as normalQuoted 5 lines
In which
verify_cgroupfs_optionswill be a new function that ensures the invariants thatrebind_subsystemsexpects are still there; if not, bail out by jumping todrop_new_superjust as ifparse_cgroupfs_optionshad failed in the first place.
Quoted 3 lines
The current code doesn't need this
verify_cgroupfs_options, so why it will become necessary? I think what we need is grab module refcnt inparse_cgroupfs_options, and then we can dropsubsys_mutex.
Oh, good point. I thought pinning the modules had to happen in rebinding since there's a case where rebind_subsystems is called without parsing, but that's just in kill_sb where no new subsystems are added. So, better would be to make sure we can't get owned while we drop the lock instead of checking afterwards if we got owned and bailing if so.
Quoted 1 line
But why you are using a rw semaphore? I think a mutex is fine.
The "most of cgroups wants to look at the subsys array" versus "module loading/unloading modifies the array" is clearly a readers/writers case.
Quoted 2 lines
And why not just use
cgroup_mutexto protect the subsys[] array? The adding and spreading ofsubsys_mutexlooks ugly to me.
The reasoning for this is that there are various chunks of code that need to be protected by a mutex guarding subsys[] that aren't already under cgroup_mutex - like parse_cgroupfs_options, or the first stage of cgroup_load_subsys. Do you think those critical sections are small enough that sacrificing reentrancy for simplicity of code is worth it?
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanQuoted 39 lines · Li Zefan
Quoted 33 lines
Quoted 13 lines
And it can really lead to deadlock, though not so obivously:
thread 1 thread 2 thread 3
-------------------------------------------| read(A) write(B) | | write(A) | | read(A) | | write(B) |Quoted 6 lines
t3 is waiting for t1 to release the lock, then t2 tries to acquire A lock to read, but it has to wait because of t3, and t1 has to wait t2.
Note: a read lock has to wait if a write lock is already waiting for the lock.
Okay, clever, the deadlock happens because of a behavioural optimization of the rwsems. Good catch on the whole issue.
How does this sound as a possible solution, in
cgroup_get_sb:1) Take
subsys_mutex
2) Callparse_cgroupfs_options()
3) Dropsubsys_mutex
4) Callsget(), which getssb->s_umountwithoutsubsys_mutexheld
5) Takesubsys_mutex
6) Callverify_cgroupfs_options()
7) Proceed as normalQuoted 5 lines
In which
verify_cgroupfs_optionswill be a new function that ensures the invariants thatrebind_subsystemsexpects are still there; if not, bail out by jumping todrop_new_superjust as ifparse_cgroupfs_optionshad failed in the first place.
Quoted 3 lines
The current code doesn't need this
verify_cgroupfs_options, so why it will become necessary? I think what we need is grab module refcnt inparse_cgroupfs_options, and then we can dropsubsys_mutex.
Oh, good point. I thought pinning the modules had to happen in rebinding since there's a case where rebind_subsystems is called without parsing, but that's just in kill_sb where no new subsystems are added. So, better would be to make sure we can't get owned while we drop the lock instead of checking afterwards if we got owned and bailing if so.
Quoted 1 line
But why you are using a rw semaphore? I think a mutex is fine.
The "most of cgroups wants to look at the subsys array" versus "module loading/unloading modifies the array" is clearly a readers/writers case.
Quoted 2 lines
And why not just use
cgroup_mutexto protect the subsys[] array? The adding and spreading ofsubsys_mutexlooks ugly to me.
The reasoning for this is that there are various chunks of code that need to be protected by a mutex guarding subsys[] that aren't already under cgroup_mutex - like parse_cgroupfs_options, or the first stage of cgroup_load_subsys. Do you think those critical sections are small enough that sacrificing reentrancy for simplicity of code is worth it?
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 25 lines
Quoted 10 lines
Quoted 10 lines
How does this sound as a possible solution, in
cgroup_get_sb:1) Take
subsys_mutex
2) Callparse_cgroupfs_options()
3) Dropsubsys_mutex
4) Callsget(), which getssb->s_umountwithoutsubsys_mutexheld
5) Takesubsys_mutex
6) Callverify_cgroupfs_options()
7) Proceed as normalQuoted 8 lines
Quoted 5 lines
In which
verify_cgroupfs_optionswill be a new function that ensures the invariants thatrebind_subsystemsexpects are still there; if not, bail out by jumping todrop_new_superjust as ifparse_cgroupfs_optionshad failed in the first place.The current code doesn't need this
verify_cgroupfs_options, so why it will become necessary? I think what we need is grab module refcnt inparse_cgroupfs_options, and then we can dropsubsys_mutex.Oh, good point. I thought pinning the modules had to happen in rebinding since there's a case where
rebind_subsystemsis called without parsing, but that's just inkill_sbwhere no new subsystems are added. So, better would be to make sure we can't get owned while we drop the lock instead of checking afterwards if we got owned and bailing if so.
Quoted 5 lines
Quoted 1 line
But why you are using a rw semaphore? I think a mutex is fine.
The "most of cgroups wants to look at the subsys array" versus "module loading/unloading modifies the array" is clearly a readers/writers case.
Yes, but it doesn't mean we should use rw lock or rw semaphore is preferable than plain mutex.
- the read side of subsys_mutex is mainly at mount/remount/umount, the write side is in cgroup_load_subsys() and cgroup_unload_subsys(). None is in critical path.
- In most callsites, cgroup_mutex is held just after acquiring subsys_mutex.
So what does it gain us to use this rw_sem?
Quoted 9 lines
Quoted 2 lines
And why not just use
cgroup_mutexto protect the subsys[] array? The adding and spreading ofsubsys_mutexlooks ugly to me.The reasoning for this is that there are various chunks of code that need to be protected by a mutex guarding subsys[] that aren't already under
cgroup_mutex- likeparse_cgroupfs_options, or the first stage ofcgroup_load_subsys. Do you think those critical sections are small enough that sacrificing reentrancy for simplicity of code is worth it?
Except parse_cgroupfs_options() which is called without cgroup_mutex held, in all other callsites, cgroup_mutex is held right after acquiring subsys_mutex.
So yes, I don't think use cgroup_mutex will harm scalibility.
In contrast, this subsys_mutex is quite ugly and deadlock-prone. For example, see this:
static int cgroup_remount(struct super_block *sb, int *flags, char *data)
{
...
lock_kernel();
mutex_lock(&cgrp->dentry->d_inode->i_mutex);
down_read(&subsys_mutex);
mutex_lock(&cgroup_mutex);
...
}Four locks here!
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
Quoted 22 lines · Li Zefan
Yes, but it doesn't mean we should use rw lock or rw semaphore is preferable than plain mutex.
- the read side of
subsys_mutexis mainly at mount/remount/umount, the write side is incgroup_load_subsys()andcgroup_unload_subsys(). None is in critical path.- In most callsites,
cgroup_mutexis held just after acquiringsubsys_mutex.So what does it gain us to use this
rw_sem?Quoted 9 lines
Quoted 2 lines
And why not just use
cgroup_mutexto protect the subsys[] array? The adding and spreading ofsubsys_mutexlooks ugly to me.The reasoning for this is that there are various chunks of code that need to be protected by a mutex guarding subsys[] that aren't already under
cgroup_mutex- likeparse_cgroupfs_options, or the first stage ofcgroup_load_subsys. Do you think those critical sections are small enough that sacrificing reentrancy for simplicity of code is worth it?
Quoted 7 lines
Except
parse_cgroupfs_options()which is called withoutcgroup_mutexheld, in all other callsites,cgroup_mutexis held right after acquiringsubsys_mutex.So yes, I don't think use
cgroup_mutexwill harm scalibility.In contrast, this
subsys_mutexis quite ugly and deadlock-prone.
to be fair the deadlock case would still be as much to worry about if we were protecting subsys[] with cgroup_mutex instead.
Quoted 12 lines
For example, see this:
static int
cgroup_remount(struct super_block *sb, int *flags, char *data){ ... lock_kernel(); mutex_lock(&cgrp->dentry->d_inode->i_mutex); down_read(&subsys_mutex); mutex_lock(&cgroup_mutex); ... }
Quoted 3 lines
Four locks here!
if we really don't care about performance in those places, it doesn't really gain anything. when I presented the previous patch series, one guy (at google) asked me if anything could be done about the "big ugly cgroup_mutex" and that he wished most of cgroups could be less all under one single lock. so, I wrote this with that in mind...
anyway, I think the best solution here is - whichever lock ends up being used for this - have parse_cgroupfs_options and rebind_subsystems each take the lock when they are called and drop before returning, instead of having their calling functions hold the lock between calls. division of labour would be that parse_cgroupfs_options takes the module refcounts, and that rebind_subsystems drops them.
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanQuoted 22 lines · Li Zefan
Yes, but it doesn't mean we should use rw lock or rw semaphore is preferable than plain mutex.
- the read side of
subsys_mutexis mainly at mount/remount/umount, the write side is incgroup_load_subsys()andcgroup_unload_subsys(). None is in critical path.- In most callsites,
cgroup_mutexis held just after acquiringsubsys_mutex.So what does it gain us to use this
rw_sem?Quoted 9 lines
Quoted 2 lines
And why not just use
cgroup_mutexto protect the subsys[] array? The adding and spreading ofsubsys_mutexlooks ugly to me.The reasoning for this is that there are various chunks of code that need to be protected by a mutex guarding subsys[] that aren't already under
cgroup_mutex- likeparse_cgroupfs_options, or the first stage ofcgroup_load_subsys. Do you think those critical sections are small enough that sacrificing reentrancy for simplicity of code is worth it?
Quoted 7 lines
Except
parse_cgroupfs_options()which is called withoutcgroup_mutexheld, in all other callsites,cgroup_mutexis held right after acquiringsubsys_mutex.So yes, I don't think use
cgroup_mutexwill harm scalibility.In contrast, this
subsys_mutexis quite ugly and deadlock-prone.
to be fair the deadlock case would still be as much to worry about if we were protecting subsys[] with cgroup_mutex instead.
Quoted 12 lines
For example, see this:
static int
cgroup_remount(struct super_block *sb, int *flags, char *data){ ... lock_kernel(); mutex_lock(&cgrp->dentry->d_inode->i_mutex); down_read(&subsys_mutex); mutex_lock(&cgroup_mutex); ... }
Quoted 3 lines
Four locks here!
if we really don't care about performance in those places, it doesn't really gain anything. when I presented the previous patch series, one guy (at google) asked me if anything could be done about the "big ugly cgroup_mutex" and that he wished most of cgroups could be less all under one single lock. so, I wrote this with that in mind...
anyway, I think the best solution here is - whichever lock ends up being used for this - have parse_cgroupfs_options and rebind_subsystems each take the lock when they are called and drop before returning, instead of having their calling functions hold the lock between calls. division of labour would be that parse_cgroupfs_options takes the module refcounts, and that rebind_subsystems drops them.
Re: [RFC] [PATCH 1/5] cgroups: revamp subsys array
In reply to Li ZefanQuoted 30 lines · Li Zefan
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1327
Quoted 5 lines
kernel/cgroup.c +1
-0@@ -1291,6 +1324,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type,struct cgroupfs_root *new_root;/* First find the desired set of subsystems */ +down_read (&subsys_mutex);Hmm.. this can lead to deadlock.
sget()returns success withsb->s_umountheld, so here we have:down_read(&subsys_mutex); down_write(&sb->s_umount);On the other hand,
sb->s_umountis held before callingkill_sb(), so when umounting we have:down_write(&sb->s_umount); down_read(&subsys_mutex);Quoted 10 lines
kernel/cgroup.c +2
-0ret =parse_cgroupfs_options (data, &opts);if (ret) goto out_err;@@ -1367,6 +1401,8 @@ static int cgroup_get_sb(struct file_system_type *fs_type,free_cg_links (&tmp_cg_links); goto drop_new_super; } +/* done with subsys stuff and no other failure case */ +up_read (&subsys_mutex);
Quoted 9 lines
kernel/cgroup.c static int cgroup_get_sb(struct file_system_type *fs_type, · line 1456
Quoted 9 lines
kernel/cgroup.c +1
-0/* EBUSY should be the only error here */ BUG_ON (ret);@@ -1415,6 +1451,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type, drop_new_super:deactivate_locked_super (sb); out_err: +up_read (&subsys_mutex);kfree (opts.release_agent);kfree (opts.name);
There's another problem here. deactivate_locked_super() calls kill_sb(), which tries to down_read(&subsys_mutex), and rwsems can't recurse. This can be fixed by just moving the up_read(&subsys_mutex); at the end to before each 'goto' in each error case, though, so not a big deal.
-- bblum
RFC PATCH 2/5 cgroups: subsystem module loading interface
In reply to Ben BlumAdd interface between cgroups subsystem management and module loading
From: Ben Blum <[email protected]>
This patch implements rudimentary module-loading support for cgroups - namely, a cgroup_load_subsys (similar to cgroup_init_subsys) for use as a module initcall, and a struct module pointer in struct cgroup_subsys.
Several functions that might be wanted by modules have had EXPORT_SYMBOL added to them, but it's unclear exactly which functions want it and which won't.
---
Documentation/cgroups/cgroups.txt | 3 + include/linux/cgroup.h | 4 + kernel/cgroup.c | 119 +++++++++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+), 0 deletions(-)
Documentation/cgroups/cgroups.txt +3 -0
@@ -492,6 +492,9 @@ Each subsystem should: - add an entry in linux/cgroup_subsys.h - define a cgroup_subsys object called <name>_subsys +If a subsystem can be compiled as a module, it should also have in its +module initcall a call to cgroup_load_subsys(). + Each subsystem may export the following methods. The only mandatory methods are create/destroy. Any others that are null are presumed to be successful no-ops.
include/linux/cgroup.h +4 -0
@@ -38,6 +38,7 @@ extern void cgroup_fork_failed(struct task_struct *p, int run_callbacks, unsigned long clone_flags);extern int cgroupstats_build (struct cgroupstats *stats,struct dentry *dentry); +extern int cgroup_load_subsys (struct cgroup_subsys *ss);extern struct file_operations proc_cgroup_operations;
@@ -477,6 +478,9 @@ struct cgroup_subsys {/* used when use_id == true */ struct idr idr;spinlock_t id_lock; + +/* should be defined only by modular subsystems */ +struct module *module; };#define SUBSYS (_x) extern struct cgroup_subsys _x ## _subsys;
kernel/cgroup.c +119 -0
138 lines not on this page · raw message
RFC PATCH 3/5 cgroups: net_cls as module
In reply to Ben BlumAllows the net_cls cgroup subsystem to be compiled as a module
From: Ben Blum <[email protected]>
This patch modifies net/sched/cls_cgroup.c to allow the net_cls subsystem to be optionally compiled as a module instead of builtin. The cgroup_subsys struct is moved around a bit to allow the subsys_id to be either declared as a compile-time constant by the cgroup_subsys.h include in cgroup.h, or, if it's a module, initialized within the struct by cgroup_load_subsys.
---
net/sched/Kconfig | 5 ++++- net/sched/cls_cgroup.c | 37 ++++++++++++++++++++++++++++--------- 2 files changed, 32 insertions(+), 10 deletions(-)
net/sched/Kconfig +4 -1
@@ -328,13 +328,16 @@ config NET_CLS_FLOW module will be called cls_flow. config NET_CLS_CGROUP - bool "Control Group Classifier" + tristate "Control Group Classifier" select NET_CLS depends on CGROUPS ---help--- Say Y here if you want to classify packets based on the control cgroup of their process. + To compile this code as a module, choose M here: the + module will be called cls_cgroup. + config NET_EMATCH bool "Extended Matches" select NET_CLS
net/sched/cls_cgroup.c +28 -9
@@ -24,6 +24,25 @@ struct cgroup_cls_stateu32 classid; }; +static struct cgroup_subsys_state *cgrp_create (struct cgroup_subsys *ss, +struct cgroup *cgrp); +static void cgrp_destroy (struct cgroup_subsys *ss,struct cgroup *cgrp); +static int cgrp_populate (struct cgroup_subsys *ss,struct cgroup *cgrp); + +struct cgroup_subsys net_cls_subsys = { + .name ="net_cls" , + .create = cgrp_create, + .destroy = cgrp_destroy, + .populate = cgrp_populate, +#ifdef CONFIG_NET_CLS_CGROUP + .subsys_id = net_cls_subsys_id, +#else +#define net_cls_subsys_id net_cls_subsys.subsys_id +#endif + .module =THIS_MODULE , +}; + +static inline struct cgroup_cls_state *cgrp_cls_state (struct cgroup *cgrp) { return container_of(cgroup_subsys_state(cgrp, net_cls_subsys_id),
@@ -79,14 +98,6 @@ static int cgrp_populate(struct cgroup_subsys *ss, struct cgroup *cgrp)return cgroup_add_files (cgrp, ss, ss_files,ARRAY_SIZE (ss_files)); } -struct cgroup_subsys net_cls_subsys = { - .name ="net_cls" , - .create = cgrp_create, - .destroy = cgrp_destroy, - .populate = cgrp_populate, - .subsys_id = net_cls_subsys_id, -}; -struct cls_cgroup_head {u32 handle;
@@ -277,12 +288,20 @@ static struct tcf_proto_ops cls_cgroup_ops __read_mostly = {static int __init init_cgroup_cls (void ) { -return register_tcf_proto_ops (&cls_cgroup_ops); +int ret =register_tcf_proto_ops (&cls_cgroup_ops); +if (ret) +return ret; + ret =cgroup_load_subsys (&net_cls_subsys); +if (ret) +unregister_tcf_proto_ops (&cls_cgroup_ops); +return ret; } static void __exit exit_cgroup_cls(void) {unregister_tcf_proto_ops (&cls_cgroup_ops); +/* TODO: unload subsystem. for now, the try_module_get in load_subsys +* prevents us from getting here. */ }module_init (init_cgroup_cls);
Re: [RFC] [PATCH 3/5] cgroups: net_cls as module
net/sched/cls_cgroup.c struct cgroup_cls_state · line 41
Quoted 10 lines
net/sched/cls_cgroup.c +10
-0+struct cgroup_subsys net_cls_subsys = { + .name ="net_cls" , + .create = cgrp_create, + .destroy = cgrp_destroy, + .populate = cgrp_populate, +#ifdef CONFIG_NET_CLS_CGROUP + .subsys_id = net_cls_subsys_id, +#else +#define net_cls_subsys_id net_cls_subsys.subsys_id +#endif
This looks a bit odd. I think below is the more standard way:
#ifndef MODULES
...
#else
...
#endif
Quoted 2 lines
net/sched/cls_cgroup.c +2
-0+ .module = THIS_MODULE, +};
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 3/5] cgroups: net_cls as module
net/sched/cls_cgroup.c struct cgroup_cls_state · line 41
Quoted 10 lines
net/sched/cls_cgroup.c +10
-0+struct cgroup_subsys net_cls_subsys = { + .name ="net_cls" , + .create = cgrp_create, + .destroy = cgrp_destroy, + .populate = cgrp_populate, +#ifdef CONFIG_NET_CLS_CGROUP + .subsys_id = net_cls_subsys_id, +#else +#define net_cls_subsys_id net_cls_subsys.subsys_id +#endif
This looks a bit odd. I think below is the more standard way:
#ifndef MODULES
...
#else
...
#endif
Quoted 2 lines
net/sched/cls_cgroup.c +2
-0+ .module = THIS_MODULE, +};
RFC PATCH 4/5 cgroups: subsystem module unloading
In reply to Ben BlumProvides support for unloading modular subsystems.
From: Ben Blum <[email protected]>
This patch adds a new function cgroup_unload_subsys which is to be used for removing a loaded subsystem during module deletion. Reference counting of the subsystems' modules is moved from once (at load time) to once per attached hierarchy (in rebind_subsystems) (i.e., 0 or 1).
It also adds a proper module_delete call in net/sched/cls_cgroup.c.
---
Documentation/cgroups/cgroups.txt | 3 + include/linux/cgroup.h | 4 +- kernel/cgroup.c | 92 ++++++++++++++++++++++++++++++++----- net/sched/cls_cgroup.c | 3 - 4 files changed, 86 insertions(+), 16 deletions(-)
Documentation/cgroups/cgroups.txt +2 -1
@@ -493,7 +493,8 @@ Each subsystem should: - define a cgroup_subsys object called <name>_subsys If a subsystem can be compiled as a module, it should also have in its -module initcall a call to cgroup_load_subsys(). +module initcall a call to cgroup_load_subsys(), and in its exitcall a +call to cgroup_unload_subsys(). Each subsystem may export the following methods. The only mandatory methods are create/destroy. Any others that are null are presumed to
include/linux/cgroup.h +3 -1
@@ -39,6 +39,7 @@ extern void cgroup_fork_failed(struct task_struct *p, int run_callbacks,extern int cgroupstats_build (struct cgroupstats *stats,struct dentry *dentry);extern int cgroup_load_subsys (struct cgroup_subsys *ss); +extern void cgroup_unload_subsys (struct cgroup_subsys *ss);extern struct file_operations proc_cgroup_operations;
@@ -264,7 +265,8 @@ struct css_set {/* * Set of subsystem states, one for each subsystem. This array * is immutable after creation apart from the init_css_set -* during subsystem registration (at boot time). +* during subsystem registration (at boot time) and modular subsystem +* loading/unloading. */ struct cgroup_subsys_state *subsys[CGROUP_SUBSYS_COUNT ];
kernel/cgroup.c +80 -12
140 lines not on this page · raw message
net/sched/cls_cgroup.c +1 -2
@@ -300,8 +300,7 @@ static int __init init_cgroup_cls(void) static void __exit exit_cgroup_cls(void) {unregister_tcf_proto_ops (&cls_cgroup_ops); -/* TODO: unload subsystem. for now, the try_module_get in load_subsys -* prevents us from getting here. */ +cgroup_unload_subsys (&net_cls_subsys); }module_init (init_cgroup_cls);
RFC PATCH 5/5 cgroups: subsystem dependencies
In reply to Ben BlumAdds support for cgroup subsystems to specify dependencies on other subsystems
From: Ben Blum <[email protected]>
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end of parse_cgroupfs_options.
---
Documentation/cgroups/cgroups.txt | 4 ++++ include/linux/cgroup.h | 8 ++++++++ kernel/cgroup.c | 36 ++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 0 deletions(-)
Documentation/cgroups/cgroups.txt +4 -0
@@ -458,6 +458,10 @@ Other fields in the cgroup_subsys object include: - name: should be initialized to a unique subsystem name. Should be no longer than MAX_CGROUP_TYPE_NAMELEN. +- depends: allows subsystems to specify dependencies in the form of + a list (NULL-terminated) of names of other subsystems. A subsystem + cannot be mounted unless all of its dependencies are also mounted. + - early_init: indicate if the subsystem needs early initialization at system boot.
include/linux/cgroup.h +8 -0
@@ -457,6 +457,14 @@ struct cgroup_subsys {bool use_id;#define MAX_CGROUP_TYPE_NAMELEN 32const char *name; +/* +* A subsystem can specify dependencies in the form of a list of names +* of other subsystems. Such a subsystem can only be mounted on a +* hierarchy if all subsystems named in this list are also to be +* mounted. If this list of strings is initialized, it must be null +* terminated. +*/ +const char **depends;/* * Protects sibling/children links of cgroups in this
kernel/cgroup.c +36 -0
@@ -1015,6 +1015,38 @@ struct cgroup_sb_opts { };/* +* Given a mask of subsystems to be mounted, ensure all dependencies are met. +*/ +static bool check_subsys_dependencies (unsigned long subsys_bits) +{ +unsigned long i, j, k; +bool matched; + +for_each_bit (i, &subsys_bits,CGROUP_SUBSYS_COUNT ) { +BUG_ON (!subsys[i]); +if (!subsys[i]->depends) +continue ; +/* For each specified dependency, make sure there's a matching +* subsystem among the ones intended for mounting. */ +for (j =0 ; subsys[i]->depends[j]; j++) { + matched = false; +for_each_bit (k, &subsys_bits,CGROUP_SUBSYS_COUNT ) { +BUG_ON (!subsys[k]); +BUG_ON (!subsys[k]->name); +if (!strcmp (subsys[i]->depends[j], + subsys[k]->name)) { + matched = true; +break ; + } + } +if (!matched) +return false; + } + } +return true; +} + +/* * Convert a hierarchy specifier into a bitmask of subsystems and flags. Call * with subsys_mutex held to guard subsys[] between us and rebind_subsystems. */
@@ -1123,6 +1155,10 @@ static int parse_cgroupfs_options(char *data,if (!opts->subsys_bits && !opts->name)return -EINVAL ; +/* Check subsystem dependencies */ +if (opts->subsys_bits && !check_subsys_dependencies (opts->subsys_bits)) +return -ESRCH ; +return 0 ; }
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
Quoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.
I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
In reply to Ben BlumQuoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.
I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
Quoted 11 lines · Li Zefan
Quoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
Dunno - Paul mentioned it might be useful in the future, so I threw this together in an hour or so. Not entirely sure what he had in mind, but this series can go with or without this particular patch.
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
In reply to Li ZefanQuoted 11 lines · Li Zefan
Quoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
Dunno - Paul mentioned it might be useful in the future, so I threw this together in an hour or so. Not entirely sure what he had in mind, but this series can go with or without this particular patch.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
Quoted 7 lines · Ben Blum
Quoted 6 lines · Li Zefan
Quoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.
Quoted 9 lines
Quoted 4 lines
I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
Dunno - Paul mentioned it might be useful in the future, so I threw this together in an hour or so. Not entirely sure what he had in mind, but this series can go with or without this particular patch.
So I think Andrew's suggestion still applies: drop the patch for now, until it finds its user.
Re: [RFC] [PATCH 5/5] cgroups: subsystem dependencies
In reply to Ben BlumQuoted 7 lines · Ben Blum
Quoted 6 lines · Li Zefan
Quoted 6 lines
This patch adds a mechanism with which a subsystem can specify dependencies on other subsystems. Since subsystems can come and go, it would not be useful to refer to a depended-upon subsystem by
subsys_id, so instead a NULL-terminated array of strings must be specified. Validation is done by scanning for each potential subsystem its list of subsystem names with strcmp at the end ofparse_cgroupfs_options.
Quoted 9 lines
Quoted 4 lines
I once posted a patch to implement this feature, but it was rejected by Andrew, because there was no user using it.
And still there is no user, right?
Dunno - Paul mentioned it might be useful in the future, so I threw this together in an hour or so. Not entirely sure what he had in mind, but this series can go with or without this particular patch.
So I think Andrew's suggestion still applies: drop the patch for now, until it finds its user.