cgroup: fix deadlock on cgroup_mutex via drop_parsed_module_refcounts()

PATCH In review Tejun Heo 1 participant 5 messages Jun 28, 2013 – Jun 29, 2013 2 messages not in the archive cgroup

Share & tools

Tejun Heo

author

2013-06-28 02:39 UTC #0 lore raw

PATCH 1/2 cgroup: fix deadlock on cgroup_mutex via drop_parsed_module_refcounts()

From e2bd416f6246d11be29999c177d2534943a5c2df Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Thu, 27 Jun 2013 19:37:23 -0700

eb178d06332 ("cgroup: grab cgroup_mutex in drop_parsed_module_refcounts()") made drop_parsed_module_refcounts() grab cgroup_mutex to make lockdep assertion in for_each_subsys() happy. Unfortunately, cgroup_remount() calls the function while holding cgroup_mutex in its failure path leading to the following deadlock.

# mount -t cgroup -o remount,memory,blkio cgroup blkio

cgroup: option changes via remount are deprecated (pid=525 comm=mount)

=============================================
[ INFO: possible recursive locking detected ]
3.10.0-rc4-work+ #1 Not tainted
---------------------------------------------
mount/525 is trying to acquire lock:
(cgroup_mutex){+.+.+.}, at: [<ffffffff8110a3e1>] drop_parsed_module_refcounts+0x21/0xb0

but task is already holding lock:
(cgroup_mutex){+.+.+.}, at: [<ffffffff8110e4e1>] cgroup_remount+0x51/0x200

other info that might help us debug this:

  Possible unsafe locking scenario:

	CPU0
	----

lock(cgroup_mutex);
lock(cgroup_mutex);

*** DEADLOCK ***

May be due to missing lock nesting notation

4 locks held by mount/525:

  #0:  (&type->s_umount_key#30){+.+...}, at: [<ffffffff811e9a0d>] do_mount+0x2bd/0xa30
  #1:  (&sb->s_type->i_mutex_key#9){+.+.+.}, at: [<ffffffff8110e4d3>] cgroup_remount+0x43/0x200
  #2:  (cgroup_mutex){+.+.+.}, at: [<ffffffff8110e4e1>] cgroup_remount+0x51/0x200
  #3:  (cgroup_root_mutex){+.+.+.}, at: [<ffffffff8110e4ef>] cgroup_remount+0x5f/0x200

 stack backtrace:
 CPU: 2 PID: 525 Comm: mount Not tainted 3.10.0-rc4-work+ #1
 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
  ffffffff829651f0 ffff88000ec2fc28 ffffffff81c24bb1 ffff88000ec2fce8
  ffffffff810f420d 0000000000000006 0000000000000001 0000000000000056
  ffff8800153b4640 ffff880000000000 ffffffff81c2e468 ffff8800153b4640
 Call Trace:
  [<ffffffff81c24bb1>] dump_stack+0x19/0x1b
  [<ffffffff810f420d>] __lock_acquire+0x15dd/0x1e60
  [<ffffffff810f531c>] lock_acquire+0x9c/0x1f0
  [<ffffffff81c2a805>] mutex_lock_nested+0x65/0x410
  [<ffffffff8110a3e1>] drop_parsed_module_refcounts+0x21/0xb0
  [<ffffffff8110e63e>] cgroup_remount+0x1ae/0x200
  [<ffffffff811c9bb2>] do_remount_sb+0x82/0x190
  [<ffffffff811e9d41>] do_mount+0x5f1/0xa30
  [<ffffffff811ea203>] SyS_mount+0x83/0xc0
  [<ffffffff81c2fb82>] system_call_fastpath+0x16/0x1b

Fix it by moving the drop_parsed_module_refcounts() invocation outside cgroup_mutex.

Signed-off-by Tejun Heo <[email protected]>

---
Two patches to fix regressions that I introduced lately. Given that -rc1 is imminent, I applied them to for-3.11 directly so that they can get exposure in -next ASAP.

Thanks!

 kernel/cgroup.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

kernel/cgroup.c +2 -2

index 4ed8677..1b7b567 100644
@@ -1365,7 +1365,6 @@ static int cgroup_remount(struct super_block *sb, int *flags, char *data)
 	if (opts.flags != root->flags ||
 	    (opts.name && strcmp(opts.name, root->name))) {
 		ret = -EINVAL;
-		drop_parsed_module_refcounts(opts.subsys_mask);
 		goto out_unlock;
 	}

@@ -1380,7 +1379,6 @@ static int cgroup_remount(struct super_block *sb, int *flags, char *data)
 	if (ret) {
 		/* rebind_subsystems failed, re-populate the removed files */
 		cgroup_populate_dir(cgrp, false, removed_mask);
-		drop_parsed_module_refcounts(opts.subsys_mask);
 		goto out_unlock;
 	}

@@ -1395,6 +1393,8 @@ static int cgroup_remount(struct super_block *sb, int *flags, char *data)
 	mutex_unlock(&cgroup_root_mutex);
 	mutex_unlock(&cgroup_mutex);
 	mutex_unlock(&cgrp->dentry->d_inode->i_mutex);
+	if (ret)
+		drop_parsed_module_refcounts(opts.subsys_mask);
 	return ret;
 }

Signature · 1 line

1.8.3.1

A message that is not in the archive (sent off-list or not yet received): 20130628023930.GA2500-Gd/HAXX7CRxy/[email protected]

Tejun Heo

author

2013-06-28 02:40 UTC #2 lore raw

PATCH 2/2 cgroup: CGRP_ROOT_SUBSYS_BOUND should be ignored when comparing mount options

From 0ce6cba35777cf96a54ce0d5856dc962566b8717 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Thu, 27 Jun 2013 19:37:26 -0700

1672d04070 ("cgroup: fix cgroupfs_root early destruction path") introduced CGRP_ROOT_SUBSYS_BOUND which is used to mark completion of subsys binding on a new root; however, this broke remounts. cgroup_remount() doesn't allow changing root options via remount and CGRP_ROOT_SUBSYS_BOUND, which is set on all fully initialized roots, makes the function reject all remounts.

Fix it by putting the options part in the lower 16 bits of root->flags and masking the comparions. While at it, make cgroup_remount() emit an error message explaining why it's rejecting a remount request, so that it's less of a mystery.

Signed-off-by Tejun Heo <[email protected]>

---
 include/linux/cgroup.h | 6 +++++-
 kernel/cgroup.c        | 5 ++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

include/linux/cgroup.h +5 -1

index ad3555b..8db5397 100644
@@ -276,7 +276,11 @@ enum {
 
 	CGRP_ROOT_NOPREFIX	= (1 << 1), /* mounted subsystems have no named prefix */
 	CGRP_ROOT_XATTR		= (1 << 2), /* supports extended attributes */
-	CGRP_ROOT_SUBSYS_BOUND	= (1 << 3), /* subsystems finished binding */
+
+	/* mount options live below bit 16 */
+	CGRP_ROOT_OPTION_MASK	= (1 << 16) - 1,
+
+	CGRP_ROOT_SUBSYS_BOUND	= (1 << 16), /* subsystems finished binding */
 };
 
 /*

kernel/cgroup.c +4 -1

index 1b7b567..5a2fcf5 100644
@@ -1362,8 +1362,11 @@ static int cgroup_remount(struct super_block *sb, int *flags, char *data)
 	removed_mask = root->subsys_mask & ~opts.subsys_mask;
 
 	/* Don't allow flags or name to change at remount */
-	if (opts.flags != root->flags ||
+	if (((opts.flags ^ root->flags) & CGRP_ROOT_OPTION_MASK) ||
 	    (opts.name && strcmp(opts.name, root->name))) {
+		pr_err("cgroup: option or name mismatch, new: 0x%lx \"%s\", old: 0x%lx \"%s\"\n",
+		       opts.flags & CGRP_ROOT_OPTION_MASK, opts.name ?: "",
+		       root->flags & CGRP_ROOT_OPTION_MASK, root->name);
 		ret = -EINVAL;
 		goto out_unlock;
 	}

Signature · 1 line

1.8.3.1

A message that is not in the archive (sent off-list or not yet received): 20130628024003.GB2500-Gd/HAXX7CRxy/[email protected]

Tejun Heo

author

2013-06-29 21:17 UTC #4 lore raw

[PATCH cgroup/for-3.11 3/2] cgroup: CGRP_ROOT_SUBSYS_BOUND should also be ignored when mounting an existing hierarchy

From c7ba8287cd11f2fc9e2feee9e1fac34b7293658f Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Sat, 29 Jun 2013 14:06:10 -0700

0ce6cba357 ("cgroup: CGRP_ROOT_SUBSYS_BOUND should be ignored when comparing mount options") only updated the remount path but CGRP_ROOT_SUBSYS_BOUND should also be ignored when comparing options while mounting an existing hierarchy. As option mismatch triggers a warning but doesn't fail the mount without sane_behavior, this only triggers a spurious warning message.

Fix it by only comparing CGRP_ROOT_OPTION_MASK bits when comparing new and existing root options.

Signed-off-by Tejun Heo <[email protected]>

---
Oops, missed one. This one isn't critical but let's get it fixed too.

Thanks.

 kernel/cgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/cgroup.c +1 -1

index 5a2fcf5..e5583d1 100644
@@ -1703,7 +1703,7 @@ static struct dentry *cgroup_mount(struct file_system_type *fs_type,
 		 */
 		cgroup_free_root(opts.new_root);
 
-		if (root->flags != opts.flags) {
+		if ((root->flags ^ opts.flags) & CGRP_ROOT_OPTION_MASK) {
 			if ((root->flags | opts.flags) & CGRP_ROOT_SANE_BEHAVIOR) {
 				pr_err("cgroup: sane_behavior: new mount options should match the existing superblock\n");
 				ret = -EINVAL;

Signature · 1 line

1.8.3.1

Tejun Heo

author

2013-06-29 21:17 UTC #5 lore raw

[PATCH cgroup/for-3.11 3/2] cgroup: CGRP_ROOT_SUBSYS_BOUND should also be ignored when mounting an existing hierarchy

In reply to a message not in the archive

From c7ba8287cd11f2fc9e2feee9e1fac34b7293658f Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Sat, 29 Jun 2013 14:06:10 -0700

Fix it by only comparing CGRP_ROOT_OPTION_MASK bits when comparing new and existing root options.

Signed-off-by Tejun Heo <[email protected]>

---
Oops, missed one. This one isn't critical but let's get it fixed too.

Thanks.

 kernel/cgroup.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/cgroup.c +1 -1

index 5a2fcf5..e5583d1 100644
@@ -1703,7 +1703,7 @@ static struct dentry *cgroup_mount(struct file_system_type *fs_type,
 		 */
 		cgroup_free_root(opts.new_root);
 
-		if (root->flags != opts.flags) {
+		if ((root->flags ^ opts.flags) & CGRP_ROOT_OPTION_MASK) {
 			if ((root->flags | opts.flags) & CGRP_ROOT_SANE_BEHAVIOR) {
 				pr_err("cgroup: sane_behavior: new mount options should match the existing superblock\n");
 				ret = -EINVAL;

Signature · 1 line

1.8.3.1

Tejun Heo

author

2013-06-28 02:40 UTC #6 lore raw

PATCH 2/2 cgroup: CGRP_ROOT_SUBSYS_BOUND should be ignored when comparing mount options

In reply to a message not in the archive

From 0ce6cba35777cf96a54ce0d5856dc962566b8717 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Thu, 27 Jun 2013 19:37:26 -0700

Signed-off-by Tejun Heo <[email protected]>

---
 include/linux/cgroup.h | 6 +++++-
 kernel/cgroup.c        | 5 ++++-
 2 files changed, 9 insertions(+), 2 deletions(-)

include/linux/cgroup.h +5 -1

index ad3555b..8db5397 100644
@@ -276,7 +276,11 @@ enum {
 
 	CGRP_ROOT_NOPREFIX	= (1 << 1), /* mounted subsystems have no named prefix */
 	CGRP_ROOT_XATTR		= (1 << 2), /* supports extended attributes */
-	CGRP_ROOT_SUBSYS_BOUND	= (1 << 3), /* subsystems finished binding */
+
+	/* mount options live below bit 16 */
+	CGRP_ROOT_OPTION_MASK	= (1 << 16) - 1,
+
+	CGRP_ROOT_SUBSYS_BOUND	= (1 << 16), /* subsystems finished binding */
 };
 
 /*

kernel/cgroup.c +4 -1

index 1b7b567..5a2fcf5 100644
@@ -1362,8 +1362,11 @@ static int cgroup_remount(struct super_block *sb, int *flags, char *data)
 	removed_mask = root->subsys_mask & ~opts.subsys_mask;
 
 	/* Don't allow flags or name to change at remount */
-	if (opts.flags != root->flags ||
+	if (((opts.flags ^ root->flags) & CGRP_ROOT_OPTION_MASK) ||
 	    (opts.name && strcmp(opts.name, root->name))) {
+		pr_err("cgroup: option or name mismatch, new: 0x%lx \"%s\", old: 0x%lx \"%s\"\n",
+		       opts.flags & CGRP_ROOT_OPTION_MASK, opts.name ?: "",
+		       root->flags & CGRP_ROOT_OPTION_MASK, root->name);
 		ret = -EINVAL;
 		goto out_unlock;
 	}

Signature · 1 line

1.8.3.1