Skip to contentSkip to conversation

fs/resctrl: Fix deadlock for errors during mount

PATCH Superseded Tony Luck 2 participants 5 messages May 1 – May 4 RDT - resource allocation

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

PATCH 1/1 fs/resctrl: Fix deadlock for errors during mount

Sashiko noticed[1] a deadlock in the resctrl mount code.

rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.

Add a boolean rdt_kill_sb_locked flag. Set it for the duration of kernfs_get_tree() and check in rdt_kill_sb() to determine if locks are already held.

Fixes 5ff193fbde20 ("x86/intel_rdt: Add basic resctrl filesystem support")
Assisted-by GitHub-Copilot:claude-sonnet-4.6
---
 fs/resctrl/rdtgroup.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
fs/resctrl/rdtgroup.c +13 -4
index 5dfdaa6f9d8f..8544020ef420 100644
@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void)
 	}
 }
 
+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */
+static bool rdt_kill_sb_locked;
+
 static int rdt_get_tree(struct fs_context *fc)
 {
 	struct rdt_fs_context *ctx = rdt_fc2context(fc);
@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
+	rdt_kill_sb_locked = true;
 	ret = kernfs_get_tree(fc);
+	rdt_kill_sb_locked = false;
 	if (ret < 0)
 		goto out_psl;
 
@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb)
 {
 	struct rdt_resource *r;
 
-	cpus_read_lock();
-	mutex_lock(&rdtgroup_mutex);
+	if (!rdt_kill_sb_locked) {
+		cpus_read_lock();
+		mutex_lock(&rdtgroup_mutex);
+	}
 
 	rdt_disable_ctx();
 
@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)
 		resctrl_arch_disable_mon();
 	resctrl_mounted = false;
 	kernfs_kill_sb(sb);
-	mutex_unlock(&rdtgroup_mutex);
-	cpus_read_unlock();
+	if (!rdt_kill_sb_locked) {
+		mutex_unlock(&rdtgroup_mutex);
+		cpus_read_unlock();
+	}
 }
 
 static struct file_system_type rdt_fs_type = {
Signature · 1 line
2.54.0
maintainer
#1 lore raw

Hi Tony,

Quoted 6 lines · Tony Luck

Sashiko noticed[1] a deadlock in the resctrl mount code.

rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.

Thank you for addressing this.

Quoted 5 lines

Add a boolean rdt_kill_sb_locked flag. Set it for the duration of kernfs_get_tree() and check in rdt_kill_sb() to determine if locks are already held.

...

fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785

Quoted 9 lines
fs/resctrl/rdtgroup.c +1 -0
index 5dfdaa6f9d8f..8544020ef420 100644
@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void)
 	}
 }
 
+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */

I interpret above to mean that every access to rdt_kill_sb_locked can be expected to be done with rdtgroup_mutex held ...

Quoted 16 lines
fs/resctrl/rdtgroup.c +4 -0
+static bool rdt_kill_sb_locked;
+
 static int rdt_get_tree(struct fs_context *fc)
 {
 	struct rdt_fs_context *ctx = rdt_fc2context(fc);
@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
+	rdt_kill_sb_locked = true;
 	ret = kernfs_get_tree(fc);
+	rdt_kill_sb_locked = false;
 	if (ret < 0)
 		goto out_psl;
 

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183

Quoted 8 lines
fs/resctrl/rdtgroup.c +3 -2
@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb)
 {
 	struct rdt_resource *r;
 
-	cpus_read_lock();
-	mutex_lock(&rdtgroup_mutex);
+	if (!rdt_kill_sb_locked) {
+		cpus_read_lock();
+		mutex_lock(&rdtgroup_mutex);

... but here clearly rdt_kill_sb_locked can be accessed without rdtgroup_mutex held.

It appears that while this change claims that rdt_kill_sb_locked is protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.

Quoted 7 lines
fs/resctrl/rdtgroup.c +1 -0
+	}
 
 	rdt_disable_ctx();
 
@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)
 		resctrl_arch_disable_mon();
 	resctrl_mounted = false;

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205

Quoted 10 lines
fs/resctrl/rdtgroup.c +4 -2
 	kernfs_kill_sb(sb);
-	mutex_unlock(&rdtgroup_mutex);
-	cpus_read_unlock();
+	if (!rdt_kill_sb_locked) {
+		mutex_unlock(&rdtgroup_mutex);
+		cpus_read_unlock();
+	}
 }
 
 static struct file_system_type rdt_fs_type = {

Did you or your AI assistant consider running kernfs_get_tree() without rdtgroup_mutex and CPU hotplug lock held? Consider, for example:

fs/resctrl/rdtgroup.c +3 -7
index 36d21652616e..9ee6295d6521 100644
@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())
@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return kernfs_get_tree(fc);
 
-out_psl:
-	rdt_pseudo_lock_release();
 out_mondata:
 	if (resctrl_arch_mon_capable())
 		kernfs_remove(kn_mondata);

This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb() called and then all rdt_get_tree()'s failure path),
* maintaining symmetry with rdt_kill_sb() by providing it the state it is expected to be called with (i.e resctrl_mounted = true).

From what I can tell it is safe to call kernfs_kill_sb() on failure of kernfs_get_tree(), but this needs to have been be considered as part of this submission anyway.

Oh, maybe there is a new lock ordering issue with this that I am missing?

Reinette

author maintainer
#2 lore raw
Quoted 28 lines · Reinette Chatre

Hi Tony,

Quoted 6 lines · Tony Luck

Sashiko noticed[1] a deadlock in the resctrl mount code.

rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.

Thank you for addressing this.

Quoted 5 lines

Add a boolean rdt_kill_sb_locked flag. Set it for the duration of kernfs_get_tree() and check in rdt_kill_sb() to determine if locks are already held.

...

Quoted 8 lines
fs/resctrl/rdtgroup.c +0 -0
index 5dfdaa6f9d8f..8544020ef420 100644
@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void)
 	}
 }
 
Quoted 4 lines

fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785

Quoted 1 line
fs/resctrl/rdtgroup.c +1 -0
+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */

I interpret above to mean that every access to rdt_kill_sb_locked can be expected to be done with rdtgroup_mutex held ...

The comment could be much more descriptive about locking and limited use case.

Quoted 19 lines
Quoted 19 lines
fs/resctrl/rdtgroup.c +4 -0
+static bool rdt_kill_sb_locked;
+
 static int rdt_get_tree(struct fs_context *fc)
 {
 	struct rdt_fs_context *ctx = rdt_fc2context(fc);
@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
+	rdt_kill_sb_locked = true;
 	ret = kernfs_get_tree(fc);
+	rdt_kill_sb_locked = false;
 	if (ret < 0)
 		goto out_psl;
 
@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb)
 {
 	struct rdt_resource *r;
 
Quoted 7 lines

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183

Quoted 5 lines
fs/resctrl/rdtgroup.c +3 -2
-	cpus_read_lock();
-	mutex_lock(&rdtgroup_mutex);
+	if (!rdt_kill_sb_locked) {
+		cpus_read_lock();
+		mutex_lock(&rdtgroup_mutex);

... but here clearly rdt_kill_sb_locked can be accessed without rdtgroup_mutex held.

A much better name for this flag would be "resctrl_mount_in_progress". With The header comment noting that it is set-and cleared inside rdtgroup_mutex protected code, it is used only in rdt_kill_sb(). This specific use case seems safe as there are only call chains leading to rdt_kill_sb():
1) Error cleanup from failure of kernfs_fill_super() within the call to kernfs_get_tree() [rdtgroup_mutex still held in this case]
2) From user call to unmount the filesystem. In which case rdt_get_tree() must have completed successfully. Any new calls are blocked from changing this flag by the early exit based on resctrl_mounted.

Quoted 22 lines

It appears that while this change claims that rdt_kill_sb_locked is protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.

Quoted 16 lines
fs/resctrl/rdtgroup.c +5 -2
+	}
 
 	rdt_disable_ctx();
 
@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)
 		resctrl_arch_disable_mon();
 	resctrl_mounted = false;
 	kernfs_kill_sb(sb);
-	mutex_unlock(&rdtgroup_mutex);
-	cpus_read_unlock();
+	if (!rdt_kill_sb_locked) {
+		mutex_unlock(&rdtgroup_mutex);
+		cpus_read_unlock();
+	}
 }
 
Quoted 4 lines

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205

Quoted 1 line
fs/resctrl/rdtgroup.c +0 -0
 static struct file_system_type rdt_fs_type = {

Did you or your AI assistant consider running kernfs_get_tree() without rdtgroup_mutex and CPU hotplug lock held? Consider, for example:

Not considered. Thanks for the suggestion ... But, see below.

Quoted 31 lines
fs/resctrl/rdtgroup.c +3 -7
index 36d21652616e..9ee6295d6521 100644
@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())
@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return kernfs_get_tree(fc);
 
-out_psl:
-	rdt_pseudo_lock_release();
 out_mondata:
 	if (resctrl_arch_mon_capable())
 		kernfs_remove(kn_mondata);
Quoted 6 lines

This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb() called and then all rdt_get_tree()'s failure path),
* maintaining symmetry with rdt_kill_sb() by providing it the state it is expected to be called with (i.e resctrl_mounted = true).

All these are excellent points in favor of this approach.

Quoted 3 lines
Quoted 2 lines

From what I can tell it is safe to call kernfs_kill_sb() on failure of kernfs_get_tree(),

but this needs to have been be considered as part of this submission anyway.

Looks OK to me too.

Quoted 1 line

Oh, maybe there is a new lock ordering issue with this that I am missing?

I can't see any lock issues.

But ... there is a problem. kernfs_get_tree() can fail for many reasons. Only the specific case of failure in kernfs_get_super() makes the cleanup call to rdt_kill_sb(). rdt_get_tree() has no way to tell from the error code from kernfs_get_tree() whether cleanup has been done.

Plausibly I could do some surgery on the kernfs subsystem to make kernfs_get_tree() take a second argument "bool *did_i_call_kill_sb". Only other user is the cgroup code. So this might not be too invasive.

Or, I could fix up the comments to justify use of "resctrl_mount_in_progress" Also fix up rdt_kill_sb() to look like this:

static void rdt_kill_sb(struct super_block *sb)

{
	if (resctrl_mount_in_progress) {
		resctrl_clean_up_failed_mount();
		return;
	}

	... existing unmount path code here ...
}

Or ... do you have some other suggestion?

Quoted 2 lines

Reinette

-Tony

maintainer
#3 lore raw

Hi Tony,

Quoted 150 lines · Tony
Quoted 28 lines · Reinette Chatre

Hi Tony,

Quoted 6 lines · Tony Luck

Sashiko noticed[1] a deadlock in the resctrl mount code.

rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.

Thank you for addressing this.

Quoted 5 lines

Add a boolean rdt_kill_sb_locked flag. Set it for the duration of kernfs_get_tree() and check in rdt_kill_sb() to determine if locks are already held.

...

Quoted 8 lines
fs/resctrl/rdtgroup.c +0 -0
index 5dfdaa6f9d8f..8544020ef420 100644
@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void)
 	}
 }
 
Quoted 4 lines

fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785

Quoted 1 line
fs/resctrl/rdtgroup.c +1 -0
+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */

I interpret above to mean that every access to rdt_kill_sb_locked can be expected to be done with rdtgroup_mutex held ...

The comment could be much more descriptive about locking and limited use case.

Quoted 19 lines
Quoted 19 lines
fs/resctrl/rdtgroup.c +4 -0
+static bool rdt_kill_sb_locked;
+
 static int rdt_get_tree(struct fs_context *fc)
 {
 	struct rdt_fs_context *ctx = rdt_fc2context(fc);
@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
+	rdt_kill_sb_locked = true;
 	ret = kernfs_get_tree(fc);
+	rdt_kill_sb_locked = false;
 	if (ret < 0)
 		goto out_psl;
 
@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb)
 {
 	struct rdt_resource *r;
 
Quoted 7 lines

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183

Quoted 5 lines
fs/resctrl/rdtgroup.c +3 -2
-	cpus_read_lock();
-	mutex_lock(&rdtgroup_mutex);
+	if (!rdt_kill_sb_locked) {
+		cpus_read_lock();
+		mutex_lock(&rdtgroup_mutex);

... but here clearly rdt_kill_sb_locked can be accessed without rdtgroup_mutex held.

A much better name for this flag would be "resctrl_mount_in_progress". With The header comment noting that it is set-and cleared inside rdtgroup_mutex protected code, it is used only in rdt_kill_sb(). This specific use case seems safe as there are only call chains leading to rdt_kill_sb():
1) Error cleanup from failure of kernfs_fill_super() within the call to kernfs_get_tree() [rdtgroup_mutex still held in this case]
2) From user call to unmount the filesystem. In which case rdt_get_tree() must have completed successfully. Any new calls are blocked from changing this flag by the early exit based on resctrl_mounted.

Quoted 22 lines

It appears that while this change claims that rdt_kill_sb_locked is protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.

Quoted 16 lines
fs/resctrl/rdtgroup.c +5 -2
+	}
 
 	rdt_disable_ctx();
 
@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)
 		resctrl_arch_disable_mon();
 	resctrl_mounted = false;
 	kernfs_kill_sb(sb);
-	mutex_unlock(&rdtgroup_mutex);
-	cpus_read_unlock();
+	if (!rdt_kill_sb_locked) {
+		mutex_unlock(&rdtgroup_mutex);
+		cpus_read_unlock();
+	}
 }
 
Quoted 4 lines

fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205

Quoted 1 line
fs/resctrl/rdtgroup.c +0 -0
 static struct file_system_type rdt_fs_type = {

Did you or your AI assistant consider running kernfs_get_tree() without rdtgroup_mutex and CPU hotplug lock held? Consider, for example:

Not considered. Thanks for the suggestion ... But, see below.

Quoted 31 lines
fs/resctrl/rdtgroup.c +3 -7
index 36d21652616e..9ee6295d6521 100644
@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())
@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	return kernfs_get_tree(fc);
 
-out_psl:
-	rdt_pseudo_lock_release();
 out_mondata:
 	if (resctrl_arch_mon_capable())
 		kernfs_remove(kn_mondata);
Quoted 6 lines

This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb() called and then all rdt_get_tree()'s failure path),
* maintaining symmetry with rdt_kill_sb() by providing it the state it is expected to be called with (i.e resctrl_mounted = true).

All these are excellent points in favor of this approach.

Quoted 3 lines
Quoted 2 lines

From what I can tell it is safe to call kernfs_kill_sb() on failure of kernfs_get_tree(),

but this needs to have been be considered as part of this submission anyway.

Looks OK to me too.

Quoted 8 lines
Quoted 1 line

Oh, maybe there is a new lock ordering issue with this that I am missing?

I can't see any lock issues.

But ... there is a problem. kernfs_get_tree() can fail for many reasons. Only the specific case of failure in kernfs_get_super() makes the cleanup call to rdt_kill_sb(). rdt_get_tree() has no way to tell from the error code from kernfs_get_tree() whether cleanup has been done.

Thanks for highlighting this.

From what I can tell, kernfs_get_tree() can fail in two places: allocation of superblock fails, in which case rdt_kill_sb() is not called, or allocation of superblock succeeded but its initialization failed, in which case rdt_kill_sb() is called.

It seems reasonable to me to expect that rdt_kill_sb() was called if the superblock was allocated. In this case kernfs_fs_context::new_sb_created is set.

Could kernfs_fs_context::new_sb_created be used instead of kernfs_get_tree() error code to determine if cleanup has been done?

Quoted 4 lines

Plausibly I could do some surgery on the kernfs subsystem to make kernfs_get_tree() take a second argument "bool *did_i_call_kill_sb". Only other user is the cgroup code. So this might not be too invasive.

It is not clear to me yet that additional flags are needed to support this.

Quoted 5 lines

Or, I could fix up the comments to justify use of "resctrl_mount_in_progress" Also fix up rdt_kill_sb() to look like this:

static void rdt_kill_sb(struct super_block *sb)

Quoted 8 lines
{
	if (resctrl_mount_in_progress) {
		resctrl_clean_up_failed_mount();
		return;
	}

	... existing unmount path code here ...
}

I find the reasoning about safe access to resctrl_mount_in_progress to be very complicated. It is not clear to me that it is required when considering existing kernfs_fs_context::new_sb_created.

Reinette