Skip to contentSkip to conversation

fs/resctrl: Fix deadlock for errors during mount

PATCH In review Tony Luck 2 participants 6 messages May 4 – May 8 RDT - resource allocation

lore
Share & tools
lore

Tony Luck

author maintainer
#0 lore raw

PATCH v2 1/1 fs/resctrl: Fix deadlock for errors during mount

Sashiko noticed[1] a deadlock in the resctrl mount code.

rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.

Move the call to kernfs_get_tree() outside of locks.

If kernfs_get_tree() fails and ctx->kfc.new_sb_created is set, then rdt_kill_sb() has already been called and no further cleanup is needed.

Fixes 5ff193fbde20 ("x86/intel_rdt: Add basic resctrl filesystem support")

---

Changes since v1:

AI solution from Claude abandoned after human (Reinette) and AI (Sashiko) reviews found it lacking.

Reinette: Feel free to claim full authorship of this fix since you provided direction for every aspect. At minimum the "Suggested-by:" should be "Co-authored-by:" (with your sign-off).

 fs/resctrl/rdtgroup.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
fs/resctrl/rdtgroup.c +18 -6
index 5dfdaa6f9d8f..6709b74bd655 100644
@@ -2855,10 +2855,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())
@@ -2874,10 +2870,26 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	/* Release locks because kernfs_get_tree() may call rdt_kill_sb() */
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	ret = kernfs_get_tree(fc);
+	if (!ret || ctx->kfc.new_sb_created) {
+		/* mount succeeded, or failed and already cleaned up */
+		return ret;
+	}
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	if (resctrl_arch_alloc_capable())
+		resctrl_arch_disable_alloc();
+	if (resctrl_arch_mon_capable())
+		resctrl_arch_disable_mon();
+
+	resctrl_mounted = false;
 
-out_psl:
 	rdt_pseudo_lock_release();
+
 out_mondata:
 	if (resctrl_arch_mon_capable())
 		kernfs_remove(kn_mondata);
Signature · 1 line
2.54.0
maintainer
#1 lore raw

Hi Tony,

Quoted 16 lines · Tony Luck
fs/resctrl/rdtgroup.c +0 -4
index 5dfdaa6f9d8f..6709b74bd655 100644
@@ -2855,10 +2855,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())

fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2873

Quoted 5 lines
fs/resctrl/rdtgroup.c +1 -1
@@ -2874,10 +2870,26 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	/* Release locks because kernfs_get_tree() may call rdt_kill_sb() */

I neglected to add a rdt_last_cmd_clear() to this path now used for success and failure. It is needed on success path to ensure that the last_cmd_status file does not show stale information.

Quoted 6 lines
fs/resctrl/rdtgroup.c +6 -0
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+	ret = kernfs_get_tree(fc);
+	if (!ret || ctx->kfc.new_sb_created) {
+		/* mount succeeded, or failed and already cleaned up */
+		return ret;

fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2889

Quoted 10 lines
fs/resctrl/rdtgroup.c +10 -0
+	}
+	cpus_read_lock();
+	mutex_lock(&rdtgroup_mutex);
+
+	if (resctrl_arch_alloc_capable())
+		resctrl_arch_disable_alloc();
+	if (resctrl_arch_mon_capable())
+		resctrl_arch_disable_mon();
+
+	resctrl_mounted = false;

I find symmetrical code significantly easier to reason about while the above introduces asymmetrical code in two ways:
- kernfs_get_tree() failures have identical resctrl state but the resctrl cleanup is done differently (rdt_kill_sb() or rdt_get_tree() error path) based on the specific failure that has nothing to do with resctrl. I think it will be simpler if resctrl state is cleaned up consistently irrespective of how kernfs_get_tree() fails.
- Above change creates inconsistency in how kernfs_get_tree() is called without locks but its "partner" function kernfs_kill_sb() is called with both CPU hotplug lock and rdtgroup_mutex held. Here too I believe it will be simpler to just keep locking consistent when interacting with these kernfs calls.

fs/resctrl/rdtgroup.c static int rdt_get_tree(struct fs_context *fc) · line 2894

Quoted 7 lines
fs/resctrl/rdtgroup.c +1 -1
-out_psl:
 	rdt_pseudo_lock_release();
+
 out_mondata:
 	if (resctrl_arch_mon_capable())
 		kernfs_remove(kn_mondata);

While comparing rdt_kill_sb() and this error exit path to make sure they do the same I seem to have stumbled on a bug where out_mondata is missing mon_put_kn_priv(). Do you agree?

Putting all comments together, how about something like below?

fs/resctrl/rdtgroup.c +28 -11
index 5dfdaa6f9d8f..6d0d6ab34985 100644
@@ -73,6 +73,11 @@ static char last_cmd_status_buf[512];
 static int rdtgroup_setup_root(struct rdt_fs_context *ctx);
 
 static void rdtgroup_destroy_root(void);
+/*
+ * Temporary forward declaration for testing only. Move functions instead.
+ */
+static void resctrl_unmount(void);
+static void mon_put_kn_priv(void);
 
 struct dentry *debugfs_resctrl;
 
@@ -2855,10 +2860,6 @@ static int rdt_get_tree(struct fs_context *fc)
 	if (ret)
 		goto out_mondata;
 
-	ret = kernfs_get_tree(fc);
-	if (ret < 0)
-		goto out_psl;
-
 	if (resctrl_arch_alloc_capable())
 		resctrl_arch_enable_alloc();
 	if (resctrl_arch_mon_capable())
@@ -2874,13 +2875,24 @@ static int rdt_get_tree(struct fs_context *fc)
 						   RESCTRL_PICK_ANY_CPU);
 	}
 
-	goto out;
+	rdt_last_cmd_clear();
+	mutex_unlock(&rdtgroup_mutex);
+	cpus_read_unlock();
+
+	ret = kernfs_get_tree(fc);
+	/*
+	 * resctrl can only be mounted once, new superblock only expected
+	 * to be created once.
+	 */
+	if (!ctx->kfc.new_sb_created)
+		resctrl_unmount();
+	return ret;
 
-out_psl:
-	rdt_pseudo_lock_release();
 out_mondata:
-	if (resctrl_arch_mon_capable())
+	if (resctrl_arch_mon_capable()) {
 		kernfs_remove(kn_mondata);
+		mon_put_kn_priv(); /* separate fix */
+	}
 out_mongrp:
 	if (resctrl_arch_mon_capable()) {
 		rdtgroup_unassign_cntrs(&rdtgroup_default);
@@ -2896,7 +2908,6 @@ static int rdt_get_tree(struct fs_context *fc)
 out_root:
 	rdtgroup_destroy_root();
 out:
-	rdt_last_cmd_clear();
 	mutex_unlock(&rdtgroup_mutex);
 	cpus_read_unlock();
 	return ret;
@@ -3169,7 +3180,7 @@ static void resctrl_fs_teardown(void)
 	rdtgroup_destroy_root();
 }
 
-static void rdt_kill_sb(struct super_block *sb)
+static void resctrl_unmount(void)
 {
 	struct rdt_resource *r;
 
@@ -3188,11 +3199,17 @@ static void rdt_kill_sb(struct super_block *sb)
 	if (resctrl_arch_mon_capable())
 		resctrl_arch_disable_mon();
 	resctrl_mounted = false;
-	kernfs_kill_sb(sb);
 	mutex_unlock(&rdtgroup_mutex);
 	cpus_read_unlock();
 }
 
+
+static void rdt_kill_sb(struct super_block *sb)
+{
+	resctrl_unmount();
+	kernfs_kill_sb(sb);
+}
+
 static struct file_system_type rdt_fs_type = {
 	.name			= "resctrl",
 	.init_fs_context	= rdt_init_fs_context,

Reinette

author maintainer
#2 lore raw

Reinette,

This looks promising.

I'll split out the missing call to mon_put_kn_priv(); into its own patch before the deadlock fix.

Going to re-run the tests I did before forcing kernfs_get_tree() to fail early without setting new_sb_created, and also late.

fs/resctrl/rdtgroup.c static char last_cmd_status_buf[512]; · line 80

Quoted 5 lines
fs/resctrl/rdtgroup.c +5 -0
+/*
+ * Temporary forward declaration for testing only. Move functions instead.
+ */
+static void resctrl_unmount(void);
+static void mon_put_kn_priv(void);

Question: How much are forward declarations hated? And how to handle this?

Moving the functions around in the same patch really obscures the actual change. Is it OK to have a patch to make the functional change including the forward declarations. Then a separate commit that does the re-order (where it is obvious that functions are being picked up and moved without any code changes)?

-Tony

maintainer
#3 lore raw

Hi Tony,

Quoted 9 lines · Tony

Reinette,

This looks promising.

I'll split out the missing call to mon_put_kn_priv(); into its own patch before the deadlock fix.

Going to re-run the tests I did before forcing kernfs_get_tree() to fail early without setting new_sb_created, and also late.

Thank you very much.

Quoted 9 lines

fs/resctrl/rdtgroup.c static char last_cmd_status_buf[512]; · line 80

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+/*
+ * Temporary forward declaration for testing only. Move functions instead.
+ */
+static void resctrl_unmount(void);
+static void mon_put_kn_priv(void);

Question: How much are forward declarations hated? And how to handle this?

I am not aware of forward declarations being hated and I am not aware of documented tip rules about this. Even so, I do find it cleaner if they can be avoided. To handle this a prep patch that just moves the code without any functional change should work?

Quoted 5 lines

Moving the functions around in the same patch really obscures the actual change. Is it OK to have a patch to make the functional change including the forward declarations. Then a separate commit that does the re-order (where it is obvious that functions are being picked up and moved without any code changes)?

My preference would be a prep patch that does the move with new capability built on top. It seems unnecessary to me to add a forward declaration in one patch just to remove it in a following patch. I agree that moving code as part of functional change should be avoided.

Reinette

author maintainer
#4 lore raw
Quoted 7 lines · Reinette Chatre

Hi Tony,

Quoted 4 lines · Tony

Reinette,

This looks promising.

Quoted 7 lines
Quoted 5 lines

I'll split out the missing call to mon_put_kn_priv(); into its own patch before the deadlock fix.

Going to re-run the tests I did before forcing kernfs_get_tree() to fail early without setting new_sb_created, and also late.

Thank you very much.

Tests pass.

Quoted 21 lines
Quoted 9 lines

fs/resctrl/rdtgroup.c static char last_cmd_status_buf[512]; · line 80

Quoted 6 lines
fs/resctrl/rdtgroup.c +5 -0
+/*
+ * Temporary forward declaration for testing only. Move functions instead.
+ */
+static void resctrl_unmount(void);
+static void mon_put_kn_priv(void);

Question: How much are forward declarations hated? And how to handle this?

I am not aware of forward declarations being hated and I am not aware of documented tip rules about this. Even so, I do find it cleaner if they can be avoided. To handle this a prep patch that just moves the code without any functional change should work?

Quoted 5 lines

Moving the functions around in the same patch really obscures the actual change. Is it OK to have a patch to make the functional change including the forward declarations. Then a separate commit that does the re-order (where it is obvious that functions are being picked up and moved without any code changes)?

Quoted 4 lines

My preference would be a prep patch that does the move with new capability built on top. It seems unnecessary to me to add a forward declaration in one patch just to remove it in a following patch. I agree that moving code as part of functional change should be avoided.

OK. I have a three patch series:

1) Pure code move, no changes [while I picked up a block of functions and moved them earlier, "git show" presents the move in terms of a different set of functions moving]

2) Add missing mon_put_kn_priv()

3) Fix the mount error deadlock

Internal AI scans have only two complaints. Both appear spurious;

First it says that:

	if (!ctx->kfc.new_sb_created)
		resctrl_unmount();

should be:

	if (ret && !ctx->kfc.new_sb_created)
		resctrl_unmount();

It admits that current code can't return ret == 0 (success) while failing to create a super block. But argues in some twisted future that might be possible.

Second:

Code used to call rdt_last_cmd_clear() unconditionally on success or failure of the mount. Now the call for the end-of-function error path is gone.

This is true, but it doesn't matter. If the filesystem did not mount then there is no /sys/fs/resctrl/info/last_cmd_status file to leak an old error string.

Quoted 2 lines

Reinette

I just need to write up the cover letter.

-Tony

maintainer
#5 lore raw

Hi Tony,

Quoted 9 lines · Tony
Quoted 8 lines · Reinette Chatre
Quoted 7 lines · Tony

Reinette,

This looks promising.

I'll split out the missing call to mon_put_kn_priv(); into its own patch before the deadlock fix.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Going to re-run the tests I did before forcing kernfs_get_tree() to fail early without setting new_sb_created, and also late.

Thank you very much.

Tests pass.

Great. Thank you very much.

Quoted 27 lines
Quoted 22 lines
Quoted 10 lines

fs/resctrl/rdtgroup.c static char last_cmd_status_buf[512]; · line 80

Quoted 7 lines
fs/resctrl/rdtgroup.c +5 -0
+/*
+ * Temporary forward declaration for testing only. Move functions instead.
+ */
+static void resctrl_unmount(void);
+static void mon_put_kn_priv(void);

Question: How much are forward declarations hated? And how to handle this?

I am not aware of forward declarations being hated and I am not aware of documented tip rules about this. Even so, I do find it cleaner if they can be avoided. To handle this a prep patch that just moves the code without any functional change should work?

Quoted 5 lines

Moving the functions around in the same patch really obscures the actual change. Is it OK to have a patch to make the functional change including the forward declarations. Then a separate commit that does the re-order (where it is obvious that functions are being picked up and moved without any code changes)?

Quoted 4 lines

My preference would be a prep patch that does the move with new capability built on top. It seems unnecessary to me to add a forward declaration in one patch just to remove it in a following patch. I agree that moving code as part of functional change should be avoided.

Quoted 5 lines

OK. I have a three patch series:

1) Pure code move, no changes [while I picked up a block of functions and moved them earlier, "git show" presents the move in terms of a different set of functions moving]

If the patches seem awkward it may be worthwhile to try --patience or --histogram to see if readability improves.

Quoted 11 lines

2) Add missing mon_put_kn_priv()

3) Fix the mount error deadlock

Internal AI scans have only two complaints. Both appear spurious;

First it says that:

	if (!ctx->kfc.new_sb_created)
		resctrl_unmount();
Quoted 8 lines

should be:

	if (ret && !ctx->kfc.new_sb_created)
		resctrl_unmount();

It admits that current code can't return ret == 0 (success) while failing to create a super block. But argues in some twisted future that might be possible.

I think kernfs_get_tree() can indeed return 0 without creating a superblock when this is a re-mount but since resctrl does not call kernfs_get_tree() on a remount (instead returning -EBUSY) this should not happen. If indeed something changes with kernfs API to cause this to happen then it would be safer to return success with state than return success with all state removed.

While the change is not necessary in current flow it does improve robustness while making the flow easier to understand so I am ok with it. I think a comment above the check will be helpful though.

Quoted 10 lines

Second:

Code used to call rdt_last_cmd_clear() unconditionally on success or failure of the mount. Now the call for the end-of-function error path is gone.

This is true, but it doesn't matter. If the filesystem did not mount then there is no /sys/fs/resctrl/info/last_cmd_status file to leak an old error string.

ack. I see this similar to unmount where this state is not cleared either.

Quoted 2 lines

I just need to write up the cover letter.

Thank you very much.

Reinette