fs/resctrl: Fix deadlock for errors during mount
Tony Luck
maintainerPATCH 1/1 fs/resctrl: Fix deadlock for errors during mount
Sashiko noticed[1] a deadlock in the resctrl mount code.
rdt_get_tree() acquires rdtgroup_mutex before calling kernfs_get_tree(). If superblock setup fails inside kernfs_get_tree(), the VFS calls kill_sb on the same thread before the call returns. rdt_kill_sb() unconditionally attempts to acquire rdtgroup_mutex and deadlock occurs.
Add a boolean rdt_kill_sb_locked flag. Set it for the duration of kernfs_get_tree() and check in rdt_kill_sb() to determine if locks are already held.
--- fs/resctrl/rdtgroup.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
fs/resctrl/rdtgroup.c +13 -4
@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void) } } +/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */ +static bool rdt_kill_sb_locked; +static int rdt_get_tree (struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);
@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; + rdt_kill_sb_locked = true; ret =kernfs_get_tree (fc); + rdt_kill_sb_locked = false;if (ret <0 ) goto out_psl;
@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb) {struct rdt_resource *r; -cpus_read_lock (); -mutex_lock (&rdtgroup_mutex); +if (!rdt_kill_sb_locked) { +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex); + }rdt_disable_ctx ();
@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_mon (); resctrl_mounted = false;kernfs_kill_sb (sb); -mutex_unlock (&rdtgroup_mutex); -cpus_read_unlock (); +if (!rdt_kill_sb_locked) { +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + } }static struct file_system_type rdt_fs_type = {
Signature · 1 line
2.54.0
Reinette Chatre
maintainerHi Tony,
Quoted 6 lines · Tony Luck
Sashiko noticed[1] a deadlock in the resctrl mount code.
rdt_get_tree()acquiresrdtgroup_mutexbefore callingkernfs_get_tree(). If superblock setup fails insidekernfs_get_tree(), the VFS callskill_sbon the same thread before the call returns.rdt_kill_sb()unconditionally attempts to acquirerdtgroup_mutexand deadlock occurs.
Thank you for addressing this.
Quoted 5 lines
Add a boolean
rdt_kill_sb_lockedflag. Set it for the duration ofkernfs_get_tree()and check inrdt_kill_sb()to determine if locks are already held.
...
fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785
Quoted 9 lines
fs/resctrl/rdtgroup.c +1
-0@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void) } } +/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */
I interpret above to mean that every access to rdt_kill_sb_locked can be expected to be done with rdtgroup_mutex held ...
Quoted 16 lines
fs/resctrl/rdtgroup.c +4
-0+static bool rdt_kill_sb_locked; +static int rdt_get_tree (struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; + rdt_kill_sb_locked = true; ret =kernfs_get_tree (fc); + rdt_kill_sb_locked = false;if (ret <0 ) goto out_psl;
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183
Quoted 8 lines
fs/resctrl/rdtgroup.c +3
-2@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb) {struct rdt_resource *r; -cpus_read_lock (); -mutex_lock (&rdtgroup_mutex); +if (!rdt_kill_sb_locked) { +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex);
... but here clearly rdt_kill_sb_locked can be accessed without rdtgroup_mutex held.
It appears that while this change claims that rdt_kill_sb_locked is protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.
Quoted 7 lines
fs/resctrl/rdtgroup.c +1
-0+ }rdt_disable_ctx ();@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_mon (); resctrl_mounted = false;
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205
Quoted 10 lines
fs/resctrl/rdtgroup.c +4
-2kernfs_kill_sb (sb); -mutex_unlock (&rdtgroup_mutex); -cpus_read_unlock (); +if (!rdt_kill_sb_locked) { +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + } }static struct file_system_type rdt_fs_type = {
Did you or your AI assistant consider running kernfs_get_tree() without rdtgroup_mutex and CPU hotplug lock held? Consider, for example:
fs/resctrl/rdtgroup.c +3 -7
@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; - ret =kernfs_get_tree (fc); -if (ret <0 ) - goto out_psl; -if (resctrl_arch_alloc_capable ())resctrl_arch_enable_alloc ();if (resctrl_arch_mon_capable ())
@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc) RESCTRL_PICK_ANY_CPU); } - goto out; +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return kernfs_get_tree (fc); -out_psl: -rdt_pseudo_lock_release (); out_mondata:if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata);
This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb() called and then all rdt_get_tree()'s failure path),
* maintaining symmetry with rdt_kill_sb() by providing it the state it is expected to be called with (i.e resctrl_mounted = true).
From what I can tell it is safe to call kernfs_kill_sb() on failure of kernfs_get_tree(), but this needs to have been be considered as part of this submission anyway.
Oh, maybe there is a new lock ordering issue with this that I am missing?
Reinette
Luck, Tony
maintainerQuoted 28 lines · Reinette Chatre
Hi Tony,
Quoted 6 lines · Tony Luck
Sashiko noticed[1] a deadlock in the resctrl mount code.
rdt_get_tree()acquiresrdtgroup_mutexbefore callingkernfs_get_tree(). If superblock setup fails insidekernfs_get_tree(), the VFS callskill_sbon the same thread before the call returns.rdt_kill_sb()unconditionally attempts to acquirerdtgroup_mutexand deadlock occurs.Thank you for addressing this.
Quoted 5 lines
Add a boolean
rdt_kill_sb_lockedflag. Set it for the duration ofkernfs_get_tree()and check inrdt_kill_sb()to determine if locks are already held....
Quoted 8 lines
fs/resctrl/rdtgroup.c +0
-0@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void) } }
Quoted 4 lines
fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785
Quoted 1 line
fs/resctrl/rdtgroup.c +1
-0+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */ I interpret above to mean that every access to
rdt_kill_sb_lockedcan be expected to be done withrdtgroup_mutexheld ...
The comment could be much more descriptive about locking and limited use case.
Quoted 19 lines
Quoted 19 lines
fs/resctrl/rdtgroup.c +4
-0+static bool rdt_kill_sb_locked; +static int rdt_get_tree (struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; + rdt_kill_sb_locked = true; ret =kernfs_get_tree (fc); + rdt_kill_sb_locked = false;if (ret <0 ) goto out_psl;@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb) {struct rdt_resource *r;
Quoted 7 lines
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183
Quoted 5 lines
fs/resctrl/rdtgroup.c +3
-2-cpus_read_lock (); -mutex_lock (&rdtgroup_mutex); +if (!rdt_kill_sb_locked) { +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex);... but here clearly
rdt_kill_sb_lockedcan be accessed withoutrdtgroup_mutexheld.
A much better name for this flag would be "resctrl_mount_in_progress". With The header comment noting that it is set-and cleared inside rdtgroup_mutex protected code, it is used only in rdt_kill_sb(). This specific use case seems safe as there are only call chains leading to rdt_kill_sb():
1) Error cleanup from failure of kernfs_fill_super() within the call to kernfs_get_tree() [rdtgroup_mutex still held in this case]
2) From user call to unmount the filesystem. In which case rdt_get_tree() must have completed successfully. Any new calls are blocked from changing this flag by the early exit based on resctrl_mounted.
Quoted 22 lines
It appears that while this change claims that
rdt_kill_sb_lockedis protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.Quoted 16 lines
fs/resctrl/rdtgroup.c +5
-2+ }rdt_disable_ctx ();@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_mon (); resctrl_mounted = false;kernfs_kill_sb (sb); -mutex_unlock (&rdtgroup_mutex); -cpus_read_unlock (); +if (!rdt_kill_sb_locked) { +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + } }
Quoted 4 lines
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205
Quoted 1 line
fs/resctrl/rdtgroup.c +0
-0static struct file_system_type rdt_fs_type = {Did you or your AI assistant consider running
kernfs_get_tree()withoutrdtgroup_mutexand CPU hotplug lock held? Consider, for example:
Not considered. Thanks for the suggestion ... But, see below.
Quoted 31 lines
fs/resctrl/rdtgroup.c +3
-7@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; - ret =kernfs_get_tree (fc); -if (ret <0 ) - goto out_psl; -if (resctrl_arch_alloc_capable ())resctrl_arch_enable_alloc ();if (resctrl_arch_mon_capable ())@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc) RESCTRL_PICK_ANY_CPU); } - goto out; +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return kernfs_get_tree (fc); -out_psl: -rdt_pseudo_lock_release (); out_mondata:if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata);
Quoted 6 lines
This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb()called and then all rdt_get_tree()'s failure path),
* maintaining symmetry withrdt_kill_sb()by providing it the state it is expected to be called with (i.eresctrl_mounted= true).
All these are excellent points in favor of this approach.
Quoted 3 lines
Quoted 2 lines
From what I can tell it is safe to call
kernfs_kill_sb()on failure ofkernfs_get_tree(),but this needs to have been be considered as part of this submission anyway.
Looks OK to me too.
Quoted 1 line
Oh, maybe there is a new lock ordering issue with this that I am missing?
I can't see any lock issues.
But ... there is a problem. kernfs_get_tree() can fail for many reasons. Only the specific case of failure in kernfs_get_super() makes the cleanup call to rdt_kill_sb(). rdt_get_tree() has no way to tell from the error code from kernfs_get_tree() whether cleanup has been done.
Plausibly I could do some surgery on the kernfs subsystem to make kernfs_get_tree() take a second argument "bool *did_i_call_kill_sb". Only other user is the cgroup code. So this might not be too invasive.
Or, I could fix up the comments to justify use of "resctrl_mount_in_progress" Also fix up rdt_kill_sb() to look like this:
static void rdt_kill_sb(struct super_block *sb)
{
if (resctrl_mount_in_progress) {
resctrl_clean_up_failed_mount();
return;
}
... existing unmount path code here ...
}Or ... do you have some other suggestion?
Quoted 2 lines
Reinette
-Tony
Reinette Chatre
maintainerHi Tony,
Quoted 150 lines · Tony
Quoted 28 lines · Reinette Chatre
Hi Tony,
Quoted 6 lines · Tony Luck
Sashiko noticed[1] a deadlock in the resctrl mount code.
rdt_get_tree()acquiresrdtgroup_mutexbefore callingkernfs_get_tree(). If superblock setup fails insidekernfs_get_tree(), the VFS callskill_sbon the same thread before the call returns.rdt_kill_sb()unconditionally attempts to acquirerdtgroup_mutexand deadlock occurs.Thank you for addressing this.
Quoted 5 lines
Add a boolean
rdt_kill_sb_lockedflag. Set it for the duration ofkernfs_get_tree()and check inrdt_kill_sb()to determine if locks are already held....
Quoted 8 lines
fs/resctrl/rdtgroup.c +0
-0@@ -2782,6 +2782,9 @@ static void schemata_list_destroy(void) } }Quoted 4 lines
fs/resctrl/rdtgroup.c static void schemata_list_destroy(void) · line 2785
Quoted 1 line
fs/resctrl/rdtgroup.c +1
-0+/* Protected by the serialized mount path (rdtgroup_mutex + resctrl_mounted). */ I interpret above to mean that every access to
rdt_kill_sb_lockedcan be expected to be done withrdtgroup_mutexheld ...The comment could be much more descriptive about locking and limited use case.
Quoted 19 lines
Quoted 19 lines
fs/resctrl/rdtgroup.c +4
-0+static bool rdt_kill_sb_locked; +static int rdt_get_tree (struct fs_context *fc) {struct rdt_fs_context *ctx =rdt_fc2context (fc);@@ -2855,7 +2858,9 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; + rdt_kill_sb_locked = true; ret =kernfs_get_tree (fc); + rdt_kill_sb_locked = false;if (ret <0 ) goto out_psl;@@ -3173,8 +3178,10 @@ static void rdt_kill_sb(struct super_block *sb) {struct rdt_resource *r;Quoted 7 lines
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3183
Quoted 5 lines
fs/resctrl/rdtgroup.c +3
-2-cpus_read_lock (); -mutex_lock (&rdtgroup_mutex); +if (!rdt_kill_sb_locked) { +cpus_read_lock (); +mutex_lock (&rdtgroup_mutex);... but here clearly
rdt_kill_sb_lockedcan be accessed withoutrdtgroup_mutexheld.A much better name for this flag would be "
resctrl_mount_in_progress". With The header comment noting that it is set-and cleared insiderdtgroup_mutexprotected code, it is used only inrdt_kill_sb(). This specific use case seems safe as there are only call chains leading tordt_kill_sb():
1) Error cleanup from failure ofkernfs_fill_super()within the call tokernfs_get_tree()[rdtgroup_mutexstill held in this case]
2) From user call to unmount the filesystem. In which caserdt_get_tree()must have completed successfully. Any new calls are blocked from changing this flag by the early exit based onresctrl_mounted.Quoted 22 lines
It appears that while this change claims that
rdt_kill_sb_lockedis protected the implementation instead seems to actually be "this works for the scenarios cared about here" which I understand to be based on considerations of how the filesystem code interacts with resctrl callbacks _today_.Quoted 16 lines
fs/resctrl/rdtgroup.c +5
-2+ }rdt_disable_ctx ();@@ -3189,8 +3196,10 @@ static void rdt_kill_sb(struct super_block *sb)resctrl_arch_disable_mon (); resctrl_mounted = false;kernfs_kill_sb (sb); -mutex_unlock (&rdtgroup_mutex); -cpus_read_unlock (); +if (!rdt_kill_sb_locked) { +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); + } }Quoted 4 lines
fs/resctrl/rdtgroup.c static void rdt_kill_sb(struct super_block *sb) · line 3205
Quoted 1 line
fs/resctrl/rdtgroup.c +0
-0static struct file_system_type rdt_fs_type = {Did you or your AI assistant consider running
kernfs_get_tree()withoutrdtgroup_mutexand CPU hotplug lock held? Consider, for example:Not considered. Thanks for the suggestion ... But, see below.
Quoted 31 lines
fs/resctrl/rdtgroup.c +3
-7@@ -2892,10 +2892,6 @@ static int rdt_get_tree(struct fs_context *fc)if (ret) goto out_mondata; - ret =kernfs_get_tree (fc); -if (ret <0 ) - goto out_psl; -if (resctrl_arch_alloc_capable ())resctrl_arch_enable_alloc ();if (resctrl_arch_mon_capable ())@@ -2911,10 +2907,10 @@ static int rdt_get_tree(struct fs_context *fc) RESCTRL_PICK_ANY_CPU); } - goto out; +mutex_unlock (&rdtgroup_mutex); +cpus_read_unlock (); +return kernfs_get_tree (fc); -out_psl: -rdt_pseudo_lock_release (); out_mondata:if (resctrl_arch_mon_capable ())kernfs_remove (kn_mondata);Quoted 6 lines
This seems simpler by:
* avoiding introduction of additional state (rdt_kill_sb_locked) with unclear protection,
* avoiding double-cleanup on failure (rdt_kill_sb()called and then all rdt_get_tree()'s failure path),
* maintaining symmetry withrdt_kill_sb()by providing it the state it is expected to be called with (i.eresctrl_mounted= true).All these are excellent points in favor of this approach.
Quoted 3 lines
Quoted 2 lines
From what I can tell it is safe to call
kernfs_kill_sb()on failure ofkernfs_get_tree(),but this needs to have been be considered as part of this submission anyway.
Looks OK to me too.
Quoted 8 lines
Quoted 1 line
Oh, maybe there is a new lock ordering issue with this that I am missing?
I can't see any lock issues.
But ... there is a problem.
kernfs_get_tree()can fail for many reasons. Only the specific case of failure inkernfs_get_super()makes the cleanup call tordt_kill_sb().rdt_get_tree()has no way to tell from the error code fromkernfs_get_tree()whether cleanup has been done.
Thanks for highlighting this.
From what I can tell, kernfs_get_tree() can fail in two places: allocation of superblock fails, in which case rdt_kill_sb() is not called, or allocation of superblock succeeded but its initialization failed, in which case rdt_kill_sb() is called.
It seems reasonable to me to expect that rdt_kill_sb() was called if the superblock was allocated. In this case kernfs_fs_context::new_sb_created is set.
Could kernfs_fs_context::new_sb_created be used instead of kernfs_get_tree() error code to determine if cleanup has been done?
Quoted 4 lines
Plausibly I could do some surgery on the kernfs subsystem to make
kernfs_get_tree()take a second argument "bool*did_i_call_kill_sb". Only other user is the cgroup code. So this might not be too invasive.
It is not clear to me yet that additional flags are needed to support this.
Quoted 5 lines
Or, I could fix up the comments to justify use of "
resctrl_mount_in_progress" Also fix uprdt_kill_sb()to look like this:static void
rdt_kill_sb(struct super_block *sb)
Quoted 8 lines
{
if (resctrl_mount_in_progress) {
resctrl_clean_up_failed_mount();
return;
}
... existing unmount path code here ...
}I find the reasoning about safe access to resctrl_mount_in_progress to be very complicated. It is not clear to me that it is required when considering existing kernfs_fs_context::new_sb_created.
Reinette
Luck, Tony
maintainerQuoted 3 lines
I find the reasoning about safe access to
resctrl_mount_in_progressto be very complicated. It is not clear to me that it is required when considering existingkernfs_fs_context::new_sb_created.
Reinette,
Indeed. Looks like cgroup uses this technique:
https://elixir.bootlin.com/linux/v7.0.1/source/kernel/cgroup/cgroup.c#L2297
I'll dig into this.
-Tony