ocfs2: Fix deadlocks in quota recovery
cover ocfs2: Fix deadlocks in quota recovery
Hello,
this implements another approach to fixing quota recovery deadlocks. We avoid grabbing sb->s_umount semaphore from ocfs2_finish_quota_recovery() and instead stop quota recovery early in ocfs2_dismount_volume(). Please review and test, the series has been only lightly tested in local mode as I don't have proper OCFS2 test setup.
Honza
PATCH 1/3 ocfs2: Switch osb->disable_recovery to enum
We will need more recovery states than just pure enable / disable to fix deadlocks with quota recovery. Switch osb->disable_recovery to enum.
--- fs/ocfs2/journal.c | 14 ++++++++------ fs/ocfs2/ocfs2.h | 7 ++++++- 2 files changed, 14 insertions(+), 7 deletions(-)
fs/ocfs2/journal.c +8 -6
@@ -174,7 +174,7 @@ int ocfs2_recovery_init(struct ocfs2_super *osb)struct ocfs2_recovery_map *rm;mutex_init (&osb->recovery_lock); - osb->disable_recovery =0 ; + osb->recovery_state =OCFS2_REC_ENABLED ; osb->recovery_thread_task =NULL ;init_waitqueue_head (&osb->recovery_event);
@@ -206,7 +206,7 @@ void ocfs2_recovery_exit(struct ocfs2_super *osb)/* disable any new recovery threads and wait for any currently * running ones to exit. Do this before setting the vol_state. */ mutex_lock (&osb->recovery_lock); - osb->disable_recovery =1 ; + osb->recovery_state =OCFS2_REC_DISABLED ;mutex_unlock (&osb->recovery_lock);wait_event (osb->recovery_event, !ocfs2_recovery_thread_running (osb));
@@ -1582,14 +1582,16 @@ static int __ocfs2_recovery_thread(void *arg)void ocfs2_recovery_thread (struct ocfs2_super *osb,int node_num) { +int was_set =-1 ; +mutex_lock (&osb->recovery_lock); +if (osb->recovery_state <OCFS2_REC_DISABLED ) + was_set =ocfs2_recovery_map_set (osb, node_num);trace_ocfs2_recovery_thread (node_num, osb->node_num, - osb->disable_recovery, osb->recovery_thread_task, - osb->disable_recovery ? --1 :ocfs2_recovery_map_set (osb, node_num)); + osb->recovery_state, osb->recovery_thread_task, was_set); -if (osb->disable_recovery) +if (osb->recovery_state ==OCFS2_REC_DISABLED ) goto out;if (osb->recovery_thread_task)
fs/ocfs2/ocfs2.h +6 -1
@@ -308,6 +308,11 @@ enum ocfs2_journal_trigger_type {void ocfs2_initialize_journal_triggers (struct super_block *sb,struct ocfs2_triggers triggers[]); +enum ocfs2_recovery_state { +OCFS2_REC_ENABLED =0 , +OCFS2_REC_DISABLED , +}; +struct ocfs2_journal ;struct ocfs2_slot_info ;struct ocfs2_recovery_map ;
@@ -370,7 +375,7 @@ struct ocfs2_superstruct ocfs2_recovery_map *recovery_map;struct ocfs2_replay_map *replay_map;struct task_struct *recovery_thread_task; -int disable_recovery; +enum ocfs2_recovery_state recovery_state;wait_queue_head_t checkpoint_event;struct ocfs2_journal *journal;unsigned long osb_commit_interval;
Signature · 1 line
2.43.0
Re: [PATCH 1/3] ocfs2: Switch osb->disable_recovery to enum
Quoted 4 lines · Jan Kara
We will need more recovery states than just pure enable / disable to fix deadlocks with quota recovery. Switch
osb->disable_recoveryto enum.Signed-off-by Jan Kara <[email protected]>
Quoted 73 lines
--- fs/ocfs2/journal.c | 14 ++++++++------ fs/ocfs2/ocfs2.h | 7 ++++++- 2 files changed, 14 insertions(+), 7 deletions(-)fs/ocfs2/journal.c +8
-6@@ -174,7 +174,7 @@ int ocfs2_recovery_init(struct ocfs2_super *osb)struct ocfs2_recovery_map *rm;mutex_init (&osb->recovery_lock); - osb->disable_recovery =0 ; + osb->recovery_state =OCFS2_REC_ENABLED ; osb->recovery_thread_task =NULL ;init_waitqueue_head (&osb->recovery_event);@@ -206,7 +206,7 @@ void ocfs2_recovery_exit(struct ocfs2_super *osb)/* disable any new recovery threads and wait for any currently * running ones to exit. Do this before setting the vol_state. */ mutex_lock (&osb->recovery_lock); - osb->disable_recovery =1 ; + osb->recovery_state =OCFS2_REC_DISABLED ;mutex_unlock (&osb->recovery_lock);wait_event (osb->recovery_event, !ocfs2_recovery_thread_running (osb));@@ -1582,14 +1582,16 @@ static int __ocfs2_recovery_thread(void *arg)void ocfs2_recovery_thread (struct ocfs2_super *osb,int node_num) { +int was_set =-1 ; +mutex_lock (&osb->recovery_lock); +if (osb->recovery_state <OCFS2_REC_DISABLED ) + was_set =ocfs2_recovery_map_set (osb, node_num);trace_ocfs2_recovery_thread (node_num, osb->node_num, - osb->disable_recovery, osb->recovery_thread_task, - osb->disable_recovery ? --1 :ocfs2_recovery_map_set (osb, node_num)); + osb->recovery_state, osb->recovery_thread_task, was_set); -if (osb->disable_recovery) +if (osb->recovery_state ==OCFS2_REC_DISABLED ) goto out;if (osb->recovery_thread_task)fs/ocfs2/ocfs2.h +6
-1@@ -308,6 +308,11 @@ enum ocfs2_journal_trigger_type {void ocfs2_initialize_journal_triggers (struct super_block *sb,struct ocfs2_triggers triggers[]); +enum ocfs2_recovery_state { +OCFS2_REC_ENABLED =0 , +OCFS2_REC_DISABLED , +}; +struct ocfs2_journal ;struct ocfs2_slot_info ;struct ocfs2_recovery_map ;@@ -370,7 +375,7 @@ struct ocfs2_superstruct ocfs2_recovery_map *recovery_map;struct ocfs2_replay_map *replay_map;struct task_struct *recovery_thread_task; -int disable_recovery; +enum ocfs2_recovery_state recovery_state;wait_queue_head_t checkpoint_event;struct ocfs2_journal *journal;unsigned long osb_commit_interval;
PATCH 2/3 ocfs2: Implement handshaking with ocfs2 recovery thread
In reply to Jan KaraWe will need ocfs2 recovery thread to acknowledge transitions of recovery_state when disabling particular types of recovery. This is similar to what currently happens when disabling recovery completely, just more general. Implement the handshake and use it for exit from recovery.
--- fs/ocfs2/journal.c | 52 +++++++++++++++++++++++++++++++--------------- fs/ocfs2/ocfs2.h | 4 ++++ 2 files changed, 39 insertions(+), 17 deletions(-)
fs/ocfs2/journal.c +35 -17
@@ -190,31 +190,48 @@ int ocfs2_recovery_init(struct ocfs2_super *osb)return 0 ; } -/* we can't grab the goofy sem lock from inside wait_event, so we use -* memory barriers to make sure that we'll see the null task before -* being woken up */ static int ocfs2_recovery_thread_running (struct ocfs2_super *osb) { -mb ();return osb->recovery_thread_task !=NULL ; } -void ocfs2_recovery_exit (struct ocfs2_super *osb) +static void ocfs2_recovery_disable (struct ocfs2_super *osb, +enum ocfs2_recovery_state state) { -struct ocfs2_recovery_map *rm; - -/* disable any new recovery threads and wait for any currently -* running ones to exit. Do this before setting the vol_state. */ mutex_lock (&osb->recovery_lock); - osb->recovery_state =OCFS2_REC_DISABLED ; +/* +* If recovery thread is not running, we can directly transition to +* final state. +*/ +if (!ocfs2_recovery_thread_running (osb)) { + osb->recovery_state = state +1 ; + goto out_lock; + } + osb->recovery_state = state; +/* Wait for recovery thread to acknowledge state transition */ +wait_event_cmd (osb->recovery_event, + !ocfs2_recovery_thread_running (osb) || + osb->recovery_state >= state +1 , +mutex_unlock (&osb->recovery_lock), +mutex_lock (&osb->recovery_lock)); +out_lock:mutex_unlock (&osb->recovery_lock); -wait_event (osb->recovery_event, !ocfs2_recovery_thread_running (osb)); -/* At this point, we know that no more recovery threads can be -* launched, so wait for any recovery completion work to -* complete. */ +/* +* At this point we know that no more recovery work can be queued so +* wait for any recovery completion work to complete. +*/ if (osb->ocfs2_wq)flush_workqueue (osb->ocfs2_wq); +} + +void ocfs2_recovery_exit (struct ocfs2_super *osb) +{ +struct ocfs2_recovery_map *rm; + +/* disable any new recovery threads and wait for any currently +* running ones to exit. Do this before setting the vol_state. */ +ocfs2_recovery_disable (osb,OCFS2_REC_WANT_DISABLE );/* * Now that recovery is shut down, and the osb is about to be
@@ -1569,7 +1586,8 @@ static int __ocfs2_recovery_thread(void *arg)ocfs2_free_replay_slots (osb); osb->recovery_thread_task =NULL ; -mb ();/* sync with ocfs2_recovery_thread_running */ +if (osb->recovery_state ==OCFS2_REC_WANT_DISABLE ) + osb->recovery_state =OCFS2_REC_QUOTA_DISABLED ;wake_up (&osb->recovery_event);mutex_unlock (&osb->recovery_lock);
@@ -1585,13 +1603,13 @@ void ocfs2_recovery_thread(struct ocfs2_super *osb, int node_num)int was_set =-1 ;mutex_lock (&osb->recovery_lock); -if (osb->recovery_state <OCFS2_REC_DISABLED ) +if (osb->recovery_state <OCFS2_REC_WANT_DISABLE ) was_set =ocfs2_recovery_map_set (osb, node_num);trace_ocfs2_recovery_thread (node_num, osb->node_num, osb->recovery_state, osb->recovery_thread_task, was_set); -if (osb->recovery_state ==OCFS2_REC_DISABLED ) +if (osb->recovery_state >=OCFS2_REC_WANT_DISABLE ) goto out;if (osb->recovery_thread_task)
fs/ocfs2/ocfs2.h +4 -0
@@ -310,6 +310,10 @@ void ocfs2_initialize_journal_triggers(struct super_block *sb,enum ocfs2_recovery_state {OCFS2_REC_ENABLED =0 , +OCFS2_REC_WANT_DISABLE , +/* +* Must be OCFS2_REC_WANT_DISABLE + 1 for ocfs2_recovery_exit() to work +*/ OCFS2_REC_DISABLED , };
Signature · 1 line
2.43.0
Heming Zhao
reviewerRe: [PATCH 2/3] ocfs2: Implement handshaking with ocfs2 recovery thread
Quoted 80 lines · Jan Kara
We will need ocfs2 recovery thread to acknowledge transitions of
recovery_statewhen disabling particular types of recovery. This is similar to what currently happens when disabling recovery completely, just more general. Implement the handshake and use it for exit from recovery.Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/journal.c | 52 +++++++++++++++++++++++++++++++--------------- fs/ocfs2/ocfs2.h | 4 ++++ 2 files changed, 39 insertions(+), 17 deletions(-)fs/ocfs2/journal.c +31
-14@@ -190,31 +190,48 @@ int ocfs2_recovery_init(struct ocfs2_super *osb)return 0 ; } -/* we can't grab the goofy sem lock from inside wait_event, so we use -* memory barriers to make sure that we'll see the null task before -* being woken up */ static int ocfs2_recovery_thread_running (struct ocfs2_super *osb) { -mb ();return osb->recovery_thread_task !=NULL ; } -void ocfs2_recovery_exit (struct ocfs2_super *osb) +static void ocfs2_recovery_disable (struct ocfs2_super *osb, +enum ocfs2_recovery_state state) { -struct ocfs2_recovery_map *rm; - -/* disable any new recovery threads and wait for any currently -* running ones to exit. Do this before setting the vol_state. */ mutex_lock (&osb->recovery_lock); - osb->recovery_state =OCFS2_REC_DISABLED ; +/* +* If recovery thread is not running, we can directly transition to +* final state. +*/ +if (!ocfs2_recovery_thread_running (osb)) { + osb->recovery_state = state +1 ; + goto out_lock; + } + osb->recovery_state = state; +/* Wait for recovery thread to acknowledge state transition */ +wait_event_cmd (osb->recovery_event, + !ocfs2_recovery_thread_running (osb) || + osb->recovery_state >= state +1 , +mutex_unlock (&osb->recovery_lock), +mutex_lock (&osb->recovery_lock)); +out_lock:mutex_unlock (&osb->recovery_lock); -wait_event (osb->recovery_event, !ocfs2_recovery_thread_running (osb)); -/* At this point, we know that no more recovery threads can be -* launched, so wait for any recovery completion work to -* complete. */ +/* +* At this point we know that no more recovery work can be queued so +* wait for any recovery completion work to complete. +*/ if (osb->ocfs2_wq)flush_workqueue (osb->ocfs2_wq); +} + +void ocfs2_recovery_exit (struct ocfs2_super *osb) +{ +struct ocfs2_recovery_map *rm; + +/* disable any new recovery threads and wait for any currently +* running ones to exit. Do this before setting the vol_state. */ +ocfs2_recovery_disable (osb,OCFS2_REC_WANT_DISABLE );/* * Now that recovery is shut down, and the osb is about to be
fs/ocfs2/journal.c static int __ocfs2_recovery_thread(void *arg) · line 1590
Quoted 6 lines
fs/ocfs2/journal.c +2
-1@@ -1569,7 +1586,8 @@ static int __ocfs2_recovery_thread(void *arg)ocfs2_free_replay_slots (osb); osb->recovery_thread_task =NULL ; -mb ();/* sync with ocfs2_recovery_thread_running */ +if (osb->recovery_state ==OCFS2_REC_WANT_DISABLE ) + osb->recovery_state =OCFS2_REC_QUOTA_DISABLED ;
OCFS2_REC_QUOTA_DISABLED is defined in [patch 3/3], so the above line will make gcc unhappy.
All other code looks good to me.
- Heming
Quoted 34 lines
fs/ocfs2/journal.c +2
-2wake_up (&osb->recovery_event);mutex_unlock (&osb->recovery_lock);@@ -1585,13 +1603,13 @@ void ocfs2_recovery_thread(struct ocfs2_super *osb, int node_num)int was_set =-1 ;mutex_lock (&osb->recovery_lock); -if (osb->recovery_state <OCFS2_REC_DISABLED ) +if (osb->recovery_state <OCFS2_REC_WANT_DISABLE ) was_set =ocfs2_recovery_map_set (osb, node_num);trace_ocfs2_recovery_thread (node_num, osb->node_num, osb->recovery_state, osb->recovery_thread_task, was_set); -if (osb->recovery_state ==OCFS2_REC_DISABLED ) +if (osb->recovery_state >=OCFS2_REC_WANT_DISABLE ) goto out;if (osb->recovery_thread_task)fs/ocfs2/ocfs2.h +4
-0@@ -310,6 +310,10 @@ void ocfs2_initialize_journal_triggers(struct super_block *sb,enum ocfs2_recovery_state {OCFS2_REC_ENABLED =0 , +OCFS2_REC_WANT_DISABLE , +/* +* Must be OCFS2_REC_WANT_DISABLE + 1 for ocfs2_recovery_exit() to work +*/ OCFS2_REC_DISABLED , };
Heming Zhao
reviewerRe: [PATCH 2/3] ocfs2: Implement handshaking with ocfs2 recovery thread
Hi Jan,
Quoted 87 lines · Heming Zhao
Quoted 76 lines · Jan Kara
We will need ocfs2 recovery thread to acknowledge transitions of
recovery_statewhen disabling particular types of recovery. This is similar to what currently happens when disabling recovery completely, just more general. Implement the handshake and use it for exit from recovery.Signed-off-by Jan Kara <[email protected]>---
fs/ocfs2/journal.c| 52 +++++++++++++++++++++++++++++++--------------- fs/ocfs2/ocfs2.h | 4 ++++2 files changed, 39 insertions(+), 17 deletions(-)fs/ocfs2/journal.c +31
-14@@ -190,31 +190,48 @@ int ocfs2_recovery_init(struct ocfs2_super *osb) return 0; } -/* we can't grab the goofy sem lock from inside wait_event, so we use -* memory barriers to make sure that we'll see the null task before -* being woken up */ static int ocfs2_recovery_thread_running(struct ocfs2_super *osb) { - mb(); return osb->recovery_thread_task != NULL; } -void ocfs2_recovery_exit (struct ocfs2_super *osb) +static void ocfs2_recovery_disable (struct ocfs2_super *osb, + enum ocfs2_recovery_state state) { - struct ocfs2_recovery_map *rm; - - /* disable any new recovery threads and wait for any currently - * running ones to exit. Do this before setting the vol_state. */ mutex_lock(&osb->recovery_lock); - osb->recovery_state = OCFS2_REC_DISABLED; + /* + * If recovery thread is not running, we can directly transition to + * final state. + */ + if (!ocfs2_recovery_thread_running(osb)) { + osb->recovery_state = state + 1; + goto out_lock; + } + osb->recovery_state = state; + /* Wait for recovery thread to acknowledge state transition */ + wait_event_cmd(osb->recovery_event, + !ocfs2_recovery_thread_running(osb) || + osb->recovery_state >= state + 1, + mutex_unlock(&osb->recovery_lock), + mutex_lock(&osb->recovery_lock)); +out_lock: mutex_unlock(&osb->recovery_lock); - wait_event(osb->recovery_event, !ocfs2_recovery_thread_running(osb)); - /* At this point, we know that no more recovery threads can be - * launched, so wait for any recovery completion work to - * complete. */ + /* + * At this point we know that no more recovery work can be queued so + * wait for any recovery completion work to complete. + */ if (osb->ocfs2_wq) flush_workqueue(osb->ocfs2_wq); +} + +void ocfs2_recovery_exit (struct ocfs2_super *osb) +{ + struct ocfs2_recovery_map *rm; + + /* disable any new recovery threads and wait for any currently + * running ones to exit. Do this before setting the vol_state. */ + ocfs2_recovery_disable(osb, OCFS2_REC_WANT_DISABLE); /** Now that recovery is shut down, and the osb is about to be fs/ocfs2/journal.c static int __ocfs2_recovery_thread(void *arg) · line 1589
Quoted 5 lines
fs/ocfs2/journal.c +2
-1@@ -1569,7 +1586,8 @@ static int __ocfs2_recovery_thread(void *arg) ocfs2_free_replay_slots(osb); osb->recovery_thread_task = NULL; - mb(); /* sync with ocfs2_recovery_thread_running */ + if (osb->recovery_state == OCFS2_REC_WANT_DISABLE) + osb->recovery_state = OCFS2_REC_QUOTA_DISABLED;
OCFS2_REC_QUOTA_DISABLEDis defined in [patch 3/3], so the above line will make gcc unhappy.
All other code looks good to me.
Quoted 1 line
- Heming
If you are busy with other affairs, I can do the code adjustment for you. The adjustment only involves copying the OCFS2_REC_QUOTA_DISABLED definition from [patch 3/3].
- Heming
Quoted 29 lines
Quoted 28 lines
fs/ocfs2/journal.c +2
-2wake_up(&osb->recovery_event); mutex_unlock(&osb->recovery_lock);@@ -1585,13 +1603,13 @@ void ocfs2_recovery_thread(struct ocfs2_super *osb, int node_num) int was_set = -1; mutex_lock(&osb->recovery_lock); - if (osb->recovery_state < OCFS2_REC_DISABLED) + if (osb->recovery_state < OCFS2_REC_WANT_DISABLE) was_set = ocfs2_recovery_map_set(osb, node_num); trace_ocfs2_recovery_thread(node_num, osb->node_num, osb->recovery_state, osb->recovery_thread_task, was_set); - if (osb->recovery_state == OCFS2_REC_DISABLED) + if (osb->recovery_state >= OCFS2_REC_WANT_DISABLE) goto out; if (osb->recovery_thread_task)fs/ocfs2/ocfs2.h +4
-0@@ -310,6 +310,10 @@ void ocfs2_initialize_journal_triggers(struct super_block *sb, enum ocfs2_recovery_state { OCFS2_REC_ENABLED = 0, + OCFS2_REC_WANT_DISABLE, + /* + * Must be OCFS2_REC_WANT_DISABLE + 1 for ocfs2_recovery_exit() to work + */ OCFS2_REC_DISABLED, };
Re: [PATCH 2/3] ocfs2: Implement handshaking with ocfs2 recovery thread
In reply to Heming ZhaoQuoted 78 lines · Heming Zhao
Quoted 77 lines · Jan Kara
We will need ocfs2 recovery thread to acknowledge transitions of
recovery_statewhen disabling particular types of recovery. This is similar to what currently happens when disabling recovery completely, just more general. Implement the handshake and use it for exit from recovery.Signed-off-by Jan Kara <[email protected]>--- fs/ocfs2/journal.c | 52 +++++++++++++++++++++++++++++++--------------- fs/ocfs2/ocfs2.h | 4 ++++ 2 files changed, 39 insertions(+), 17 deletions(-)fs/ocfs2/journal.c +31
-14@@ -190,31 +190,48 @@ int ocfs2_recovery_init(struct ocfs2_super *osb)return 0 ; } -/* we can't grab the goofy sem lock from inside wait_event, so we use -* memory barriers to make sure that we'll see the null task before -* being woken up */ static int ocfs2_recovery_thread_running (struct ocfs2_super *osb) { -mb ();return osb->recovery_thread_task !=NULL ; } -void ocfs2_recovery_exit (struct ocfs2_super *osb) +static void ocfs2_recovery_disable (struct ocfs2_super *osb, +enum ocfs2_recovery_state state) { -struct ocfs2_recovery_map *rm; - -/* disable any new recovery threads and wait for any currently -* running ones to exit. Do this before setting the vol_state. */ mutex_lock (&osb->recovery_lock); - osb->recovery_state =OCFS2_REC_DISABLED ; +/* +* If recovery thread is not running, we can directly transition to +* final state. +*/ +if (!ocfs2_recovery_thread_running (osb)) { + osb->recovery_state = state +1 ; + goto out_lock; + } + osb->recovery_state = state; +/* Wait for recovery thread to acknowledge state transition */ +wait_event_cmd (osb->recovery_event, + !ocfs2_recovery_thread_running (osb) || + osb->recovery_state >= state +1 , +mutex_unlock (&osb->recovery_lock), +mutex_lock (&osb->recovery_lock)); +out_lock:mutex_unlock (&osb->recovery_lock); -wait_event (osb->recovery_event, !ocfs2_recovery_thread_running (osb)); -/* At this point, we know that no more recovery threads can be -* launched, so wait for any recovery completion work to -* complete. */ +/* +* At this point we know that no more recovery work can be queued so +* wait for any recovery completion work to complete. +*/ if (osb->ocfs2_wq)flush_workqueue (osb->ocfs2_wq); +} + +void ocfs2_recovery_exit (struct ocfs2_super *osb) +{ +struct ocfs2_recovery_map *rm; + +/* disable any new recovery threads and wait for any currently +* running ones to exit. Do this before setting the vol_state. */ +ocfs2_recovery_disable (osb,OCFS2_REC_WANT_DISABLE );/* * Now that recovery is shut down, and the osb is about to be @@ -1569,7 +1586,8 @@ static int __ocfs2_recovery_thread(void *arg)ocfs2_free_replay_slots (osb);
Quoted 8 lines
fs/ocfs2/journal.c static int __ocfs2_recovery_thread(void *arg) · line 1590
Quoted 4 lines
fs/ocfs2/journal.c +2
-1osb->recovery_thread_task =NULL ; -mb ();/* sync with ocfs2_recovery_thread_running */ +if (osb->recovery_state ==OCFS2_REC_WANT_DISABLE ) + osb->recovery_state =OCFS2_REC_QUOTA_DISABLED ;
OCFS2_REC_QUOTA_DISABLEDis defined in [patch 3/3], so the above line will make gcc unhappy.
All other code looks good to me.
Thanks for review and testing! I'll fix this up and send v2.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
PATCH 3/3 ocfs2: Stop quota recovery before disabling quotas
In reply to Jan KaraCurrently quota recovery is synchronized with unmount using sb->s_umount semaphore. That is however prone to deadlocks because flush_workqueue(osb->ocfs2_wq) called from umount code can wait for quota recovery to complete while ocfs2_finish_quota_recovery() waits for sb->s_umount semaphore.
Grabbing of sb->s_umount semaphore in ocfs2_finish_quota_recovery() is only needed to protect that function from disabling of quotas from ocfs2_dismount_volume(). Handle this problem by disabling quota recovery early during unmount in ocfs2_dismount_volume() instead so that we can drop acquisition of sb->s_umount from ocfs2_finish_quota_recovery().
--- fs/ocfs2/journal.c | 20 ++++++++++++++++++-- fs/ocfs2/journal.h | 1 + fs/ocfs2/ocfs2.h | 6 ++++++ fs/ocfs2/quota_local.c | 9 ++------- fs/ocfs2/super.c | 3 +++ 5 files changed, 30 insertions(+), 9 deletions(-)
fs/ocfs2/journal.c +18 -2
@@ -225,6 +225,11 @@ static void ocfs2_recovery_disable(struct ocfs2_super *osb,flush_workqueue (osb->ocfs2_wq); } +void ocfs2_recovery_disable_quota (struct ocfs2_super *osb) +{ +ocfs2_recovery_disable (osb,OCFS2_REC_QUOTA_WANT_DISABLE ); +} +void ocfs2_recovery_exit (struct ocfs2_super *osb) {struct ocfs2_recovery_map *rm;
@@ -1489,6 +1494,18 @@ static int __ocfs2_recovery_thread(void *arg) } } restart: +if (quota_enabled) { +mutex_lock (&osb->recovery_lock); +/* Confirm that recovery thread will no longer recover quotas */ +if (osb->recovery_state ==OCFS2_REC_QUOTA_WANT_DISABLE ) { + osb->recovery_state =OCFS2_REC_QUOTA_DISABLED ; +wake_up (&osb->recovery_event); + } +if (osb->recovery_state >=OCFS2_REC_QUOTA_DISABLED ) + quota_enabled =0 ; +mutex_unlock (&osb->recovery_lock); + } + status =ocfs2_super_lock (osb,1 );if (status <0 ) {mlog_errno (status);
@@ -1592,8 +1609,7 @@ static int __ocfs2_recovery_thread(void *arg)mutex_unlock (&osb->recovery_lock); -if (quota_enabled) -kfree (rm_quota); +kfree (rm_quota);return status; }
fs/ocfs2/journal.h +1 -0
@@ -148,6 +148,7 @@ void ocfs2_wait_for_recovery(struct ocfs2_super *osb);int ocfs2_recovery_init (struct ocfs2_super *osb);void ocfs2_recovery_exit (struct ocfs2_super *osb); +void ocfs2_recovery_disable_quota (struct ocfs2_super *osb);int ocfs2_compute_replay_slots (struct ocfs2_super *osb);void ocfs2_free_replay_slots (struct ocfs2_super *osb);
fs/ocfs2/ocfs2.h +6 -0
@@ -310,6 +310,12 @@ void ocfs2_initialize_journal_triggers(struct super_block *sb,enum ocfs2_recovery_state {OCFS2_REC_ENABLED =0 , +OCFS2_REC_QUOTA_WANT_DISABLE , +/* +* Must be OCFS2_REC_QUOTA_WANT_DISABLE + 1 for +* ocfs2_recovery_disable_quota() to work. +*/ +OCFS2_REC_QUOTA_DISABLED ,OCFS2_REC_WANT_DISABLE ,/* * Must be OCFS2_REC_WANT_DISABLE + 1 for ocfs2_recovery_exit() to work
fs/ocfs2/quota_local.c +2 -7
@@ -453,8 +453,7 @@ struct ocfs2_quota_recovery *ocfs2_begin_quota_recovery(/* Sync changes in local quota file into global quota file and * reinitialize local quota file. -* The function expects local quota file to be already locked and -* s_umount locked in shared mode. */ +* The function expects local quota file to be already locked. */ static int ocfs2_recover_local_quota_file (struct inode *lqinode,int type,struct ocfs2_quota_recovery *rec)
@@ -588,7 +587,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb, {unsigned int ino[OCFS2_MAXQUOTAS ] = {LOCAL_USER_QUOTA_SYSTEM_INODE ,LOCAL_GROUP_QUOTA_SYSTEM_INODE }; -struct super_block *sb = osb->sb;struct ocfs2_local_disk_dqinfo *ldinfo;struct buffer_head *bh;handle_t *handle;
@@ -600,7 +598,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,printk (KERN_NOTICE "ocfs2: Finishing quota recovery on device (%s) for " "slot %u\n" , osb->dev_str, slot_num); -down_read (&sb->s_umount);for (type =0 ; type <OCFS2_MAXQUOTAS ; type++) {if (list_empty (&(rec->r_list[type])))continue ;
@@ -677,7 +674,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,break ; } out: -up_read (&sb->s_umount);kfree (rec);return status; }
@@ -843,8 +839,7 @@ static int ocfs2_local_free_info(struct super_block *sb, int type)ocfs2_release_local_quota_bitmaps (&oinfo->dqi_chunk);/* -* s_umount held in exclusive mode protects us against racing with -* recovery thread... +* ocfs2_dismount_volume() has already aborted quota recovery... */ if (oinfo->dqi_rec) {ocfs2_free_quota_recovery (oinfo->dqi_rec);
fs/ocfs2/super.c +3 -0
@@ -1812,6 +1812,9 @@ static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)/* Orphan scan should be stopped as early as possible */ ocfs2_orphan_scan_stop (osb); +/* Stop quota recovery so that we can disable quotas */ +ocfs2_recovery_disable_quota (osb); +ocfs2_disable_quotas (osb);/* All dquots should be freed by now */
Signature · 1 line
2.43.0
Re: [PATCH 3/3] ocfs2: Stop quota recovery before disabling quotas
Quoted 12 lines · Jan Kara
Currently quota recovery is synchronized with unmount using
sb->s_umountsemaphore. That is however prone to deadlocks becauseflush_workqueue(osb->ocfs2_wq)called from umount code can wait for quota recovery to complete whileocfs2_finish_quota_recovery()waits forsb->s_umountsemaphore.Grabbing of
sb->s_umountsemaphore inocfs2_finish_quota_recovery()is only needed to protect that function from disabling of quotas fromocfs2_dismount_volume(). Handle this problem by disabling quota recovery early during unmount inocfs2_dismount_volume()instead so that we can drop acquisition ofsb->s_umountfromocfs2_finish_quota_recovery().
Quoted 3 lines
Reported-by Shichangkuo <[email protected]>Reported-by Murad Masimov <[email protected]>Signed-off-by Jan Kara <[email protected]>
Quoted 144 lines
--- fs/ocfs2/journal.c | 20 ++++++++++++++++++-- fs/ocfs2/journal.h | 1 + fs/ocfs2/ocfs2.h | 6 ++++++ fs/ocfs2/quota_local.c | 9 ++------- fs/ocfs2/super.c | 3 +++ 5 files changed, 30 insertions(+), 9 deletions(-)fs/ocfs2/journal.c +18
-2@@ -225,6 +225,11 @@ static void ocfs2_recovery_disable(struct ocfs2_super *osb,flush_workqueue (osb->ocfs2_wq); } +void ocfs2_recovery_disable_quota (struct ocfs2_super *osb) +{ +ocfs2_recovery_disable (osb,OCFS2_REC_QUOTA_WANT_DISABLE ); +} +void ocfs2_recovery_exit (struct ocfs2_super *osb) {struct ocfs2_recovery_map *rm;@@ -1489,6 +1494,18 @@ static int __ocfs2_recovery_thread(void *arg) } } restart: +if (quota_enabled) { +mutex_lock (&osb->recovery_lock); +/* Confirm that recovery thread will no longer recover quotas */ +if (osb->recovery_state ==OCFS2_REC_QUOTA_WANT_DISABLE ) { + osb->recovery_state =OCFS2_REC_QUOTA_DISABLED ; +wake_up (&osb->recovery_event); + } +if (osb->recovery_state >=OCFS2_REC_QUOTA_DISABLED ) + quota_enabled =0 ; +mutex_unlock (&osb->recovery_lock); + } + status =ocfs2_super_lock (osb,1 );if (status <0 ) {mlog_errno (status);@@ -1592,8 +1609,7 @@ static int __ocfs2_recovery_thread(void *arg)mutex_unlock (&osb->recovery_lock); -if (quota_enabled) -kfree (rm_quota); +kfree (rm_quota);return status; }fs/ocfs2/journal.h +1
-0@@ -148,6 +148,7 @@ void ocfs2_wait_for_recovery(struct ocfs2_super *osb);int ocfs2_recovery_init (struct ocfs2_super *osb);void ocfs2_recovery_exit (struct ocfs2_super *osb); +void ocfs2_recovery_disable_quota (struct ocfs2_super *osb);int ocfs2_compute_replay_slots (struct ocfs2_super *osb);void ocfs2_free_replay_slots (struct ocfs2_super *osb);fs/ocfs2/ocfs2.h +6
-0@@ -310,6 +310,12 @@ void ocfs2_initialize_journal_triggers(struct super_block *sb,enum ocfs2_recovery_state {OCFS2_REC_ENABLED =0 , +OCFS2_REC_QUOTA_WANT_DISABLE , +/* +* Must be OCFS2_REC_QUOTA_WANT_DISABLE + 1 for +* ocfs2_recovery_disable_quota() to work. +*/ +OCFS2_REC_QUOTA_DISABLED ,OCFS2_REC_WANT_DISABLE ,/* * Must be OCFS2_REC_WANT_DISABLE + 1 for ocfs2_recovery_exit() to work fs/ocfs2/quota_local.c +2
-7@@ -453,8 +453,7 @@ struct ocfs2_quota_recovery *ocfs2_begin_quota_recovery(/* Sync changes in local quota file into global quota file and * reinitialize local quota file. -* The function expects local quota file to be already locked and -* s_umount locked in shared mode. */ +* The function expects local quota file to be already locked. */ static int ocfs2_recover_local_quota_file (struct inode *lqinode,int type,struct ocfs2_quota_recovery *rec)@@ -588,7 +587,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb, {unsigned int ino[OCFS2_MAXQUOTAS ] = {LOCAL_USER_QUOTA_SYSTEM_INODE ,LOCAL_GROUP_QUOTA_SYSTEM_INODE }; -struct super_block *sb = osb->sb;struct ocfs2_local_disk_dqinfo *ldinfo;struct buffer_head *bh;handle_t *handle;@@ -600,7 +598,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,printk (KERN_NOTICE "ocfs2: Finishing quota recovery on device (%s) for " "slot %u\n" , osb->dev_str, slot_num); -down_read (&sb->s_umount);for (type =0 ; type <OCFS2_MAXQUOTAS ; type++) {if (list_empty (&(rec->r_list[type])))continue ;@@ -677,7 +674,6 @@ int ocfs2_finish_quota_recovery(struct ocfs2_super *osb,break ; } out: -up_read (&sb->s_umount);kfree (rec);return status; }@@ -843,8 +839,7 @@ static int ocfs2_local_free_info(struct super_block *sb, int type)ocfs2_release_local_quota_bitmaps (&oinfo->dqi_chunk);/* -* s_umount held in exclusive mode protects us against racing with -* recovery thread... +* ocfs2_dismount_volume() has already aborted quota recovery... */ if (oinfo->dqi_rec) {ocfs2_free_quota_recovery (oinfo->dqi_rec);fs/ocfs2/super.c +3
-0@@ -1812,6 +1812,9 @@ static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)/* Orphan scan should be stopped as early as possible */ ocfs2_orphan_scan_stop (osb); +/* Stop quota recovery so that we can disable quotas */ +ocfs2_recovery_disable_quota (osb); +ocfs2_disable_quotas (osb);/* All dquots should be freed by now */
Heming Zhao
reviewerI created a branch for these 3 patch files. Let's ask syzbot to test them.
syzbot page: https://syzkaller.appspot.com/bug?extid=f59a1ae7b7227c859b8f
#syz test: https://github.com/zhaohem/linux jans_ocfs2
Quoted 10 lines · Jan Kara
Hello,
this implements another approach to fixing quota recovery deadlocks. We avoid grabbing
sb->s_umountsemaphore fromocfs2_finish_quota_recovery()and instead stop quota recovery early inocfs2_dismount_volume(). Please review and test, the series has been only lightly tested in local mode as I don't have proper OCFS2 test setup.Honza
syzbot
syzbotRe: [syzbot] [ocfs2?] possible deadlock in ocfs2_finish_quota_recovery
syzbot has tested the proposed patch but the reproducer is still triggering an issue: unregister_netdevice: waiting for DEV to become free
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:unregister_netdevice: waiting for DEV to become free
unregister_netdevice: waiting for batadv0 to become free. Usage count = 3
Tested on:
commit: 11376431 ocfs2: Stop quota recovery before disabling q.. git tree: https://github.com/zhaohem/linux jans_ocfs2 console output: https://syzkaller.appspot.com/x/log.txt?x=14183a3f980000 kernel config: https://syzkaller.appspot.com/x/.config?x=5348e7fd1b89a770 dashboard link: https://syzkaller.appspot.com/bug?extid=f59a1ae7b7227c859b8f compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
Note: no patches were applied.
Heming Zhao
reviewerRe: [syzbot] [ocfs2?] possible deadlock in ocfs2_finish_quota_recovery
Quoted 6 lines · syzbot
Hello,
syzbot has tested the proposed patch but the reproducer is still triggering an issue:
unregister_netdevice: waiting for DEV to become free
unregister_netdevice: waiting for batadv0 to become free. Usage count = 3
This is another module's issue, and the deadlock issue disappeared.
- Heming
Quoted 12 lines
Tested on:
commit: 11376431 ocfs2: Stop quota recovery before disabling q.. git tree: https://github.com/zhaohem/linux jans_ocfs2 console output: https://syzkaller.appspot.com/x/log.txt?x=14183a3f980000 kernel config: https://syzkaller.appspot.com/x/.config?x=5348e7fd1b89a770 dashboard link: https://syzkaller.appspot.com/bug?extid=f59a1ae7b7227c859b8f compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40Note: no patches were applied.
Heming Zhao
reviewerQuoted 10 lines · Jan Kara
Hello,
this implements another approach to fixing quota recovery deadlocks. We avoid grabbing
sb->s_umountsemaphore fromocfs2_finish_quota_recovery()and instead stop quota recovery early inocfs2_dismount_volume(). Please review and test, the series has been only lightly tested in local mode as I don't have proper OCFS2 test setup.Honza
Sorry for the late comments.
I tested the three patches, and only patch [2/3] has a minor issue. Please check my comment in that patch for details.
- Heming