ocfs2: fix deadlock in dio write orphan cleanup path
Subsystems (2)
PATCH 1/1 ocfs2: fix deadlock in dio write orphan cleanup path
PREEMPT_RT's rtmutex PI chain walker detected a lock dependency cycle in a single thread:
ocfs2_file_write_iter()
inode_lock(file_inode) [Lock A]
ocfs2_dio_end_io_write()
ocfs2_inode_lock() [Lock B]
ocfs2_del_inode_from_orphan()
inode_lock(orphan_dir) [Lock D] <- cycle detected!
The problem is lock ordering. Lock B is held when Lock D is acquired. Recovery paths acquire these locks in a different order creating a potential cycle in the lock dependency graph.
Fix this by releasing Lock B (ocfs2_inode_unlock + brelse(di_bh)) BEFORE calling ocfs2_del_inode_from_orphan(). Pass NULL for di_bh to signal that ocfs2_del_inode_from_orphan() should acquire its own fresh cluster lock and di_bh internally.
This ensures consistent lock ordering:
Before: B held -> D acquired (inconsistent)
After: B released -> B' fresh -> D (consistent)
--- fs/ocfs2/aops.c | 21 +++++++++++++++------ fs/ocfs2/namei.c | 17 ++++++++++++++++- 2 files changed, 31 insertions(+), 7 deletions(-)
fs/ocfs2/aops.c +15 -6
@@ -2280,6 +2280,7 @@ static int ocfs2_dio_end_io_write(struct inode *inode,handle_t *handle =NULL ;loff_t end = offset + bytes;int ret =0 , credits =0 , batch =0 ; +bool orphaned = false;ocfs2_init_dealloc_ctxt (&dealloc);
@@ -2371,17 +2372,25 @@ static int ocfs2_dio_end_io_write(struct inode *inode,ocfs2_commit_trans (osb, handle); unlock:up_write (&oi->ip_alloc_sem); +/* +* Release the cluster lock and di_bh BEFORE calling +* ocfs2_del_inode_from_orphan(). That function will acquire +* inode_lock(orphan_dir_inode) which would cause an AB-BA +* deadlock with recovery paths that hold orphan_dir lock +* before acquiring the file inode lock. +*/ + orphaned = (!ret && dwc->dw_orphaned); +ocfs2_inode_unlock (inode,1 ); +brelse (di_bh); + di_bh =NULL ; -/* everything looks good, let's start the cleanup */ -if (!ret && dwc->dw_orphaned) { +/* everything looks good, let's start the orphan cleanup */ +if (orphaned) {BUG_ON (dwc->dw_writer_pid !=task_pid_nr (current)); - - ret =ocfs2_del_inode_from_orphan (osb, inode, di_bh,0 ,0 ); + ret =ocfs2_del_inode_from_orphan (osb, inode,NULL ,0 ,0 );if (ret <0 )mlog_errno (ret); } -ocfs2_inode_unlock (inode,1 ); -brelse (di_bh); out:if (data_ac)ocfs2_free_alloc_context (data_ac);
fs/ocfs2/namei.c +16 -1
@@ -2712,10 +2712,21 @@ int ocfs2_del_inode_from_orphan(struct ocfs2_super *osb, {struct inode *orphan_dir_inode =NULL ;struct buffer_head *orphan_dir_bh =NULL ; -struct ocfs2_dinode *di = (struct ocfs2_dinode *)di_bh->b_data; +struct ocfs2_dinode *di;handle_t *handle =NULL ;int status =0 ; +struct buffer_head *local_di_bh =NULL ; +if (!di_bh) { + status =ocfs2_inode_lock (inode, &local_di_bh,1 ); +if (status <0 ) { +mlog_errno (status); +return status; + } + di_bh = local_di_bh; + } + + di = (struct ocfs2_dinode *)di_bh->b_data; orphan_dir_inode =ocfs2_get_system_file_inode (osb,ORPHAN_DIR_SYSTEM_INODE ,le16_to_cpu (di->i_dio_orphaned_slot));
@@ -2779,6 +2790,10 @@ int ocfs2_del_inode_from_orphan(struct ocfs2_super *osb,iput (orphan_dir_inode); bail: +if (local_di_bh) { +ocfs2_inode_unlock (inode,1 ); +brelse (local_di_bh); + }return status; }
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
Pre-existing issues:
- [High] The patch completely fails to resolve the AB-BA deadlock between the write path and the orphan recovery path, as the exact same lock dependency is immediately recreated.
- [High] The identical lock dependency cycle (A -> B -> D) exists unmitigated in the orphan addition path.
- [High] Directory i_rwsem operations use the wrong subclass, violating VFS lock ordering rules and hiding parent-child deadlocks.
--
commit a3ea3ef9d81bdb971acfa026b14e772f33af54fa
Author: Deepanshu Kartikey <[email protected]>
ocfs2: fix deadlock in dio write orphan cleanup path
This commit attempts to fix an AB-BA deadlock in the direct IO orphan cleanup path by releasing the file's cluster lock before calling ocfs2_del_inode_from_orphan(). It passes NULL for di_bh to signal the function to acquire its own lock.
Quoted 4 lines
fs/ocfs2/aops.c +0
-0
[ ... ]
Quoted 30 lines
fs/ocfs2/aops.c +14
-6@@ -2371,17 +2372,25 @@ static int ocfs2_dio_end_io_write(struct inode *inode,ocfs2_commit_trans (osb, handle); unlock:up_write (&oi->ip_alloc_sem); +/* +* Release the cluster lock and di_bh BEFORE calling +* ocfs2_del_inode_from_orphan(). That function will acquire +* inode_lock(orphan_dir_inode) which would cause an AB-BA +* deadlock with recovery paths that hold orphan_dir lock +* before acquiring the file inode lock. +*/ + orphaned = (!ret && dwc->dw_orphaned); +ocfs2_inode_unlock (inode,1 ); +brelse (di_bh); + di_bh =NULL ; -/* everything looks good, let's start the cleanup */ -if (!ret && dwc->dw_orphaned) { +/* everything looks good, let's start the orphan cleanup */ +if (orphaned) {BUG_ON (dwc->dw_writer_pid !=task_pid_nr (current)); - - ret =ocfs2_del_inode_from_orphan (osb, inode, di_bh,0 ,0 ); + ret =ocfs2_del_inode_from_orphan (osb, inode,NULL ,0 ,0 );if (ret <0 )mlog_errno (ret); } -ocfs2_inode_unlock (inode,1 ); -brelse (di_bh); out:
[ ... ]
Quoted 6 lines
fs/ocfs2/namei.c +0
-0@@ -2712,10 +2712,21 @@ int ocfs2_del_inode_from_orphan(struct ocfs2_super *osb, {
fs/ocfs2/namei.c int ocfs2_del_inode_from_orphan(struct ocfs2_super *osb, · line 2721
Quoted 10 lines
fs/ocfs2/namei.c +4
-1struct inode *orphan_dir_inode =NULL ;struct buffer_head *orphan_dir_bh =NULL ; -struct ocfs2_dinode *di = (struct ocfs2_dinode *)di_bh->b_data; +struct ocfs2_dinode *di;handle_t *handle =NULL ;int status =0 ; +struct buffer_head *local_di_bh =NULL ; +if (!di_bh) { + status =ocfs2_inode_lock (inode, &local_di_bh,1 );
[Severity: High]
This is a pre-existing issue, but is the AB-BA deadlock between the write path and the orphan recovery path truly resolved?
When di_bh is NULL, this immediately reacquires the exact same ocfs2_inode_lock on the file inode (Lock B) before the orphan directory's VFS lock (Lock D) is acquired later in this function:
fs/ocfs2/namei.c:ocfs2_del_inode_from_orphan() {
...
inode_lock(orphan_dir_inode);
...
}Since the lock acquisition order remains exactly the same (B -> D) as it was before the patch, and the recovery path acquires these locks in the reverse order (D -> B), doesn't the deadlock cycle remain unmitigated?
Also, as a related pre-existing issue, couldn't the identical lock dependency cycle (A -> B -> D) still occur unmitigated in the orphan addition path?
During direct IO writes, ocfs2_dio_get_block() calls ocfs2_add_inode_to_orphan() while holding the file's VFS lock (Lock A). Inside ocfs2_add_inode_to_orphan(), the file's cluster lock (Lock B) is acquired:
fs/ocfs2/namei.c:ocfs2_add_inode_to_orphan() {
status = ocfs2_inode_lock(inode, &di_bh, 1);
...
status = ocfs2_prepare_orphan_dir(osb, &orphan_dir_inode,
OCFS2_I(inode)->ip_blkno,
orphan_name,
&orphan_insert,
true);
}This calls ocfs2_prepare_orphan_dir(), which acquires the orphan directory's VFS lock (Lock D) via inode_lock(). This seems to establish the exact same A -> B -> D deadlock cycle against the recovery path.
Finally, this is a pre-existing issue, but might the use of inode_lock() on the orphan directory violate VFS lock ordering rules?
When modifying a directory, the VFS requires acquiring the directory's i_rwsem with subclass I_MUTEX_PARENT. In ocfs2_del_inode_from_orphan() and ocfs2_lookup_lock_orphan_dir(), the default inode_lock() is used instead:
fs/ocfs2/namei.c:ocfs2_del_inode_from_orphan() {
...
inode_lock(orphan_dir_inode);
...
}Because the caller already holds the file's inode_lock (also subclass 0), acquiring the orphan directory's lock using subclass 0 can create a lockdep AA self-deadlock or cycle warning. Furthermore, taking a child's lock before its parent directory's lock intrinsically violates the VFS hierarchy (I_MUTEX_PARENT -> I_MUTEX_NORMAL). Should this be using inode_lock_nested(..., I_MUTEX_PARENT) instead?
Quoted 11 lines
fs/ocfs2/namei.c +8
-0+if (status <0 ) { +mlog_errno (status); +return status; + } + di_bh = local_di_bh; + } + + di = (struct ocfs2_dinode *)di_bh->b_data; orphan_dir_inode =ocfs2_get_system_file_inode (osb,ORPHAN_DIR_SYSTEM_INODE ,le16_to_cpu (di->i_dio_orphaned_slot));
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 9 lines · Deepanshu Kartikey
PREEMPT_RT's rtmutex PI chain walker detected a lock dependency cycle in a single thread:
ocfs2_file_write_iter()
inode_lock(file_inode)[Lock A]
ocfs2_dio_end_io_write()
ocfs2_inode_lock()[Lock B]
ocfs2_del_inode_from_orphan()
inode_lock(orphan_dir)[Lock D] <- cycle detected!
This seems like a false positive. You can't call write_iter() on a directory, and orphan_dir is always a directory.
I would suggest that the easiest way to make this warning go away is to replace inode_lock(orphan_dir) with inode_lock_nested(orphan_dir, I_MUTEX_NONDIR2). It's a bit quirky because, well, orphan2 is a directory. We could add a seventh lock class to inode_i_mutex_lock_class, but that feels a bit excessive.
Quoted 10 lines · Matthew Wilcox
This seems like a false positive. You can't call
write_iter()on a directory, andorphan_diris always a directory.I would suggest that the easiest way to make this warning go away is to replace
inode_lock(orphan_dir)withinode_lock_nested(orphan_dir, I_MUTEX_NONDIR2). It's a bit quirky because, well, orphan2 is a directory. We could add a seventh lock class toinode_i_mutex_lock_class, but that feels a bit excessive.
Thanks for the review. I have sent the patch v2 with
inode_lock_nested(orphan_dir_inode, I_MUTEX_NONDIR2)