xfs: introduce in-memory inode unlink log items
v4 cover xfs: introduce in-memory inode unlink log items
Hi Folks,
V4 of the iunlink log item patchset. Note that this has been rebased on the xfs-perag-conv-5.20 topic branch to avoid the horrible merge conflicts the previous version of this patchset had with the perag conversions. hence you'll need to pull this branch:
git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs tags/xfs-perag-conv-5.20
before applying this patchset to test it.
-Dave.
-----
Inode cluster buffer pinning by dirty inodes allows us to improve dirty inode tracking efficiency in the log by logging the inode cluster buffer as an ordered transaction. However, this brings with it some new issues, namely the order in which we lock inode cluster buffers. While the dirty buffer pinning doesn't currently result in nested cluster buffer locking in xfs_trans_log_inode() and so there are no current problems, future changes will need to nest cluster buffer locking in this context.
That is, transactions that dirty and commit multiple inodes in a transaction will now need to locking multiple inode cluster buffers in each transaction (e.g. create, rename, etc). This introduces new lock ordering constraints in these operations. It also introduces lock ordering constraints between the AGI and inode cluster buffers as a result of allocation/freeing being serialised by the AGI buffer lock. And then there is unlinked inode list logging, which currently has no fixed order of inode cluster buffer locking.
It's all a bit Wild West right now.
Locking pure inode modifications in order is relatively easy. We don't actually need to attach and log the buffer to the transaction until the last moment. We have all the inodes locked, so nothing other than unlinked inode list modification can race with the transaction modifying inodes. Hence we can safely move the attachment of the inodes to the cluster buffer from when we first dirty them in xfs_trans_log_inode to just before we commit the transaction.
At this point, all the inodes that have been dirtied in the transaction have already been locked, modified, logged and attached to the transaction. Hence if we add a hook into xfs_trans_commit() to run a "precommit" operation on these log items, we can use this operation to attach the inodes to the cluster buffer at commit time instead of in xfs_trans_log_inode().
This, by itself, doesn't solve the lock ordering problem. What it does do, however, is give us a place where we can -order- all the dirty items in the transaction list. Hence before we call the precommit operation on each log item, we sort them. This allows us to sort all the inode items so that the pre-commit functions that locks and logs the cluster buffers are run in a deterministic order. This solves the lock order problem for pure inode modifications.
The unlinked inode list buffer locking is more complex. The unlinked list is unordered - we add to the tail, remove from where-ever the inode is in the list. Hence we might need to lock two inode buffers here (previous inode in list and the one being removed). While we can order the locking of these buffers correctly within the confines of the unlinked list, there may be other inodes that need buffer locking in the same transaction. e.g. O_TMPFILE being linked into a directory also modifies the directory inode.
Hence we need a mechanism for defering unlinked inode list updates to the pre-commit operation where it can be sorted into the correct order. We can do this by first observing that we serialise unlinked list modifications by holding the AGI buffer lock. IOWs, the AGI is going to be locked until the transaction commits any time we modify the unlinked list. Hence it doesn't matter when in the unlink transactions that we actually load, lock and modify the inode cluster buffer.
IOWs, what we need is an unlinked inode log item to defer the inode cluster buffer update to transaction commit time where it can be ordered with all the other inode cluster operations. Essentially all we need to do is record the inodes that need to have their unlinked list pointer updated in a new log item that we attached to the transaction.
This log item exists purely for the purpose of delaying the update of the unlinked list pointer until the inode cluster buffer can be locked in the correct order around the other inode cluster buffers. It plays no part in the actual commit, and there's no change to anything that is written to the log. i.e. the inode cluster buffers still have to be fully logged here (not just ordered) as log recovery depedends on this to replay mods to the unlinked inode list.
To make this unlinked inode list processing simpler and easier to implement as a log item, we need to change the way we track the unlinked list in memory. Starting from the observation that an inode on the unlinked list is pinned in memory by the VFS, we can use the xfs_inode itself to track the unlinked list. To do this efficiently, we want the unlinked list to be a double linked list. The problem here is that we need a list per AGI unlinked list, and there are 64 of these per AGI. The approach taken in this patchset is to shadow the AGI unlinked list heads in the perag, and link inodes by agino, hence requiring only 8 extra bytes per inode to track this state.
We can then use the agino pointers for lockless inode cache lookups to retreive the inode. The aginos in the inode are modified only under the AGI lock, just like the cluster buffer pointers, so we don't need any extra locking here. The i_next_unlinked field tracks the on-disk value of the unlinked list, and the i_prev_unlinked is a purely in-memory pointer that enables us to efficiently remove inodes from the middle of the list.
This results in moving a lot of the unlink modification work into the precommit operations on the unlink log item. tracking all the unlinked inodes in the inodes themselves also gets rid of the unlinked list reference hash table that is used to track this back pointer relationship. This greatly simplifies the the unlinked list modification code, and removes memory allocations in this hot path to track back pointers. This, overall, slightly reduces the CPU overhead of the unlink path.
The result of this log item means that we move all the actual manipulation of objects to be logged out of the iunlink path and into the iunlink item. This allows for future optimisation of this mechanism without needing changes to high level unlink path, as well as making the unlink lock ordering predictable and synchronised with other operations that may require inode cluster locking.
Changelog:
Version 4:
- rebase on 5.19-rc2 + xfs-perag-conv-5.20 as the changes to AGI interfaces cause complex merge conflicts that are non-trivial to resolve correctly.
- fix typos in commit messages
- fix random whitespace damage
- clean up log recovery changes to avoid passing both mp and pag around
- clean up error handling for xfs_iunlink_lookup
- remove unused tracepoints
- remove stale references to potential dquot modifications
Version 3:
- https://lore.kernel.org/linux-xfs/[email protected]/
- rebase on 5.19-rc2
- Note: has significant, non-trivial merge conflicts with xfs-perag-conv-5.20. Should probably be rebased on that branch to avoid these. Need to know what order these branches are going to get merged first.
Version 2:
- unpublished
- rebase on 5.15-rc2 + xlog-intent-whiteouts
- reverts changes made in V1 that moved to a list_head based double linked list. This requires on-disk format changes to allow the AGI to use just a single bucket and so is not generically applicable to all XFS filesystem formats. This version goes back to using prev/next agino values and radix tree lookups as the original RFC used.
Version 1:
- https://lore.kernel.org/linux-xfs/[email protected]/
- split up into many smaller patches
- includes Xiang's single unlinked list bucket modification
- uses a list_head for the in memory double unlinked inode list rather than aginos and lockless inode lookups
- much simpler as it doesn't need to look up inodes from agino values
- iunlink log item changed to take an xfs_inode pointer rather than an imap and agino values
- a handful of small cleanups that breaking up into small patches allowed.
PATCH v4 1/9 xfs: factor the xfs_iunlink functions
From: Dave Chinner <[email protected]>
Prep work that separates the locking that protects the unlinked list from the actual operations being performed. This also helps document the fact they are performing list insert and remove operations. No functional code change.
--- fs/xfs/xfs_inode.c | 108 +++++++++++++++++++++++++++------------------ 1 file changed, 66 insertions(+), 42 deletions(-)
fs/xfs/xfs_inode.c +66 -42
@@ -2115,39 +2115,20 @@ xfs_iunlink_update_inode(return error; } -/* -* This is called when the inode's link count has gone to 0 or we are creating -* a tmpfile via O_TMPFILE. The inode @ip must have nlink == 0. -* -* We place the on-disk inode on a list in the AGI. It will be pulled from this -* list when the inode is freed. -*/ -STATIC int -xfs_iunlink ( +static int +xfs_iunlink_insert_inode (struct xfs_trans *tp, +struct xfs_perag *pag, +struct xfs_buf *agibp,struct xfs_inode *ip) {struct xfs_mount *mp = tp->t_mountp; -struct xfs_perag *pag; -struct xfs_agi *agi; -struct xfs_buf *agibp; +struct xfs_agi *agi = agibp->b_addr;xfs_agino_t next_agino;xfs_agino_t agino =XFS_INO_TO_AGINO (mp, ip->i_ino);short bucket_index = agino %XFS_AGI_UNLINKED_BUCKETS ;int error; -ASSERT (VFS_I (ip)->i_nlink ==0 ); -ASSERT (VFS_I (ip)->i_mode !=0 ); -trace_xfs_iunlink (ip); - - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); - -/* Get the agi buffer first. It ensures lock ordering on the list. */ - error =xfs_read_agi (pag, tp, &agibp); -if (error) - goto out; - agi = agibp->b_addr; -/* * Get the index into the agi hash table for the list this inode will * go on. Make sure the pointer isn't garbage and that this inode
@@ -2157,8 +2138,7 @@ xfs_iunlink(if (next_agino == agino || !xfs_verify_agino_or_null (pag, next_agino)) {xfs_buf_mark_corrupt (agibp); - error = -EFSCORRUPTED ; - goto out; +return -EFSCORRUPTED ; }if (next_agino !=NULLAGINO ) {
@@ -2171,7 +2151,7 @@ xfs_iunlink( error =xfs_iunlink_update_inode (tp, ip, pag, next_agino, &old_agino);if (error) - goto out; +return error;ASSERT (old_agino ==NULLAGINO );/*
@@ -2180,11 +2160,42 @@ xfs_iunlink(*/ error =xfs_iunlink_add_backref (pag, agino, next_agino);if (error) - goto out; +return error; }/* Point the head of the list to point to this inode. */ - error =xfs_iunlink_update_bucket (tp, pag, agibp, bucket_index, agino); +return xfs_iunlink_update_bucket (tp, pag, agibp, bucket_index, agino); +} + +/* +* This is called when the inode's link count has gone to 0 or we are creating +* a tmpfile via O_TMPFILE. The inode @ip must have nlink == 0. +* +* We place the on-disk inode on a list in the AGI. It will be pulled from this +* list when the inode is freed. +*/ +STATIC int +xfs_iunlink ( +struct xfs_trans *tp, +struct xfs_inode *ip) +{ +struct xfs_mount *mp = tp->t_mountp; +struct xfs_perag *pag; +struct xfs_buf *agibp; +int error; + +ASSERT (VFS_I (ip)->i_nlink ==0 ); +ASSERT (VFS_I (ip)->i_mode !=0 ); +trace_xfs_iunlink (ip); + + pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); + +/* Get the agi buffer first. It ensures lock ordering on the list. */ + error =xfs_read_agi (pag, tp, &agibp); +if (error) + goto out; + + error =xfs_iunlink_insert_inode (tp, pag, agibp, ip); out:xfs_perag_put (pag);return error;
@@ -2305,18 +2316,15 @@ xfs_iunlink_map_prev(return 0 ; } -/* -* Pull the on-disk inode from the AGI unlinked list. -*/ -STATIC int -xfs_iunlink_remove ( +static int +xfs_iunlink_remove_inode (struct xfs_trans *tp,struct xfs_perag *pag, +struct xfs_buf *agibp,struct xfs_inode *ip) {struct xfs_mount *mp = tp->t_mountp; -struct xfs_agi *agi; -struct xfs_buf *agibp; +struct xfs_agi *agi = agibp->b_addr;struct xfs_buf *last_ibp;struct xfs_dinode *last_dip =NULL ;xfs_agino_t agino =XFS_INO_TO_AGINO (mp, ip->i_ino);
@@ -2327,12 +2335,6 @@ xfs_iunlink_remove(trace_xfs_iunlink_remove (ip); -/* Get the agi buffer first. It ensures lock ordering on the list. */ - error =xfs_read_agi (pag, tp, &agibp); -if (error) -return error; - agi = agibp->b_addr; -/* * Get the index into the agi hash table for the list this inode will * go on. Make sure the head pointer isn't garbage.
@@ -2397,6 +2399,28 @@ xfs_iunlink_remove( next_agino); } +/* +* Pull the on-disk inode from the AGI unlinked list. +*/ +STATIC int +xfs_iunlink_remove ( +struct xfs_trans *tp, +struct xfs_perag *pag, +struct xfs_inode *ip) +{ +struct xfs_buf *agibp; +int error; + +trace_xfs_iunlink_remove (ip); + +/* Get the agi buffer first. It ensures lock ordering on the list. */ + error =xfs_read_agi (pag, tp, &agibp); +if (error) +return error; + +return xfs_iunlink_remove_inode (tp, pag, agibp, ip); +} +/* * Look up the inode number specified and if it is not already marked XFS_ISTALE * mark it stale. We should only find clean inodes in this lookup that aren't
Signature · 1 line
2.36.1
PATCH v4 2/9 xfs: track the iunlink list pointer in the xfs_inode
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Having direct access to the i_next_unlinked pointer in unlinked inodes greatly simplifies the processing of inodes on the unlinked list. We no longer need to look up the inode buffer just to find next inode in the list if the xfs_inode is in memory. These improvements will be realised over upcoming patches as other dependencies on the inode buffer for unlinked list processing are removed.
--- fs/xfs/libxfs/xfs_inode_buf.c | 3 ++- fs/xfs/xfs_inode.c | 5 ++++- fs/xfs/xfs_inode.h | 3 +++ fs/xfs/xfs_log_recover.c | 17 +---------------- 4 files changed, 10 insertions(+), 18 deletions(-)
fs/xfs/libxfs/xfs_inode_buf.c +2 -1
@@ -229,7 +229,8 @@ xfs_inode_from_disk( ip->i_nblocks =be64_to_cpu (from->di_nblocks); ip->i_extsize =be32_to_cpu (from->di_extsize); ip->i_forkoff = from->di_forkoff; - ip->i_diflags =be16_to_cpu (from->di_flags); + ip->i_diflags =be16_to_cpu (from->di_flags); + ip->i_next_unlinked =be32_to_cpu (from->di_next_unlinked);if (from->di_dmevmask || from->di_dmstate)xfs_iflags_set (ip,XFS_IPRESERVE_DM_FIELDS );
fs/xfs/xfs_inode.c +4 -1
@@ -2084,7 +2084,8 @@ xfs_iunlink_update_inode(/* Make sure the old pointer isn't garbage. */ old_value =be32_to_cpu (dip->di_next_unlinked); -if (!xfs_verify_agino_or_null (pag, old_value)) { +if (old_value != ip->i_next_unlinked || + !xfs_verify_agino_or_null (pag, old_value)) {xfs_inode_verifier_error (ip, -EFSCORRUPTED , __func__, dip,sizeof (*dip), __this_address); error = -EFSCORRUPTED ;
@@ -2153,6 +2154,7 @@ xfs_iunlink_insert_inode(if (error)return error;ASSERT (old_agino ==NULLAGINO ); + ip->i_next_unlinked = next_agino;/* * agino has been unlinked, add a backref from the next inode
@@ -2354,6 +2356,7 @@ xfs_iunlink_remove_inode( error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO , &next_agino);if (error)return error; + ip->i_next_unlinked =NULLAGINO ;/* * If there was a backref pointing from the next inode back to this
fs/xfs/xfs_inode.h +3 -0
@@ -68,6 +68,9 @@ typedef struct xfs_inode {uint64_t i_diflags2;/* XFS_DIFLAG2_... */ struct timespec64 i_crtime;/* time created */ +/* unlinked list pointers */ +xfs_agino_t i_next_unlinked; +/* VFS inode */ struct inode i_vnode;/* embedded VFS inode */
fs/xfs/xfs_log_recover.c +1 -16
@@ -2673,8 +2673,6 @@ xlog_recover_process_one_iunlink( xfs_agino_t agino, int bucket) { -struct xfs_buf *ibp; -struct xfs_dinode *dip;struct xfs_inode *ip;xfs_ino_t ino;int error;
@@ -2684,27 +2682,14 @@ xlog_recover_process_one_iunlink(if (error) goto fail; -/* -* Get the on disk inode to find the next inode in the bucket. -*/ - error =xfs_imap_to_bp (pag->pag_mount,NULL , &ip->i_imap, &ibp); -if (error) - goto fail_iput; - dip =xfs_buf_offset (ibp, ip->i_imap.im_boffset); -xfs_iflags_clear (ip,XFS_IRECOVERY );ASSERT (VFS_I (ip)->i_nlink ==0 );ASSERT (VFS_I (ip)->i_mode !=0 ); -/* setup for the next pass */ - agino =be32_to_cpu (dip->di_next_unlinked); -xfs_buf_relse (ibp); - + agino = ip->i_next_unlinked;xfs_irele (ip);return agino; - fail_iput: -xfs_irele (ip); fail:/* * We can't read in the inode this bucket points to, or this inode
Signature · 1 line
2.36.1
Re: [PATCH 2/9] xfs: track the iunlink list pointer in the xfs_inode
Quoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Having direct access to the
i_next_unlinkedpointer in unlinked inodes greatly simplifies the processing of inodes on the unlinked list. We no longer need to look up the inode buffer just to find next inode in the list if thexfs_inodeis in memory. These improvements will be realised over upcoming patches as other dependencies on the inode buffer for unlinked list processing are removed.
Quoted 2 lines
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>
Looks good:
PATCH v4 3/9 xfs: refactor xlog_recover_process_iunlinks()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
For upcoming changes to the way inode unlinked list processing is done, the structure of recovery needs to change slightly. We also really need to untangle the messy error handling in list recovery so that actions like emptying the bucket on inode lookup failure are associated with the bucket list walk failing, not failing to look up the inode.
Refactor the recovery code now to keep the re-organisation seperate to the algorithm changes.
--- fs/xfs/xfs_log_recover.c | 135 ++++++++++++++++++++------------------- 1 file changed, 70 insertions(+), 65 deletions(-)
fs/xfs/xfs_log_recover.c +70 -65
@@ -2667,40 +2667,35 @@ xlog_recover_clear_agi_bucket(return ; } -STATIC xfs_agino_t -xlog_recover_process_one_iunlink ( -struct xfs_perag *pag, -xfs_agino_t agino, -int bucket) +static int +xlog_recover_iunlink_bucket ( +struct xfs_perag *pag, +struct xfs_agi *agi, +int bucket) { -struct xfs_inode *ip; -xfs_ino_t ino; -int error; +struct xfs_mount *mp = pag->pag_mount; +struct xfs_inode *ip; +xfs_agino_t agino; - ino =XFS_AGINO_TO_INO (pag->pag_mount, pag->pag_agno, agino); - error =xfs_iget (pag->pag_mount,NULL , ino,0 ,0 , &ip); -if (error) - goto fail; + agino =be32_to_cpu (agi->agi_unlinked[bucket]); +while (agino !=NULLAGINO ) { +int error; -xfs_iflags_clear (ip,XFS_IRECOVERY ); -ASSERT (VFS_I (ip)->i_nlink ==0 ); -ASSERT (VFS_I (ip)->i_mode !=0 ); + error =xfs_iget (mp,NULL , +XFS_AGINO_TO_INO (mp, pag->pag_agno, agino), +0 ,0 , &ip); +if (error) +return error;; - agino = ip->i_next_unlinked; -xfs_irele (ip); -return agino; +ASSERT (VFS_I (ip)->i_nlink ==0 ); +ASSERT (VFS_I (ip)->i_mode !=0 ); +xfs_iflags_clear (ip,XFS_IRECOVERY ); + agino = ip->i_next_unlinked; - fail: -/* -* We can't read in the inode this bucket points to, or this inode -* is messed up. Just ditch this bucket of inodes. We will lose -* some inodes and space, but at least we won't hang. -* -* Call xlog_recover_clear_agi_bucket() to perform a transaction to -* clear the inode pointer in the bucket. -*/ -xlog_recover_clear_agi_bucket (pag, bucket); -return NULLAGINO ; +xfs_irele (ip); +cond_resched (); + } +return 0 ; }/*
@@ -2726,59 +2721,69 @@ xlog_recover_process_one_iunlink(* scheduled on this CPU to ensure other scheduled work can run without undue * latency. */ -STATIC void -xlog_recover_process_iunlinks ( -struct xlog *log) +static void +xlog_recover_iunlink_ag ( +struct xfs_perag *pag) { -struct xfs_mount *mp = log->l_mp; -struct xfs_perag *pag; -xfs_agnumber_t agno;struct xfs_agi *agi;struct xfs_buf *agibp; -xfs_agino_t agino;int bucket;int error; -for_each_perag (mp, agno, pag) { - error =xfs_read_agi (pag,NULL , &agibp); + error =xfs_read_agi (pag,NULL , &agibp); +if (error) { +/* +* AGI is b0rked. Don't process it. +* +* We should probably mark the filesystem as corrupt after we've +* recovered all the ag's we can.... +*/ +return ; + } + +/* +* Unlock the buffer so that it can be acquired in the normal course of +* the transaction to truncate and free each inode. Because we are not +* racing with anyone else here for the AGI buffer, we don't even need +* to hold it locked to read the initial unlinked bucket entries out of +* the buffer. We keep buffer reference though, so that it stays pinned +* in memory while we need the buffer. +*/ + agi = agibp->b_addr; +xfs_buf_unlock (agibp); + +for (bucket =0 ; bucket <XFS_AGI_UNLINKED_BUCKETS ; bucket++) { + error =xlog_recover_iunlink_bucket (pag, agi, bucket);if (error) {/* -* AGI is b0rked. Don't process it. -* -* We should probably mark the filesystem as corrupt -* after we've recovered all the ag's we can.... +* Bucket is unrecoverable, so only a repair scan can +* free the remaining unlinked inodes. Just empty the +* bucket and remaining inodes on it unreferenced and +* unfreeable. */ -continue ; - } -/* -* Unlock the buffer so that it can be acquired in the normal -* course of the transaction to truncate and free each inode. -* Because we are not racing with anyone else here for the AGI -* buffer, we don't even need to hold it locked to read the -* initial unlinked bucket entries out of the buffer. We keep -* buffer reference though, so that it stays pinned in memory -* while we need the buffer. -*/ - agi = agibp->b_addr; -xfs_buf_unlock (agibp); - -for (bucket =0 ; bucket <XFS_AGI_UNLINKED_BUCKETS ; bucket++) { - agino =be32_to_cpu (agi->agi_unlinked[bucket]); -while (agino !=NULLAGINO ) { - agino =xlog_recover_process_one_iunlink (pag, - agino, bucket); -cond_resched (); - } +xlog_recover_clear_agi_bucket (pag, bucket); } -xfs_buf_rele (agibp); } +xfs_buf_rele (agibp); +} + +static void +xlog_recover_process_iunlinks ( +struct xlog *log) +{ +struct xfs_perag *pag; +xfs_agnumber_t agno; + +for_each_perag (log->l_mp, agno, pag) +xlog_recover_iunlink_ag (pag); +/* * Flush the pending unlinked inodes to ensure that the inactivations * are fully completed on disk and the incore inodes can be reclaimed * before we signal that recovery is complete. */ -xfs_inodegc_flush (mp); +xfs_inodegc_flush (log->l_mp); }STATIC void
Signature · 1 line
2.36.1
Re: [PATCH 3/9] xfs: refactor xlog_recover_process_iunlinks()
Quoted 9 lines · Dave Chinner
From: Dave Chinner <[email protected]>
For upcoming changes to the way inode unlinked list processing is done, the structure of recovery needs to change slightly. We also really need to untangle the messy error handling in list recovery so that actions like emptying the bucket on inode lookup failure are associated with the bucket list walk failing, not failing to look up the inode.
Quoted 5 lines
Refactor the recovery code now to keep the re-organisation seperate to the algorithm changes.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>
Seems pretty simple
--D
Quoted 182 lines
--- fs/xfs/xfs_log_recover.c | 135 ++++++++++++++++++++------------------- 1 file changed, 70 insertions(+), 65 deletions(-)fs/xfs/xfs_log_recover.c +70
-65@@ -2667,40 +2667,35 @@ xlog_recover_clear_agi_bucket(return ; } -STATIC xfs_agino_t -xlog_recover_process_one_iunlink ( -struct xfs_perag *pag, -xfs_agino_t agino, -int bucket) +static int +xlog_recover_iunlink_bucket ( +struct xfs_perag *pag, +struct xfs_agi *agi, +int bucket) { -struct xfs_inode *ip; -xfs_ino_t ino; -int error; +struct xfs_mount *mp = pag->pag_mount; +struct xfs_inode *ip; +xfs_agino_t agino; - ino =XFS_AGINO_TO_INO (pag->pag_mount, pag->pag_agno, agino); - error =xfs_iget (pag->pag_mount,NULL , ino,0 ,0 , &ip); -if (error) - goto fail; + agino =be32_to_cpu (agi->agi_unlinked[bucket]); +while (agino !=NULLAGINO ) { +int error; -xfs_iflags_clear (ip,XFS_IRECOVERY ); -ASSERT (VFS_I (ip)->i_nlink ==0 ); -ASSERT (VFS_I (ip)->i_mode !=0 ); + error =xfs_iget (mp,NULL , +XFS_AGINO_TO_INO (mp, pag->pag_agno, agino), +0 ,0 , &ip); +if (error) +return error;; - agino = ip->i_next_unlinked; -xfs_irele (ip); -return agino; +ASSERT (VFS_I (ip)->i_nlink ==0 ); +ASSERT (VFS_I (ip)->i_mode !=0 ); +xfs_iflags_clear (ip,XFS_IRECOVERY ); + agino = ip->i_next_unlinked; - fail: -/* -* We can't read in the inode this bucket points to, or this inode -* is messed up. Just ditch this bucket of inodes. We will lose -* some inodes and space, but at least we won't hang. -* -* Call xlog_recover_clear_agi_bucket() to perform a transaction to -* clear the inode pointer in the bucket. -*/ -xlog_recover_clear_agi_bucket (pag, bucket); -return NULLAGINO ; +xfs_irele (ip); +cond_resched (); + } +return 0 ; }/* @@ -2726,59 +2721,69 @@ xlog_recover_process_one_iunlink(* scheduled on this CPU to ensure other scheduled work can run without undue * latency. */ -STATIC void -xlog_recover_process_iunlinks ( -struct xlog *log) +static void +xlog_recover_iunlink_ag ( +struct xfs_perag *pag) { -struct xfs_mount *mp = log->l_mp; -struct xfs_perag *pag; -xfs_agnumber_t agno;struct xfs_agi *agi;struct xfs_buf *agibp; -xfs_agino_t agino;int bucket;int error; -for_each_perag (mp, agno, pag) { - error =xfs_read_agi (pag,NULL , &agibp); + error =xfs_read_agi (pag,NULL , &agibp); +if (error) { +/* +* AGI is b0rked. Don't process it. +* +* We should probably mark the filesystem as corrupt after we've +* recovered all the ag's we can.... +*/ +return ; + } + +/* +* Unlock the buffer so that it can be acquired in the normal course of +* the transaction to truncate and free each inode. Because we are not +* racing with anyone else here for the AGI buffer, we don't even need +* to hold it locked to read the initial unlinked bucket entries out of +* the buffer. We keep buffer reference though, so that it stays pinned +* in memory while we need the buffer. +*/ + agi = agibp->b_addr; +xfs_buf_unlock (agibp); + +for (bucket =0 ; bucket <XFS_AGI_UNLINKED_BUCKETS ; bucket++) { + error =xlog_recover_iunlink_bucket (pag, agi, bucket);if (error) {/* -* AGI is b0rked. Don't process it. -* -* We should probably mark the filesystem as corrupt -* after we've recovered all the ag's we can.... +* Bucket is unrecoverable, so only a repair scan can +* free the remaining unlinked inodes. Just empty the +* bucket and remaining inodes on it unreferenced and +* unfreeable. */ -continue ; - } -/* -* Unlock the buffer so that it can be acquired in the normal -* course of the transaction to truncate and free each inode. -* Because we are not racing with anyone else here for the AGI -* buffer, we don't even need to hold it locked to read the -* initial unlinked bucket entries out of the buffer. We keep -* buffer reference though, so that it stays pinned in memory -* while we need the buffer. -*/ - agi = agibp->b_addr; -xfs_buf_unlock (agibp); - -for (bucket =0 ; bucket <XFS_AGI_UNLINKED_BUCKETS ; bucket++) { - agino =be32_to_cpu (agi->agi_unlinked[bucket]); -while (agino !=NULLAGINO ) { - agino =xlog_recover_process_one_iunlink (pag, - agino, bucket); -cond_resched (); - } +xlog_recover_clear_agi_bucket (pag, bucket); } -xfs_buf_rele (agibp); } +xfs_buf_rele (agibp); +} + +static void +xlog_recover_process_iunlinks ( +struct xlog *log) +{ +struct xfs_perag *pag; +xfs_agnumber_t agno; + +for_each_perag (log->l_mp, agno, pag) +xlog_recover_iunlink_ag (pag); +/* * Flush the pending unlinked inodes to ensure that the inactivations * are fully completed on disk and the incore inodes can be reclaimed * before we signal that recovery is complete. */ -xfs_inodegc_flush (mp); +xfs_inodegc_flush (log->l_mp); }STATIC voidSignature · 1 line
2.36.1
PATCH v4 4/9 xfs: introduce xfs_iunlink_lookup
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When an inode is on an unlinked list during normal operation, it is guaranteed to be pinned in memory as it is either referenced by the current unlink operation or it has a open file descriptor that references it and has it pinned in memory. Hence to look up an inode on the unlinked list, we can do a direct inode cache lookup and always expect the lookup to succeed.
Add a function to do this lookup based on the agino that we use to link the chain of unlinked inodes together so we can begin the conversion the unlinked list manipulations to use in-memory inodes rather than inode cluster buffers and remove the backref cache.
Use this lookup function to replace the on-disk inode buffer walk when removing inodes from the unlinked list with an in-core inode unlinked list walk.
--- fs/xfs/xfs_inode.c | 161 +++++++++++++++++++-------------------------- fs/xfs/xfs_trace.h | 1 - 2 files changed, 66 insertions(+), 96 deletions(-)
fs/xfs/xfs_inode.c +66 -95
@@ -1992,6 +1992,35 @@ xfs_iunlink_destroy(ASSERT (freed_anything == false ||xfs_is_shutdown (pag->pag_mount)); } +/* +* Find an inode on the unlinked list. This does not take references to the +* inode as we have existence guarantees by holding the AGI buffer lock and that +* only unlinked, referenced inodes can be on the unlinked inode list. If we +* don't find the inode in cache, then let the caller handle the situation. +*/ +static struct xfs_inode * +xfs_iunlink_lookup ( +struct xfs_perag *pag, +xfs_agino_t agino) +{ +struct xfs_inode *ip; + +rcu_read_lock (); + ip =radix_tree_lookup (&pag->pag_ici_root, agino); + +/* +* Inode not in memory or in RCU freeing limbo should not happen. +* Warn about this and let the caller handle the failure. +*/ +if (WARN_ON_ONCE (!ip || !ip->i_ino)) { +rcu_read_unlock (); +return NULL ; + } +ASSERT (!xfs_iflags_test (ip,XFS_IRECLAIMABLE |XFS_IRECLAIM )); +rcu_read_unlock (); +return ip; +} +/* * Point the AGI unlinked bucket at an inode and log the results. The caller * is responsible for validating the old value.
@@ -2097,7 +2126,8 @@ xfs_iunlink_update_inode(* current pointer is the same as the new value, unless we're * terminating the list. */ - *old_next_agino = old_value; +if (old_next_agino) + *old_next_agino = old_value;if (old_value == next_agino) {if (next_agino !=NULLAGINO ) { xfs_inode_verifier_error(ip, -EFSCORRUPTED, __func__,
@@ -2203,38 +2233,6 @@ xfs_iunlink(return error; } -/* Return the imap, dinode pointer, and buffer for an inode. */ -STATIC int -xfs_iunlink_map_ino ( -struct xfs_trans *tp, -xfs_agnumber_t agno, -xfs_agino_t agino, -struct xfs_imap *imap, -struct xfs_dinode **dipp, -struct xfs_buf **bpp) -{ -struct xfs_mount *mp = tp->t_mountp; -int error; - - imap->im_blkno =0 ; - error =xfs_imap (mp, tp,XFS_AGINO_TO_INO (mp, agno, agino), imap,0 ); -if (error) { -xfs_warn (mp,"%s: xfs_imap returned error %d." , - __func__, error); -return error; - } - - error =xfs_imap_to_bp (mp, tp, imap, bpp); -if (error) { -xfs_warn (mp,"%s: xfs_imap_to_bp returned error %d." , - __func__, error); -return error; - } - - *dipp =xfs_buf_offset (*bpp, imap->im_boffset); -return 0 ; -} -/* * Walk the unlinked chain from @head_agino until we find the inode that * points to @target_agino. Return the inode number, map, dinode pointer,
@@ -2245,77 +2243,49 @@ xfs_iunlink_map_ino(* * Do not call this function if @target_agino is the head of the list. */ -STATIC int -xfs_iunlink_map_prev ( -struct xfs_trans *tp, +static int +xfs_iunlink_lookup_prev (struct xfs_perag *pag,xfs_agino_t head_agino,xfs_agino_t target_agino, -xfs_agino_t *agino, -struct xfs_imap *imap, -struct xfs_dinode **dipp, -struct xfs_buf **bpp) +struct xfs_inode **ipp) { -struct xfs_mount *mp = tp->t_mountp; +struct xfs_inode *ip;xfs_agino_t next_agino; -int error; - -ASSERT (head_agino != target_agino); - *bpp =NULL ; -/* See if our backref cache can find it faster. */ - *agino =xfs_iunlink_lookup_backref (pag, target_agino); -if (*agino !=NULLAGINO ) { - error =xfs_iunlink_map_ino (tp, pag->pag_agno, *agino, imap, - dipp, bpp); -if (error) -return error; + *ipp =NULL ; -if (be32_to_cpu ((*dipp)->di_next_unlinked) == target_agino) + next_agino =xfs_iunlink_lookup_backref (pag, target_agino); +if (next_agino !=NULLAGINO ) { + ip =xfs_iunlink_lookup (pag, next_agino); +if (ip && ip->i_next_unlinked == target_agino) { + *ipp = ip;return 0 ; - -/* -* If we get here the cache contents were corrupt, so drop the -* buffer and fall back to walking the bucket list. -*/ -xfs_trans_brelse (tp, *bpp); - *bpp =NULL ; -WARN_ON_ONCE (1 ); + } } -trace_xfs_iunlink_map_prev_fallback (mp, pag->pag_agno); -/* Otherwise, walk the entire bucket until we find it. */ next_agino = head_agino; -while (next_agino != target_agino) { -xfs_agino_t unlinked_agino; - -if (*bpp) -xfs_trans_brelse (tp, *bpp); +while (next_agino !=NULLAGINO ) { + ip =xfs_iunlink_lookup (pag, next_agino); +if (!ip) +return -EFSCORRUPTED ; - *agino = next_agino; - error =xfs_iunlink_map_ino (tp, pag->pag_agno, next_agino, imap, - dipp, bpp); -if (error) -return error; - - unlinked_agino =be32_to_cpu ((*dipp)->di_next_unlinked);/* * Make sure this pointer is valid and isn't an obvious * infinite loop. */ -if (!xfs_verify_agino (pag, unlinked_agino) || - next_agino == unlinked_agino) { -XFS_CORRUPTION_ERROR (__func__, -XFS_ERRLEVEL_LOW , mp, - *dipp,sizeof (**dipp)); - error = -EFSCORRUPTED ; -return error; +if (!xfs_verify_agino (pag, ip->i_next_unlinked) || + next_agino == ip->i_next_unlinked) +return -EFSCORRUPTED ; + +if (ip->i_next_unlinked == target_agino) { + *ipp = ip; +return 0 ; } - next_agino = unlinked_agino; + next_agino = ip->i_next_unlinked; } - -return 0 ; +return -EFSCORRUPTED ; } static int
@@ -2327,8 +2297,6 @@ xfs_iunlink_remove_inode( {struct xfs_mount *mp = tp->t_mountp;struct xfs_agi *agi = agibp->b_addr; -struct xfs_buf *last_ibp; -struct xfs_dinode *last_dip =NULL ;xfs_agino_t agino =XFS_INO_TO_AGINO (mp, ip->i_ino);xfs_agino_t next_agino;xfs_agino_t head_agino;
@@ -2356,7 +2324,6 @@ xfs_iunlink_remove_inode( error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO , &next_agino);if (error)return error; - ip->i_next_unlinked =NULLAGINO ;/* * If there was a backref pointing from the next inode back to this
@@ -2372,18 +2339,21 @@ xfs_iunlink_remove_inode( }if (head_agino != agino) { -struct xfs_imap imap; -xfs_agino_t prev_agino; +struct xfs_inode *prev_ip; -/* We need to search the list for the inode being freed. */ - error =xfs_iunlink_map_prev (tp, pag, head_agino, agino, - &prev_agino, &imap, &last_dip, &last_ibp); + error =xfs_iunlink_lookup_prev (pag, head_agino, agino, + &prev_ip);if (error)return error;/* Point the previous inode on the list to the next inode. */ -xfs_iunlink_update_dinode (tp, pag, prev_agino, last_ibp, - last_dip, &imap, next_agino); + error =xfs_iunlink_update_inode (tp, prev_ip, pag, next_agino, +NULL ); +if (error) +return error; + + prev_ip->i_next_unlinked = ip->i_next_unlinked; + ip->i_next_unlinked =NULLAGINO ;/* * Now we deal with the backref for this inode. If this inode
@@ -2398,6 +2368,7 @@ xfs_iunlink_remove_inode( }/* Point the head of the list to the next unlinked inode. */ + ip->i_next_unlinked =NULLAGINO ;return xfs_iunlink_update_bucket (tp, pag, agibp, bucket_index, next_agino); }
fs/xfs/xfs_trace.h +0 -1
@@ -3672,7 +3672,6 @@ DEFINE_EVENT(xfs_ag_inode_class, name, \ TP_ARGS(ip))DEFINE_AGINODE_EVENT (xfs_iunlink);DEFINE_AGINODE_EVENT (xfs_iunlink_remove); -DEFINE_AG_EVENT (xfs_iunlink_map_prev_fallback); DECLARE_EVENT_CLASS(xfs_fs_corrupt_class, TP_PROTO(struct xfs_mount *mp, unsigned int flags),
Signature · 1 line
2.36.1
Re: [PATCH 4/9] xfs: introduce xfs_iunlink_lookup
Looks good:
Re: [PATCH 4/9] xfs: introduce xfs_iunlink_lookup
In reply to Dave ChinnerQuoted 14 lines · Dave Chinner
From: Dave Chinner <[email protected]>
When an inode is on an unlinked list during normal operation, it is guaranteed to be pinned in memory as it is either referenced by the current unlink operation or it has a open file descriptor that references it and has it pinned in memory. Hence to look up an inode on the unlinked list, we can do a direct inode cache lookup and always expect the lookup to succeed.
Add a function to do this lookup based on the agino that we use to link the chain of unlinked inodes together so we can begin the conversion the unlinked list manipulations to use in-memory inodes rather than inode cluster buffers and remove the backref cache.
Quoted 5 lines
Use this lookup function to replace the on-disk inode buffer walk when removing inodes from the unlinked list with an in-core inode unlinked list walk.
Signed-off-by Dave Chinner <[email protected]>
Looks good!
--D
Quoted 265 lines
--- fs/xfs/xfs_inode.c | 161 +++++++++++++++++++-------------------------- fs/xfs/xfs_trace.h | 1 - 2 files changed, 66 insertions(+), 96 deletions(-)fs/xfs/xfs_inode.c +66
-95@@ -1992,6 +1992,35 @@ xfs_iunlink_destroy(ASSERT (freed_anything == false ||xfs_is_shutdown (pag->pag_mount)); } +/* +* Find an inode on the unlinked list. This does not take references to the +* inode as we have existence guarantees by holding the AGI buffer lock and that +* only unlinked, referenced inodes can be on the unlinked inode list. If we +* don't find the inode in cache, then let the caller handle the situation. +*/ +static struct xfs_inode * +xfs_iunlink_lookup ( +struct xfs_perag *pag, +xfs_agino_t agino) +{ +struct xfs_inode *ip; + +rcu_read_lock (); + ip =radix_tree_lookup (&pag->pag_ici_root, agino); + +/* +* Inode not in memory or in RCU freeing limbo should not happen. +* Warn about this and let the caller handle the failure. +*/ +if (WARN_ON_ONCE (!ip || !ip->i_ino)) { +rcu_read_unlock (); +return NULL ; + } +ASSERT (!xfs_iflags_test (ip,XFS_IRECLAIMABLE |XFS_IRECLAIM )); +rcu_read_unlock (); +return ip; +} +/* * Point the AGI unlinked bucket at an inode and log the results. The caller * is responsible for validating the old value. @@ -2097,7 +2126,8 @@ xfs_iunlink_update_inode(* current pointer is the same as the new value, unless we're * terminating the list. */ - *old_next_agino = old_value; +if (old_next_agino) + *old_next_agino = old_value;if (old_value == next_agino) {if (next_agino !=NULLAGINO ) { xfs_inode_verifier_error(ip, -EFSCORRUPTED, __func__,@@ -2203,38 +2233,6 @@ xfs_iunlink(return error; } -/* Return the imap, dinode pointer, and buffer for an inode. */ -STATIC int -xfs_iunlink_map_ino ( -struct xfs_trans *tp, -xfs_agnumber_t agno, -xfs_agino_t agino, -struct xfs_imap *imap, -struct xfs_dinode **dipp, -struct xfs_buf **bpp) -{ -struct xfs_mount *mp = tp->t_mountp; -int error; - - imap->im_blkno =0 ; - error =xfs_imap (mp, tp,XFS_AGINO_TO_INO (mp, agno, agino), imap,0 ); -if (error) { -xfs_warn (mp,"%s: xfs_imap returned error %d." , - __func__, error); -return error; - } - - error =xfs_imap_to_bp (mp, tp, imap, bpp); -if (error) { -xfs_warn (mp,"%s: xfs_imap_to_bp returned error %d." , - __func__, error); -return error; - } - - *dipp =xfs_buf_offset (*bpp, imap->im_boffset); -return 0 ; -} -/* * Walk the unlinked chain from @head_agino until we find the inode that * points to @target_agino. Return the inode number, map, dinode pointer, @@ -2245,77 +2243,49 @@ xfs_iunlink_map_ino(* * Do not call this function if @target_agino is the head of the list. */ -STATIC int -xfs_iunlink_map_prev ( -struct xfs_trans *tp, +static int +xfs_iunlink_lookup_prev (struct xfs_perag *pag,xfs_agino_t head_agino,xfs_agino_t target_agino, -xfs_agino_t *agino, -struct xfs_imap *imap, -struct xfs_dinode **dipp, -struct xfs_buf **bpp) +struct xfs_inode **ipp) { -struct xfs_mount *mp = tp->t_mountp; +struct xfs_inode *ip;xfs_agino_t next_agino; -int error; - -ASSERT (head_agino != target_agino); - *bpp =NULL ; -/* See if our backref cache can find it faster. */ - *agino =xfs_iunlink_lookup_backref (pag, target_agino); -if (*agino !=NULLAGINO ) { - error =xfs_iunlink_map_ino (tp, pag->pag_agno, *agino, imap, - dipp, bpp); -if (error) -return error; + *ipp =NULL ; -if (be32_to_cpu ((*dipp)->di_next_unlinked) == target_agino) + next_agino =xfs_iunlink_lookup_backref (pag, target_agino); +if (next_agino !=NULLAGINO ) { + ip =xfs_iunlink_lookup (pag, next_agino); +if (ip && ip->i_next_unlinked == target_agino) { + *ipp = ip;return 0 ; - -/* -* If we get here the cache contents were corrupt, so drop the -* buffer and fall back to walking the bucket list. -*/ -xfs_trans_brelse (tp, *bpp); - *bpp =NULL ; -WARN_ON_ONCE (1 ); + } } -trace_xfs_iunlink_map_prev_fallback (mp, pag->pag_agno); -/* Otherwise, walk the entire bucket until we find it. */ next_agino = head_agino; -while (next_agino != target_agino) { -xfs_agino_t unlinked_agino; - -if (*bpp) -xfs_trans_brelse (tp, *bpp); +while (next_agino !=NULLAGINO ) { + ip =xfs_iunlink_lookup (pag, next_agino); +if (!ip) +return -EFSCORRUPTED ; - *agino = next_agino; - error =xfs_iunlink_map_ino (tp, pag->pag_agno, next_agino, imap, - dipp, bpp); -if (error) -return error; - - unlinked_agino =be32_to_cpu ((*dipp)->di_next_unlinked);/* * Make sure this pointer is valid and isn't an obvious * infinite loop. */ -if (!xfs_verify_agino (pag, unlinked_agino) || - next_agino == unlinked_agino) { -XFS_CORRUPTION_ERROR (__func__, -XFS_ERRLEVEL_LOW , mp, - *dipp,sizeof (**dipp)); - error = -EFSCORRUPTED ; -return error; +if (!xfs_verify_agino (pag, ip->i_next_unlinked) || + next_agino == ip->i_next_unlinked) +return -EFSCORRUPTED ; + +if (ip->i_next_unlinked == target_agino) { + *ipp = ip; +return 0 ; } - next_agino = unlinked_agino; + next_agino = ip->i_next_unlinked; } - -return 0 ; +return -EFSCORRUPTED ; } static int@@ -2327,8 +2297,6 @@ xfs_iunlink_remove_inode( {struct xfs_mount *mp = tp->t_mountp;struct xfs_agi *agi = agibp->b_addr; -struct xfs_buf *last_ibp; -struct xfs_dinode *last_dip =NULL ;xfs_agino_t agino =XFS_INO_TO_AGINO (mp, ip->i_ino);xfs_agino_t next_agino;xfs_agino_t head_agino;@@ -2356,7 +2324,6 @@ xfs_iunlink_remove_inode( error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO , &next_agino);if (error)return error; - ip->i_next_unlinked =NULLAGINO ;/* * If there was a backref pointing from the next inode back to this @@ -2372,18 +2339,21 @@ xfs_iunlink_remove_inode( }if (head_agino != agino) { -struct xfs_imap imap; -xfs_agino_t prev_agino; +struct xfs_inode *prev_ip; -/* We need to search the list for the inode being freed. */ - error =xfs_iunlink_map_prev (tp, pag, head_agino, agino, - &prev_agino, &imap, &last_dip, &last_ibp); + error =xfs_iunlink_lookup_prev (pag, head_agino, agino, + &prev_ip);if (error)return error;/* Point the previous inode on the list to the next inode. */ -xfs_iunlink_update_dinode (tp, pag, prev_agino, last_ibp, - last_dip, &imap, next_agino); + error =xfs_iunlink_update_inode (tp, prev_ip, pag, next_agino, +NULL ); +if (error) +return error; + + prev_ip->i_next_unlinked = ip->i_next_unlinked; + ip->i_next_unlinked =NULLAGINO ;/* * Now we deal with the backref for this inode. If this inode @@ -2398,6 +2368,7 @@ xfs_iunlink_remove_inode( }/* Point the head of the list to the next unlinked inode. */ + ip->i_next_unlinked =NULLAGINO ;return xfs_iunlink_update_bucket (tp, pag, agibp, bucket_index, next_agino); }fs/xfs/xfs_trace.h +0
-1@@ -3672,7 +3672,6 @@ DEFINE_EVENT(xfs_ag_inode_class, name, \ TP_ARGS(ip))DEFINE_AGINODE_EVENT (xfs_iunlink);DEFINE_AGINODE_EVENT (xfs_iunlink_remove); -DEFINE_AG_EVENT (xfs_iunlink_map_prev_fallback); DECLARE_EVENT_CLASS(xfs_fs_corrupt_class, TP_PROTO(struct xfs_mount *mp, unsigned int flags),Signature · 1 line
2.36.1
PATCH v4 5/9 xfs: double link the unlinked inode list
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now we have forwards traversal via the incore inode in place, we now need to add back pointers to the incore inode to entirely replace the back reference cache. We use the same lookup semantics and constraints as for the forwards pointer lookups during unlinks, and so we can look up any inode in the unlinked list directly and update the list pointers, forwards or backwards, at any time.
The only wrinkle in converting the unlinked list manipulations to use in-core previous pointers is that log recovery doesn't have the incore inode state built up so it can't just read in an inode and release it to finish off the unlink. Hence we need to modify the traversal in recovery to read one inode ahead before we release the inode at the head of the list. This populates the next->prev relationship sufficient to be able to replay the unlinked list and hence greatly simplify the runtime code.
This recovery algorithm also requires that we actually remove inodes from the unlinked list one at a time as background inode inactivation will result in unlinked list removal racing with the building of the in-memory unlinked list state. We could serialise this by holding the AGI buffer lock when constructing the in memory state, but all that does is lockstep background processing with list building. It is much simpler to flush the inodegc immediately after releasing the inode so that it is unlinked immediately and there is no races present at all.
--- fs/xfs/libxfs/xfs_ag.c | 8 - fs/xfs/libxfs/xfs_ag.h | 6 - fs/xfs/xfs_icache.c | 2 + fs/xfs/xfs_inode.c | 344 +++++++-------------------------------- fs/xfs/xfs_inode.h | 4 +- fs/xfs/xfs_log_recover.c | 36 +++- 6 files changed, 90 insertions(+), 310 deletions(-)
fs/xfs/libxfs/xfs_ag.c +0 -8
@@ -194,7 +194,6 @@ xfs_free_perag(XFS_IS_CORRUPT (pag->pag_mount,atomic_read (&pag->pag_ref) !=0 );cancel_delayed_work_sync (&pag->pag_blockgc_work); -xfs_iunlink_destroy (pag);xfs_buf_hash_destroy (pag);call_rcu (&pag->rcu_head, __xfs_free_perag);
@@ -323,10 +322,6 @@ xfs_initialize_perag(if (error) goto out_remove_pag; - error =xfs_iunlink_init (pag); -if (error) - goto out_hash_destroy; -/* first new pag is fully initialized */ if (first_initialised ==NULLAGNUMBER ) first_initialised = index;
@@ -349,8 +344,6 @@ xfs_initialize_perag( mp->m_ag_prealloc_blocks =xfs_prealloc_blocks (mp);return 0 ; -out_hash_destroy: -xfs_buf_hash_destroy (pag); out_remove_pag:radix_tree_delete (&mp->m_perag_tree, index); out_free_pag:
@@ -362,7 +355,6 @@ xfs_initialize_perag(if (!pag)break ;xfs_buf_hash_destroy (pag); -xfs_iunlink_destroy (pag);kmem_free (pag); }return error;
fs/xfs/libxfs/xfs_ag.h +0 -6
@@ -103,12 +103,6 @@ struct xfs_perag {/* background prealloc block trimming */ struct delayed_work pag_blockgc_work; -/* -* Unlinked inode information. This incore information reflects -* data stored in the AGI, so callers must hold the AGI buffer lock -* or have some other means to control concurrency. -*/ -struct rhashtable pagi_unlinked_hash;#endif /* __KERNEL__ */ };
fs/xfs/xfs_icache.c +2 -0
@@ -111,6 +111,8 @@ xfs_inode_alloc(INIT_WORK (&ip->i_ioend_work, xfs_end_io);INIT_LIST_HEAD (&ip->i_ioend_list);spin_lock_init (&ip->i_ioend_lock); + ip->i_next_unlinked =NULLAGINO ; + ip->i_prev_unlinked =NULLAGINO ;return ip; }
fs/xfs/xfs_inode.c +58 -286
393 lines not on this page · raw message
fs/xfs/xfs_inode.h +1 -3
@@ -70,6 +70,7 @@ typedef struct xfs_inode {/* unlinked list pointers */ xfs_agino_t i_next_unlinked; +xfs_agino_t i_prev_unlinked;/* VFS inode */ struct inode i_vnode;/* embedded VFS inode */
@@ -508,9 +509,6 @@ extern struct kmem_cache *xfs_inode_cache;bool xfs_inode_needs_inactive (struct xfs_inode *ip); -int xfs_iunlink_init (struct xfs_perag *pag); -void xfs_iunlink_destroy (struct xfs_perag *pag); -void xfs_end_io (struct work_struct *work);int xfs_ilock2_io_mmap (struct xfs_inode *ip1,struct xfs_inode *ip2);
fs/xfs/xfs_log_recover.c +29 -7
@@ -2674,28 +2674,50 @@ xlog_recover_iunlink_bucket( int bucket) {struct xfs_mount *mp = pag->pag_mount; +struct xfs_inode *prev_ip =NULL ;struct xfs_inode *ip; -xfs_agino_t agino; +xfs_agino_t prev_agino, agino; +int error =0 ; agino =be32_to_cpu (agi->agi_unlinked[bucket]);while (agino !=NULLAGINO ) { -int error; - error =xfs_iget (mp,NULL ,XFS_AGINO_TO_INO (mp, pag->pag_agno, agino),0 ,0 , &ip);if (error) -return error;; +break ;ASSERT (VFS_I (ip)->i_nlink ==0 );ASSERT (VFS_I (ip)->i_mode !=0 );xfs_iflags_clear (ip,XFS_IRECOVERY ); agino = ip->i_next_unlinked; -xfs_irele (ip); -cond_resched (); +if (prev_ip) { + ip->i_prev_unlinked = prev_agino; +xfs_irele (prev_ip); + +/* +* Ensure the inode is removed from the unlinked list +* before we continue so that it won't race with +* building the in-memory list here. This could be +* serialised with the agibp lock, but that just +* serialises via lockstepping and it's much simpler +* just to flush the inodegc queue and wait for it to +* complete. +*/ +xfs_inodegc_flush (mp); + } + + prev_agino = agino; + prev_ip = ip; } -return 0 ; + +if (prev_ip) { + ip->i_prev_unlinked = prev_agino; +xfs_irele (prev_ip); + } +xfs_inodegc_flush (mp); +return error; }/*
Signature · 1 line
2.36.1
Re: [PATCH 5/9] xfs: double link the unlinked inode list
Quoted 9 lines · Dave Chinner
*
- * What if we modelled the unlinked list as a collection of records capturing
- * "X.next_unlinked = Y" relations? If we indexed those records on Y, we'd
- * have a fast way to look up unlinked list predecessors, which avoids the
- * slow list walk. That's exactly what we do here (in-core) with a per-AG
- * rhashtable.
+ * Hence we keep an in-memory double linked list to link each inode on an
+ * unlinked list. Because there are 64 unlinked lists per AGI, keeping pointer
+ * based lists qould require having 64 list heads in the perag, one for each
I think this qould should be would.
Otherwise looks good:
PATCH v4 6/9 xfs: clean up xfs_iunlink_update_inode()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
We no longer need to have this function return the previous next agino value from the on-disk inode as we have it in the in-core inode now.
--- fs/xfs/xfs_inode.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-)
fs/xfs/xfs_inode.c +5 -13
@@ -1935,13 +1935,12 @@ xfs_iunlink_update_dinode( }/* Set an in-core inode's unlinked pointer and return the old value. */ -STATIC int +static int xfs_iunlink_update_inode (struct xfs_trans *tp,struct xfs_inode *ip,struct xfs_perag *pag, -xfs_agino_t next_agino, -xfs_agino_t *old_next_agino) +xfs_agino_t next_agino) {struct xfs_mount *mp = tp->t_mountp;struct xfs_dinode *dip;
@@ -1971,8 +1970,6 @@ xfs_iunlink_update_inode(* current pointer is the same as the new value, unless we're * terminating the list. */ -if (old_next_agino) - *old_next_agino = old_value;if (old_value == next_agino) {if (next_agino !=NULLAGINO ) { xfs_inode_verifier_error(ip, -EFSCORRUPTED, __func__,
@@ -2026,17 +2023,13 @@ xfs_iunlink_insert_inode(return error;if (next_agino !=NULLAGINO ) { -xfs_agino_t old_agino; -/* * There is already another inode in the bucket, so point this * inode to the current head of the list. */ - error =xfs_iunlink_update_inode (tp, ip, pag, next_agino, - &old_agino); + error =xfs_iunlink_update_inode (tp, ip, pag, next_agino);if (error)return error; -ASSERT (old_agino ==NULLAGINO ); ip->i_next_unlinked = next_agino; }
@@ -2088,7 +2081,6 @@ xfs_iunlink_remove_inode(struct xfs_mount *mp = tp->t_mountp;struct xfs_agi *agi = agibp->b_addr;xfs_agino_t agino =XFS_INO_TO_AGINO (mp, ip->i_ino); -xfs_agino_t next_agino;xfs_agino_t head_agino;short bucket_index = agino %XFS_AGI_UNLINKED_BUCKETS ;int error;
@@ -2111,7 +2103,7 @@ xfs_iunlink_remove_inode(* the old pointer value so that we can update whatever was previous * to us in the list to point to whatever was next in the list. */ - error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO , &next_agino); + error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO );if (error)return error;
@@ -2132,7 +2124,7 @@ xfs_iunlink_remove_inode(return -EFSCORRUPTED ; error =xfs_iunlink_update_inode (tp, prev_ip, pag, - ip->i_next_unlinked,NULL ); + ip->i_next_unlinked); prev_ip->i_next_unlinked = ip->i_next_unlinked; }else {/* Point the head of the list to the next unlinked inode. */
Signature · 1 line
2.36.1
PATCH v4 7/9 xfs: combine iunlink inode update functions
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Combine the logging of the inode unlink list update into the calling function that looks up the buffer we end up logging. These do not need to be separate functions as they are both short, simple operations and there's only a single call path through them. This new function will end up being the core of the iunlink log item processing...
--- fs/xfs/xfs_inode.c | 52 ++++++++++++++-------------------------------- 1 file changed, 16 insertions(+), 36 deletions(-)
fs/xfs/xfs_inode.c +16 -36
@@ -1905,38 +1905,9 @@ xfs_iunlink_update_bucket(return 0 ; } -/* Set an on-disk inode's next_unlinked pointer. */ -STATIC void -xfs_iunlink_update_dinode ( -struct xfs_trans *tp, -struct xfs_perag *pag, -xfs_agino_t agino, -struct xfs_buf *ibp, -struct xfs_dinode *dip, -struct xfs_imap *imap, -xfs_agino_t next_agino) -{ -struct xfs_mount *mp = tp->t_mountp; -int offset; - -ASSERT (xfs_verify_agino_or_null (pag, next_agino)); - -trace_xfs_iunlink_update_dinode (mp, pag->pag_agno, agino, -be32_to_cpu (dip->di_next_unlinked), next_agino); - - dip->di_next_unlinked =cpu_to_be32 (next_agino); - offset = imap->im_boffset + - offsetof(struct xfs_dinode , di_next_unlinked); - -/* need to recalc the inode CRC if appropriate */ -xfs_dinode_calc_crc (mp, dip); -xfs_trans_inode_buf (tp, ibp); -xfs_trans_log_buf (tp, ibp, offset, offset +sizeof (xfs_agino_t) -1 ); -} -/* Set an in-core inode's unlinked pointer and return the old value. */ static int -xfs_iunlink_update_inode( +xfs_iunlink_log_inode( struct xfs_trans *tp, struct xfs_inode *ip, struct xfs_perag *pag,
@@ -1946,6 +1917,7 @@ xfs_iunlink_update_inode(struct xfs_dinode *dip;struct xfs_buf *ibp;xfs_agino_t old_value; +int offset;int error;ASSERT (xfs_verify_agino_or_null (pag, next_agino));
@@ -1979,9 +1951,17 @@ xfs_iunlink_update_inode( goto out; } -/* Ok, update the new pointer. */ -xfs_iunlink_update_dinode (tp, pag,XFS_INO_TO_AGINO (mp, ip->i_ino), - ibp, dip, &ip->i_imap, next_agino); +trace_xfs_iunlink_update_dinode (mp, pag->pag_agno, +XFS_INO_TO_AGINO (mp, ip->i_ino), +be32_to_cpu (dip->di_next_unlinked), next_agino); + + dip->di_next_unlinked =cpu_to_be32 (next_agino); + offset = ip->i_imap.im_boffset + + offsetof(struct xfs_dinode , di_next_unlinked); + +xfs_dinode_calc_crc (mp, dip); +xfs_trans_inode_buf (tp, ibp); +xfs_trans_log_buf (tp, ibp, offset, offset +sizeof (xfs_agino_t) -1 );return 0 ; out:xfs_trans_brelse (tp, ibp);
@@ -2027,7 +2007,7 @@ xfs_iunlink_insert_inode(* There is already another inode in the bucket, so point this * inode to the current head of the list. */ - error =xfs_iunlink_update_inode (tp, ip, pag, next_agino); + error =xfs_iunlink_log_inode (tp, ip, pag, next_agino);if (error)return error; ip->i_next_unlinked = next_agino;
@@ -2103,7 +2083,7 @@ xfs_iunlink_remove_inode(* the old pointer value so that we can update whatever was previous * to us in the list to point to whatever was next in the list. */ - error =xfs_iunlink_update_inode (tp, ip, pag,NULLAGINO ); + error =xfs_iunlink_log_inode (tp, ip, pag,NULLAGINO );if (error)return error;
@@ -2123,7 +2103,7 @@ xfs_iunlink_remove_inode(if (!prev_ip)return -EFSCORRUPTED ; - error =xfs_iunlink_update_inode (tp, prev_ip, pag, + error =xfs_iunlink_log_inode (tp, prev_ip, pag, ip->i_next_unlinked); prev_ip->i_next_unlinked = ip->i_next_unlinked; }else {
Signature · 1 line
2.36.1
PATCH v4 8/9 xfs: add log item precommit operation
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
For inodes that are dirty, we have an attached cluster buffer that we want to use to track the dirty inode through the AIL. Unfortunately, locking the cluster buffer and adding it to the transaction when the inode is first logged in a transaction leads to buffer lock ordering inversions.
The specific problem is ordering against the AGI buffer. When modifying unlinked lists, the buffer lock order is AGI -> inode cluster buffer as the AGI buffer lock serialises all access to the unlinked lists. Unfortunately, functionality like xfs_droplink() logs the inode before calling xfs_iunlink(), as do various directory manipulation functions. The inode can be logged way down in the stack as far as the bmapi routines and hence, without a major rewrite of lots of APIs there's no way we can avoid the inode being logged by something until after the AGI has been logged.
As we are going to be using ordered buffers for inode AIL tracking, there isn't a need to actually lock that buffer against modification as all the modifications are captured by logging the inode item itself. Hence we don't actually need to join the cluster buffer into the transaction until just before it is committed. This means we do not perturb any of the existing buffer lock orders in transactions, and the inode cluster buffer is always locked last in a transaction that doesn't otherwise touch inode cluster buffers.
We do this by introducing a precommit log item method. This commit just introduces the mechanism; the inode item implementation is in followup commits.
The precommit items need to be sorted into consistent order as we may be locking multiple items here. Hence if we have two dirty inodes in cluster buffers A and B, and some other transaction has two separate dirty inodes in the same cluster buffers, locking them in different orders opens us up to ABBA deadlocks. Hence we sort the items on the transaction based on the presence of a sort log item method.
--- fs/xfs/xfs_icache.c | 1 + fs/xfs/xfs_trans.c | 91 +++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_trans.h | 6 ++- 3 files changed, 96 insertions(+), 2 deletions(-)
fs/xfs/xfs_icache.c +1 -0
@@ -914,6 +914,7 @@ xfs_reclaim_inode( ip->i_checked =0 ;spin_unlock (&ip->i_flags_lock); +ASSERT (!ip->i_itemp || ip->i_itemp->ili_item.li_buf ==NULL );xfs_iunlock (ip,XFS_ILOCK_EXCL );XFS_STATS_INC (ip->i_mount, xs_ig_reclaims);
fs/xfs/xfs_trans.c +91 -0
103 lines not on this page · raw message
fs/xfs/xfs_trans.h +4 -2
@@ -71,10 +71,12 @@ struct xfs_item_ops {void (*iop_format)(struct xfs_log_item *,struct xfs_log_vec *);void (*iop_pin)(struct xfs_log_item *);void (*iop_unpin)(struct xfs_log_item *,int remove); -uint (*iop_push)(struct xfs_log_item *,struct list_head *); +uint64_t (*iop_sort)(struct xfs_log_item *lip); +int (*iop_precommit)(struct xfs_trans *tp,struct xfs_log_item *lip);void (*iop_committing)(struct xfs_log_item *lip,xfs_csn_t seq); -void (*iop_release)(struct xfs_log_item *);xfs_lsn_t (*iop_committed)(struct xfs_log_item *,xfs_lsn_t ); +uint (*iop_push)(struct xfs_log_item *,struct list_head *); +void (*iop_release)(struct xfs_log_item *);int (*iop_recover)(struct xfs_log_item *lip,struct list_head *capture_list);bool (*iop_match)(struct xfs_log_item *item,uint64_t id);
Signature · 1 line
2.36.1
Re: [PATCH 8/9] xfs: add log item precommit operation
Looks good:
PATCH v4 9/9 xfs: add in-memory iunlink log item
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we have a clean operation to update the di_next_unlinked field of inode cluster buffers, we can easily defer this operation to transaction commit time so we can order the inode cluster buffer locking consistently.
To do this, we introduce a new in-memory log item to track the unlinked list item modification that we are going to make. This follows the same observations as the in-memory double linked list used to track unlinked inodes in that the inodes on the list are pinned in memory and cannot go away, and hence we can simply reference them for the duration of the transaction without needing to take active references or pin them or look them up.
This allows us to pass the xfs_inode to the transaction commit code along with the modification to be made, and then order the logged modifications via the ->iop_sort and ->iop_precommit operations for the new log item type. As this is an in-memory log item, it doesn't have formatting, CIL or AIL operational hooks - it exists purely to run the inode unlink modifications and is then removed from the transaction item list and freed once the precommit operation has run.
--- fs/xfs/Makefile | 1 + fs/xfs/xfs_inode.c | 64 +------------- fs/xfs/xfs_iunlink_item.c | 180 ++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_iunlink_item.h | 27 ++++++ fs/xfs/xfs_super.c | 10 +++ 5 files changed, 219 insertions(+), 63 deletions(-) create mode 100644 fs/xfs/xfs_iunlink_item.c create mode 100644 fs/xfs/xfs_iunlink_item.h
fs/xfs/Makefile +1 -0
@@ -106,6 +106,7 @@ xfs-y += xfs_log.o \ xfs_icreate_item.o \ xfs_inode_item.o \ xfs_inode_item_recover.o \ + xfs_iunlink_item.o \ xfs_refcount_item.o \ xfs_rmap_item.o \ xfs_log_recover.o \
fs/xfs/xfs_inode.c +1 -63
76 lines not on this page · raw message
fs/xfs/xfs_iunlink_item.c new file +180 -0
180 lines not on this page · raw message
fs/xfs/xfs_iunlink_item.h new file +27 -0
@@ -0,0 +1,27 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Copyright (c) 2020-2022, Red Hat, Inc. +* All Rights Reserved. +*/ +#ifndef XFS_IUNLINK_ITEM_H +#define XFS_IUNLINK_ITEM_H 1 + +struct xfs_trans ; +struct xfs_inode ; +struct xfs_perag ; + +/* in memory log item structure */ +struct xfs_iunlink_item { +struct xfs_log_item item; +struct xfs_inode *ip; +struct xfs_perag *pag; +xfs_agino_t next_agino; +xfs_agino_t old_agino; +}; + +extern struct kmem_cache *xfs_iunlink_cache; + +int xfs_iunlink_log_inode (struct xfs_trans *tp,struct xfs_inode *ip, +struct xfs_perag *pag,xfs_agino_t next_agino); + +#endif /* XFS_IUNLINK_ITEM_H */
fs/xfs/xfs_super.c +10 -0
@@ -40,6 +40,7 @@#include "xfs_defer.h" #include "xfs_attr_item.h" #include "xfs_xattr.h" +#include "xfs_iunlink_item.h" #include <linux/magic.h> #include <linux/fs_context.h>
@@ -2096,8 +2097,16 @@ xfs_init_caches(void)if (!xfs_attri_cache) goto out_destroy_attrd_cache; + xfs_iunlink_cache =kmem_cache_create ("xfs_iul_item" , +sizeof (struct xfs_iunlink_item ), +0 ,0 ,NULL ); +if (!xfs_iunlink_cache) + goto out_destroy_attri_cache; +return 0 ; + out_destroy_attri_cache: +kmem_cache_destroy (xfs_attri_cache); out_destroy_attrd_cache:kmem_cache_destroy (xfs_attrd_cache); out_destroy_bui_cache:
@@ -2148,6 +2157,7 @@ xfs_destroy_caches(void)* destroy caches. */ rcu_barrier (); +kmem_cache_destroy (xfs_iunlink_cache);kmem_cache_destroy (xfs_attri_cache);kmem_cache_destroy (xfs_attrd_cache);kmem_cache_destroy (xfs_bui_cache);
Signature · 1 line
2.36.1
Re: [PATCH 9/9] xfs: add in-memory iunlink log item
Looks good: