xfs: rework inode flushing to make inode reclaim fully asynchronous
cover xfs: rework inode flushing to make inode reclaim fully asynchronous
Hi folks,
Inode flushing requires that we first lock an inode, then check it, then lock the underlying buffer, flush the inode to the buffer and finally add the inode to the buffer to be unlocked on IO completion. We then walk all the other cached inodes in the buffer range and optimistically lock and flush them to the buffer without blocking.
This cluster write effectively repeats the same code we do with the initial inode, except now it has to special case that initial inode that is already locked. Hence we have multiple copies of very similar code, and it is a result of inode cluster flushing being based on a specific inode rather than grabbing the buffer and flushing all available inodes to it.
The problem with this at the moment is that we we can't look up the buffer until we have guaranteed that an inode is held exclusively and it's not going away while we get the buffer through an imap lookup. Hence we are kinda stuck locking an inode before we can look up the buffer.
This is also a result of inodes being detached from the cluster buffer except when IO is being done. This has the further problem that the cluster buffer can be reclaimed from memory and then the inode can be dirtied. At this point cleaning the inode requires a read-modify-write cycle on the cluster buffer. If we then are put under memory pressure, cleaning that dirty inode to reclaim it requires allocating memory for the cluster buffer and this leads to all sorts of problems.
We used synchronous inode writeback in reclaim as a throttle that provided a forwards progress mechanism when RMW cycles were required to clean inodes. Async writeback of inodes (e.g. via the AIL) would immediately exhaust remaining memory reserves trying to allocate inode cluster after inode cluster. The synchronous writeback of an inode cluster allowed reclaim to release the inode cluster and have it freed almost immediately which could then be used to allocate the next inode cluster buffer. Hence the IO based throttling mechanism largely guaranteed forwards progress in inode reclaim. By removing the requirement for require memory allocation for inode writeback filesystem level, we can issue writeback asynchrnously and not have to worry about the memory exhaustion anymore.
Another issue is that if we have slow disks, we can build up dirty inodes in memory that can then take hours for an operation like unmount to flush. A RMW cycle per inode on a slow RAID6 device can mean we only clean 50 inodes a second, and when there are hundreds of thousands of dirty inodes that need to be cleaned this can take a long time. PInning the cluster buffers will greatly speed up inode writeback on slow storage systems like this.
These limitations all stem from the same source: inode writeback is inode centric, And they are largely solved by the same architectural
change: make inode writeback cluster buffer centric. This series is makes that architectural change.
Firstly, we start by pinning the inode backing buffer in memory when an inode is marked dirty (i.e. when it is logged). By tracking the number of dirty inodes on a buffer as a counter rather than a flag, we avoid the problem of overlapping inode dirtying and buffer flushing racing to set/clear the dirty flag. Hence as long as there is a dirty inode in memory, the buffer will not be able to be reclaimed. We can safely do this inode cluster buffer lookup when we dirty an inode as we do not hold the buffer locked - we merely take a reference to it and then release it - and hence we don't cause any new lock order issues.
When the inode is finally cleaned, the reference to the buffer can be removed from the inode log item and the buffer released. This is done from the inode completion callbacks that are attached to the buffer when the inode is flushed.
Pinning the cluster buffer in this way immediately avoids the RMW problem in inode writeback and reclaim contexts by moving the memory allocation and the blocking buffer read into the transaction context that dirties the inode. This inverts our dirty inode throttling mechanism - we now throttle the rate at which we can dirty inodes to rate at which we can allocate memory and read inode cluster buffers into memory rather than via throttling reclaim to rate at which we can clean dirty inodes.
Hence if we are under memory pressure, we'll block on memory allocation when trying to dirty the referenced inode, rather than in the memory reclaim path where we are trying to clean unreferenced inodes to free memory. Hence we no longer have to guarantee forwards progress in inode reclaim as we aren't doing memory allocation, and that means we can remove inode writeback from the XFS inode shrinker completely without changing the system tolerance for low memory operation.
Tracking the buffers via the inode log item also allows us to completely rework the inode flushing mechanism. While the inode log item is in the AIL, it is safe for the AIL to access any member of the log item. Hence the AIL push mechanisms can access the buffer attached to the inode without first having to lock the inode.
This means we can essentially lock the buffer directly and then call xfs_iflush_cluster() without first going through xfs_iflush() to find the buffer. Hence we can remove xfs_iflush() altogether, because the two places that call it - the inode item push code and inode reclaim - no longer need to flush inodes directly.
This can be further optimised by attaching the inode to the cluster buffer when the inode is dirtied. i.e. when we add the buffer reference to the inode log item, we also attach the inode to the buffer for IO processing. This leads to the dirty inodes always being attached to the buffer and hence we no longer need to add them when we flush the inode and remove them when IO completes. Instead the inodes are attached when the node log item is dirtied, and removed when the inode log item is cleaned.
With this structure in place, we no longer need to do lookups to find the dirty inodes in the cache to attach to the buffer in xfs_iflush_cluster() - they are already attached to the buffer. Hence when the AIL pushes an inode, we just grab the buffer from the log item, and then walk the buffer log item list to lock and flush the dirty inodes attached to the buffer.
This greatly simplifies inode writeback, and removes another memory allocation from the inode writeback path (the array used for the radix tree gang lookup). And while the radix tree lookups are fast, walking the linked list of dirty inodes is faster.
There is followup work I am doing that uses the inode cluster buffer as a replacement in the AIL for tracking dirty inodes. This part of the series is not ready yet as it has some intricate locking requirements. That is an optimisation, so I've left that out because solving the inode reclaim blocking problems is the important part of this work.
In short, this series simplifies inode writeback and fixes the long standing inode reclaim blocking issues without requiring any changes to the memory reclaim infrastructure.
Note: dquots should probably be converted to cluster flushing in a similar way, as they have many of the same issues as inode flushing.
Thoughts, comments and improvemnts welcome.
-Dave.
Version 3
git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git xfs-async-inode-reclaim-3
- rebase on 5.7 + for-next
- update comments (p3)
- update commit message (p4)
- renamed xfs_buf_ioerror_sync() (p13)
- added enum for return value from xfs_buf_iodone_error() (p13)
- moved clearing of buffer error to iodone functions (p13)
- whitespace (p13)
- rebase p14 (p13 conflicts)
- rebase p16 (p13 conflicts)
- removed a superfluous assert (p16)
- moved comment and check in xfs_iflush_done() from p16 to p25
- rebase p25 (p16 conflicts)
Version 2
git://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git xfs-async-inode-reclaim-2
- describe ili_lock better (p2)
- clean up inode logging code some more (p2)
- move "early read completion" for xfs_buf_ioend() up into p3 from p4.
- fixed conflicts in p4 due to p3 changes.
- fixed conflicts in p5 due to p4 changes.
- s/_XBF_LOGRCVY/_XBF_LOG_RECOVERY/ (p5)
- renamed the buf log item iodone callback to xfs_buf_item_iodone and reused the xfs_buf_iodone() name for the catch-all buffer write iodone completion. (p6)
- history update for commit message (p7)
- subject update for p8
- rework loop in xfs_dquot_done() (p9)
- Fixed conflicts in p10 due to p6 changes
- got rid of entire comments around li_cb (p11)
- new patch to rework buffer io error callbacks
- new patch to unwind ->iop_error calls and remove ->iop_error
- new patch to lift xfs_clear_li_failed() out of xfs_ail_delete_one()
- rebased p12 on all the prior changes
- reworked LI_FAILED handling when pinning inodes to the cluster buffer (p12)
- fixed comment about holding buffer references in xfs_trans_log_inode() (p12)
- fixed indenting of xfs_iflush_abort() (p12)
- added comments explaining "skipped" indoe reclaim return value (p14)
- cleaned up error return stack in xfs_reclaim_inode() (p14)
- cleaned up skipped return in xfs_reclaim_inodes() (p14)
- fixed bug where skipped wasn't incremented if reclaim cursor was not zero. This could leave inodes between the start of the AG and the cursor unreclaimed (p15)
- reinstate the patch removing SYNC_WAIT from xfs_reclaim_inodes(). Exposed "skipped" bug in p15.
- cleaned up inode reclaim comments (p18)
- split p19 into two - one to change xfs_ifree_cluster(), one for the buffer pinning.
- xfs_ifree_mark_inode_stale() now takes the cluster buffer and we get the perag from that rather than having to do a lookup in xfs_ifree_cluster().
- moved extra IO reference for xfs_iflush_cluster() from AIL pushing to initial xfs_iflush_cluster rework (p22 -> p20)
- fixed static declaration on xfs_iflush() (p22)
- fixed incorrect EIO return from xfs_iflush_cluster()
- rebase p23 because it all rejects now.
- fix INODE_ITEM() usage in p23
- removed long lines from commit message in p24
- new patch to fix logging of XFS_ISTALE inodes which pushes dirty inodes through reclaim.
Dave Chinner (30):
xfs: Don't allow logging of XFS_ISTALE inodes
xfs: remove logged flag from inode log item
xfs: add an inode item lock
xfs: mark inode buffers in cache
xfs: mark dquot buffers in cache
xfs: mark log recovery buffers for completion
xfs: call xfs_buf_iodone directly
xfs: clean up whacky buffer log item list reinit
xfs: make inode IO completion buffer centric
xfs: use direct calls for dquot IO completion
xfs: clean up the buffer iodone callback functions
xfs: get rid of log item callbacks
xfs: handle buffer log item IO errors directly
xfs: unwind log item error flagging
xfs: move xfs_clear_li_failed out of xfs_ail_delete_one()
xfs: pin inode backing buffer to the inode log item
xfs: make inode reclaim almost non-blocking
xfs: remove IO submission from xfs_reclaim_inode()
xfs: allow multiple reclaimers per AG
xfs: don't block inode reclaim on the ILOCK
xfs: remove SYNC_TRYLOCK from inode reclaim
xfs: remove SYNC_WAIT from xfs_reclaim_inodes()
xfs: clean up inode reclaim comments
xfs: rework stale inodes in xfs_ifree_cluster
xfs: attach inodes to the cluster buffer when dirtied
xfs: xfs_iflush() is no longer necessary
xfs: rename xfs_iflush_int()
xfs: rework xfs_iflush_cluster() dirty inode iteration
xfs: factor xfs_iflush_done
xfs: remove xfs_inobp_check()
fs/xfs/libxfs/xfs_inode_buf.c | 27 +- fs/xfs/libxfs/xfs_inode_buf.h | 6 - fs/xfs/libxfs/xfs_trans_inode.c | 110 +++++-- fs/xfs/xfs_buf.c | 40 ++- fs/xfs/xfs_buf.h | 48 +-- fs/xfs/xfs_buf_item.c | 420 ++++++++++++------------ fs/xfs/xfs_buf_item.h | 8 +- fs/xfs/xfs_buf_item_recover.c | 5 +- fs/xfs/xfs_dquot.c | 29 +- fs/xfs/xfs_dquot.h | 1 + fs/xfs/xfs_dquot_item.c | 18 -- fs/xfs/xfs_dquot_item_recover.c | 2 +- fs/xfs/xfs_file.c | 9 +- fs/xfs/xfs_icache.c | 333 ++++++------------- fs/xfs/xfs_icache.h | 2 +- fs/xfs/xfs_inode.c | 554 ++++++++++++-------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 301 +++++++++-------- fs/xfs/xfs_inode_item.h | 24 +- fs/xfs/xfs_inode_item_recover.c | 2 +- fs/xfs/xfs_log_recover.c | 5 +- fs/xfs/xfs_mount.c | 15 +- fs/xfs/xfs_mount.h | 1 - fs/xfs/xfs_super.c | 3 - fs/xfs/xfs_trans.h | 5 - fs/xfs/xfs_trans_ail.c | 10 +- fs/xfs/xfs_trans_buf.c | 15 +- 27 files changed, 881 insertions(+), 1114 deletions(-)
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 1/30 xfs: Don't allow logging of XFS_ISTALE inodes
From: Dave Chinner <[email protected]>
In tracking down a problem in this patchset, I discovered we are reclaiming dirty stale inodes. This wasn't discovered until inodes were always attached to the cluster buffer and then the rcu callback that freed inodes was assert failing because the inode still had an active pointer to the cluster buffer after it had been reclaimed.
Debugging the issue indicated that this was a pre-existing issue resulting from the way the inodes are handled in xfs_inactive_ifree. When we free a cluster buffer from xfs_ifree_cluster, all the inodes in cache are marked XFS_ISTALE. Those that are clean have nothing else done to them and so eventually get cleaned up by background reclaim. i.e. it is assumed we'll never dirty/relog an inode marked XFS_ISTALE.
On journal commit dirty stale inodes as are handled by both buffer and inode log items to run though xfs_istale_done() and removed from the AIL (buffer log item commit) or the log item will simply unpin it because the buffer log item will clean it. What happens to any specific inode is entirely dependent on which log item wins the commit race, but the result is the same - stale inodes are clean, not attached to the cluster buffer, and not in the AIL. Hence inode reclaim can just free these inodes without further care.
However, if the stale inode is relogged, it gets dirtied again and relogged into the CIL. Most of the time this isn't an issue, because relogging simply changes the inode's location in the current checkpoint. Problems arise, however, when the CIL checkpoints between two transactions in the xfs_inactive_ifree() deferops processing. This results in the XFS_ISTALE inode being redirtied and inserted into the CIL without any of the other stale cluster buffer infrastructure being in place.
Hence on journal commit, it simply gets unpinned, so it remains dirty in memory. Everything in inode writeback avoids XFS_ISTALE inodes so it can't be written back, and it is not tracked in the AIL so there's not even a trigger to attempt to clean the inode. Hence the inode just sits dirty in memory until inode reclaim comes along, sees that it is XFS_ISTALE, and goes to reclaim it. This reclaiming of a dirty inode caused use after free, list corruptions and other nasty issues later in this patchset.
Hence this patch addresses a violation of the "never log XFS_ISTALE inodes" caused by the deferops processing rolling a transaction and relogging a stale inode in xfs_inactive_free. It also adds a bunch of asserts to catch this problem in debug kernels so that we don't reintroduce this problem in future.
Reproducer for this issue was generic/558 on a v4 filesystem.
--- fs/xfs/libxfs/xfs_trans_inode.c | 2 ++ fs/xfs/xfs_icache.c | 3 ++- fs/xfs/xfs_inode.c | 25 ++++++++++++++++++++++--- 3 files changed, 26 insertions(+), 4 deletions(-)
fs/xfs/libxfs/xfs_trans_inode.c +2 -0
@@ -36,6 +36,7 @@ xfs_trans_ijoin(ASSERT (iip->ili_lock_flags ==0 ); iip->ili_lock_flags = lock_flags; +ASSERT (!xfs_iflags_test (ip,XFS_ISTALE ));/* * Get a log_item_desc to point at the new item.
@@ -89,6 +90,7 @@ xfs_trans_log_inode(ASSERT (ip->i_itemp !=NULL );ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL )); +ASSERT (!xfs_iflags_test (ip,XFS_ISTALE ));/* * Don't bother with i_lock for the I_DIRTY_TIME check here, as races
fs/xfs/xfs_icache.c +2 -1
@@ -1141,7 +1141,7 @@ xfs_reclaim_inode( goto out_ifunlock;xfs_iunpin_wait (ip); } -if (xfs_iflags_test (ip,XFS_ISTALE ) ||xfs_inode_clean (ip)) { +if (xfs_inode_clean (ip)) {xfs_ifunlock (ip); goto reclaim; }
@@ -1228,6 +1228,7 @@ xfs_reclaim_inode(xfs_ilock (ip,XFS_ILOCK_EXCL );xfs_qm_dqdetach (ip);xfs_iunlock (ip,XFS_ILOCK_EXCL ); +ASSERT (xfs_inode_clean (ip));__xfs_inode_free (ip);return error;
fs/xfs/xfs_inode.c +22 -3
@@ -1740,10 +1740,31 @@ xfs_inactive_ifree(return error; } +/* +* We do not hold the inode locked across the entire rolling transaction +* here. We only need to hold it for the first transaction that +* xfs_ifree() builds, which may mark the inode XFS_ISTALE if the +* underlying cluster buffer is freed. Relogging an XFS_ISTALE inode +* here breaks the relationship between cluster buffer invalidation and +* stale inode invalidation on cluster buffer item journal commit +* completion, and can result in leaving dirty stale inodes hanging +* around in memory. +* +* We have no need for serialising this inode operation against other +* operations - we freed the inode and hence reallocation is required +* and that will serialise on reallocating the space the deferops need +* to free. Hence we can unlock the inode on the first commit of +* the transaction rather than roll it right through the deferops. This +* avoids relogging the XFS_ISTALE inode. +* +* We check that xfs_ifree() hasn't grown an internal transaction roll +* by asserting that the inode is still locked when it returns. +*/ xfs_ilock (ip,XFS_ILOCK_EXCL ); -xfs_trans_ijoin (tp, ip,0 ); +xfs_trans_ijoin (tp, ip,XFS_ILOCK_EXCL ); error =xfs_ifree (tp, ip); +ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));if (error) {/* * If we fail to free the inode, shut down. The cancel
@@ -1756,7 +1777,6 @@ xfs_inactive_ifree(xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); }xfs_trans_cancel (tp); -xfs_iunlock (ip,XFS_ILOCK_EXCL );return error; }
@@ -1774,7 +1794,6 @@ xfs_inactive_ifree(xfs_notice (mp,"%s: xfs_trans_commit returned error %d" , __func__, error); -xfs_iunlock (ip,XFS_ILOCK_EXCL );return 0 ; }
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 2/30 xfs: remove logged flag from inode log item
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
This was used to track if the item had logged fields being flushed to disk. We log everything in the inode these days, so this logic is no longer needed. Remove it.
--- fs/xfs/xfs_inode.c | 13 ++++--------- fs/xfs/xfs_inode_item.c | 35 ++++++++++------------------------- fs/xfs/xfs_inode_item.h | 1 - 3 files changed, 14 insertions(+), 35 deletions(-)
fs/xfs/xfs_inode.c +4 -9
@@ -2677,7 +2677,6 @@ xfs_ifree_cluster(list_for_each_entry (lip, &bp->b_li_list, li_bio_list) {if (lip->li_type ==XFS_LI_INODE ) { iip = (struct xfs_inode_log_item *)lip; -ASSERT (iip->ili_logged ==1 ); lip->li_cb = xfs_istale_done; xfs_trans_ail_copy_lsn(mp->m_ail, &iip->ili_flush_lsn,
@@ -2706,7 +2705,6 @@ xfs_ifree_cluster( iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; - iip->ili_logged =1 ;xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);
@@ -3838,19 +3836,16 @@ xfs_iflush_int(* * We can play with the ili_fields bits here, because the inode lock * must be held exclusively in order to set bits there and the flush -* lock protects the ili_last_fields bits. Set ili_logged so the flush -* done routine can tell whether or not to look in the AIL. Also, store -* the current LSN of the inode so that we can tell whether the item has -* moved in the AIL from xfs_iflush_done(). In order to read the lsn we -* need the AIL lock, because it is a 64 bit value that cannot be read -* atomically. +* lock protects the ili_last_fields bits. Store the current LSN of the +* inode so that we can tell whether the item has moved in the AIL from +* xfs_iflush_done(). In order to read the lsn we need the AIL lock, +* because it is a 64 bit value that cannot be read atomically. */ error =0 ; flush_out: iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; - iip->ili_logged =1 ;xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);
fs/xfs/xfs_inode_item.c +10 -25
@@ -528,8 +528,6 @@ xfs_inode_item_push( }ASSERT (iip->ili_fields !=0 ||XFS_FORCED_SHUTDOWN (ip->i_mount)); -ASSERT (iip->ili_logged ==0 ||XFS_FORCED_SHUTDOWN (ip->i_mount)); -spin_unlock (&lip->li_ailp->ail_lock); error =xfs_iflush (ip, &bp);
@@ -690,30 +688,24 @@ xfs_iflush_done(continue ;list_move_tail (&blip->li_bio_list, &tmp); -/* -* while we have the item, do the unlocked check for needing -* the AIL lock. -*/ + +/* Do an unlocked check for needing the AIL lock. */ iip =INODE_ITEM (blip); -if ((iip->ili_logged && blip->li_lsn == iip->ili_flush_lsn) || +if (blip->li_lsn == iip->ili_flush_lsn ||test_bit (XFS_LI_FAILED , &blip->li_flags)) need_ail++; }/* make sure we capture the state of the initial inode. */ iip =INODE_ITEM (lip); -if ((iip->ili_logged && lip->li_lsn == iip->ili_flush_lsn) || +if (lip->li_lsn == iip->ili_flush_lsn ||test_bit (XFS_LI_FAILED , &lip->li_flags)) need_ail++;/* -* We only want to pull the item from the AIL if it is -* actually there and its location in the log has not -* changed since we started the flush. Thus, we only bother -* if the ili_logged flag is set and the inode's lsn has not -* changed. First we check the lsn outside -* the lock since it's cheaper, and then we recheck while -* holding the lock before removing the inode from the AIL. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ if (need_ail) {xfs_lsn_t tail_lsn =0 ;
@@ -721,8 +713,7 @@ xfs_iflush_done(/* this is an opencoded batch version of xfs_trans_ail_delete */ spin_lock (&ailp->ail_lock);list_for_each_entry (blip, &tmp, li_bio_list) { -if (INODE_ITEM (blip)->ili_logged && - blip->li_lsn ==INODE_ITEM (blip)->ili_flush_lsn) { +if (blip->li_lsn ==INODE_ITEM (blip)->ili_flush_lsn) {/* * xfs_ail_update_finish() only cares about the * lsn of the first tail item removed, any
@@ -740,14 +731,13 @@ xfs_iflush_done( }/* -* clean up and unlock the flush lock now we are done. We can clear the +* Clean up and unlock the flush lock now we are done. We can clear the * ili_last_fields bits now that we know that the data corresponding to * them is safely on disk. */ list_for_each_entry_safe (blip, n, &tmp, li_bio_list) {list_del_init (&blip->li_bio_list); iip =INODE_ITEM (blip); - iip->ili_logged =0 ; iip->ili_last_fields =0 ;xfs_ifunlock (iip->ili_inode); }
@@ -768,16 +758,11 @@ xfs_iflush_abort(if (iip) {xfs_trans_ail_delete (&iip->ili_item,0 ); - iip->ili_logged =0 ; -/* -* Clear the ili_last_fields bits now that we know that the -* data corresponding to them is safely on disk. -*/ - iip->ili_last_fields =0 ;/* * Clear the inode logging fields so no more flushes are * attempted. */ + iip->ili_last_fields =0 ; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; }
fs/xfs/xfs_inode_item.h +0 -1
@@ -19,7 +19,6 @@ struct xfs_inode_log_item {xfs_lsn_t ili_flush_lsn;/* lsn at last flush */ xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ unsigned short ili_lock_flags;/* lock flags */ -unsigned short ili_logged;/* flushed logged data */ unsigned int ili_last_fields;/* fields when flushed */ unsigned int ili_fields;/* fields to be logged */ unsigned int ili_fsync_fields;/* logged since last fsync */
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 3/30 xfs: add an inode item lock
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
The inode log item is kind of special in that it can be aggregating new changes in memory at the same time time existing changes are being written back to disk. This means there are fields in the log item that are accessed concurrently from contexts that don't share any locking at all.
e.g. updating ili_last_fields occurs at flush time under the ILOCK_EXCL and flush lock at flush time, under the flush lock at IO completion time, and is read under the ILOCK_EXCL when the inode is logged. Hence there is no actual serialisation between reading the field during logging of the inode in transactions vs clearing the field in IO completion.
We currently get away with this by the fact that we are only clearing fields in IO completion, and nothing bad happens if we accidentally log more of the inode than we actually modify. Worst case is we consume a tiny bit more memory and log bandwidth.
However, if we want to do more complex state manipulations on the log item that requires updates at all three of these potential locations, we need to have some mechanism of serialising those operations. To do this, introduce a spinlock into the log item to serialise internal state.
This could be done via the xfs_inode i_flags_lock, but this then leads to potential lock inversion issues where inode flag updates need to occur inside locks that best nest inside the inode log item locks (e.g. marking inodes stale during inode cluster freeing). Using a separate spinlock avoids these sorts of problems and simplifies future code.
This does not touch the use of ili_fields in the item formatting code - that is entirely protected by the ILOCK_EXCL at this point in time, so it remains untouched.
--- fs/xfs/libxfs/xfs_trans_inode.c | 52 ++++++++++++++++----------------- fs/xfs/xfs_file.c | 9 ++++-- fs/xfs/xfs_inode.c | 20 ++++++++----- fs/xfs/xfs_inode_item.c | 7 +++++ fs/xfs/xfs_inode_item.h | 18 ++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-)
fs/xfs/libxfs/xfs_trans_inode.c +26 -26
@@ -82,16 +82,20 @@ xfs_trans_ichgtime(*/ void xfs_trans_log_inode ( -xfs_trans_t *tp, -xfs_inode_t *ip, -uint flags) +struct xfs_trans *tp, +struct xfs_inode *ip, +uint flags) { -struct inode *inode =VFS_I (ip); +struct xfs_inode_log_item *iip = ip->i_itemp; +struct inode *inode =VFS_I (ip); +uint iversion_flags =0 ; -ASSERT (ip->i_itemp !=NULL ); +ASSERT (iip);ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));ASSERT (!xfs_iflags_test (ip,XFS_ISTALE )); + tp->t_flags |=XFS_TRANS_DIRTY ; +/* * Don't bother with i_lock for the I_DIRTY_TIME check here, as races * don't matter - we either will need an extra transaction in 24 hours
@@ -104,15 +108,6 @@ xfs_trans_log_inode(spin_unlock (&inode->i_lock); } -/* -* Record the specific change for fdatasync optimisation. This -* allows fdatasync to skip log forces for inodes that are only -* timestamp dirty. We do this before the change count so that -* the core being logged in this case does not impact on fdatasync -* behaviour. -*/ - ip->i_itemp->ili_fsync_fields |= flags; -/* * First time we log the inode in a transaction, bump the inode change * counter if it is configured for this to occur. While we have the
@@ -122,23 +117,28 @@ xfs_trans_log_inode(* set however, then go ahead and bump the i_version counter * unconditionally. */ -if (!test_and_set_bit (XFS_LI_DIRTY , &ip->i_itemp->ili_item.li_flags) && -IS_I_VERSION (VFS_I (ip))) { -if (inode_maybe_inc_iversion (VFS_I (ip), flags &XFS_ILOG_CORE )) - flags |=XFS_ILOG_CORE ; +if (!test_and_set_bit (XFS_LI_DIRTY , &iip->ili_item.li_flags)) { +if (IS_I_VERSION (inode) && +inode_maybe_inc_iversion (inode, flags &XFS_ILOG_CORE )) + iversion_flags =XFS_ILOG_CORE ; } - tp->t_flags |=XFS_TRANS_DIRTY ; +/* +* Record the specific change for fdatasync optimisation. This allows +* fdatasync to skip log forces for inodes that are only timestamp +* dirty. +*/ +spin_lock (&iip->ili_lock); + iip->ili_fsync_fields |= flags;/* -* Always OR in the bits from the ili_last_fields field. -* This is to coordinate with the xfs_iflush() and xfs_iflush_done() -* routines in the eventual clearing of the ili_fields bits. -* See the big comment in xfs_iflush() for an explanation of -* this coordination mechanism. +* Always OR in the bits from the ili_last_fields field. This is to +* coordinate with the xfs_iflush() and xfs_iflush_done() routines in +* the eventual clearing of the ili_fields bits. See the big comment in +* xfs_iflush() for an explanation of this coordination mechanism. */ - flags |= ip->i_itemp->ili_last_fields; - ip->i_itemp->ili_fields |= flags; + iip->ili_fields |= (flags | iip->ili_last_fields | iversion_flags); +spin_unlock (&iip->ili_lock); }int
fs/xfs/xfs_file.c +6 -3
@@ -94,6 +94,7 @@ xfs_file_fsync( {struct inode *inode = file->f_mapping->host;struct xfs_inode *ip =XFS_I (inode); +struct xfs_inode_log_item *iip = ip->i_itemp;struct xfs_mount *mp = ip->i_mount;int error =0 ;int log_flushed =0 ;
@@ -137,13 +138,15 @@ xfs_file_fsync(xfs_ilock (ip,XFS_ILOCK_SHARED );if (xfs_ipincount (ip)) {if (!datasync || - (ip->i_itemp->ili_fsync_fields & ~XFS_ILOG_TIMESTAMP )) - lsn = ip->i_itemp->ili_last_lsn; + (iip->ili_fsync_fields & ~XFS_ILOG_TIMESTAMP )) + lsn = iip->ili_last_lsn; }if (lsn) { error =xfs_log_force_lsn (mp, lsn,XFS_LOG_SYNC , &log_flushed); - ip->i_itemp->ili_fsync_fields =0 ; +spin_lock (&iip->ili_lock); + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); }xfs_iunlock (ip,XFS_ILOCK_SHARED );
fs/xfs/xfs_inode.c +12 -8
@@ -2702,9 +2702,11 @@ xfs_ifree_cluster(continue ; iip = ip->i_itemp; +spin_lock (&iip->ili_lock); iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock);xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);
@@ -2740,6 +2742,7 @@ xfs_ifree( {int error;struct xfs_icluster xic = {0 }; +struct xfs_inode_log_item *iip = ip->i_itemp;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));ASSERT (VFS_I (ip)->i_nlink ==0 );
@@ -2777,7 +2780,9 @@ xfs_ifree( ip->i_df.if_format =XFS_DINODE_FMT_EXTENTS ;/* Don't attempt to replay owner changes for a deleted inode */ - ip->i_itemp->ili_fields &= ~(XFS_ILOG_AOWNER |XFS_ILOG_DOWNER ); +spin_lock (&iip->ili_lock); + iip->ili_fields &= ~(XFS_ILOG_AOWNER |XFS_ILOG_DOWNER ); +spin_unlock (&iip->ili_lock);/* * Bump the generation count so no one will be confused
@@ -3833,20 +3838,19 @@ xfs_iflush_int(* know that the information those bits represent is permanently on * disk. As long as the flush completes before the inode is logged * again, then both ili_fields and ili_last_fields will be cleared. -* -* We can play with the ili_fields bits here, because the inode lock -* must be held exclusively in order to set bits there and the flush -* lock protects the ili_last_fields bits. Store the current LSN of the -* inode so that we can tell whether the item has moved in the AIL from -* xfs_iflush_done(). In order to read the lsn we need the AIL lock, -* because it is a 64 bit value that cannot be read atomically. */ error =0 ; flush_out: +spin_lock (&iip->ili_lock); iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +/* +* Store the current LSN of the inode so that we can tell whether the +* item has moved in the AIL from xfs_iflush_done(). +*/ xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);
fs/xfs/xfs_inode_item.c +7 -0
@@ -637,6 +637,7 @@ xfs_inode_item_init( iip = ip->i_itemp =kmem_zone_zalloc (xfs_ili_zone,0 ); iip->ili_inode = ip; +spin_lock_init (&iip->ili_lock);xfs_log_item_init (mp, &iip->ili_item,XFS_LI_INODE , &xfs_inode_item_ops); }
@@ -738,7 +739,11 @@ xfs_iflush_done(list_for_each_entry_safe (blip, n, &tmp, li_bio_list) {list_del_init (&blip->li_bio_list); iip =INODE_ITEM (blip); + +spin_lock (&iip->ili_lock); iip->ili_last_fields =0 ; +spin_unlock (&iip->ili_lock); +xfs_ifunlock (iip->ili_inode); }list_del (&tmp);
@@ -762,9 +767,11 @@ xfs_iflush_abort(* Clear the inode logging fields so no more flushes are * attempted. */ +spin_lock (&iip->ili_lock); iip->ili_last_fields =0 ; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); }/* * Release the inode's flush lock since we're done with it.
fs/xfs/xfs_inode_item.h +15 -3
@@ -16,12 +16,24 @@ struct xfs_mount;struct xfs_inode_log_item {struct xfs_log_item ili_item;/* common portion */ struct xfs_inode *ili_inode;/* inode ptr */ -xfs_lsn_t ili_flush_lsn;/* lsn at last flush */ -xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ -unsigned short ili_lock_flags;/* lock flags */ +unsigned short ili_lock_flags;/* inode lock flags */ +/* +* The ili_lock protects the interactions between the dirty state and +* the flush state of the inode log item. This allows us to do atomic +* modifications of multiple state fields without having to hold a +* specific inode lock to serialise them. +* +* We need atomic changes between inode dirtying, inode flushing and +* inode completion, but these all hold different combinations of +* ILOCK and iflock and hence we need some other method of serialising +* updates to the flush state. +*/ +spinlock_t ili_lock;/* flush state lock */ unsigned int ili_last_fields;/* fields when flushed */ unsigned int ili_fields;/* fields to be logged */ unsigned int ili_fsync_fields;/* logged since last fsync */ +xfs_lsn_t ili_flush_lsn;/* lsn at last flush */ +xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ };static inline int xfs_inode_clean (xfs_inode_t *ip)
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 03/30] xfs: add an inode item lock
Quoted 33 lines · Dave Chinner
From: Dave Chinner <[email protected]>
The inode log item is kind of special in that it can be aggregating new changes in memory at the same time time existing changes are being written back to disk. This means there are fields in the log item that are accessed concurrently from contexts that don't share any locking at all.
e.g. updating
ili_last_fieldsoccurs at flush time under theILOCK_EXCLand flush lock at flush time, under the flush lock at IO completion time, and is read under theILOCK_EXCLwhen the inode is logged. Hence there is no actual serialisation between reading the field during logging of the inode in transactions vs clearing the field in IO completion.We currently get away with this by the fact that we are only clearing fields in IO completion, and nothing bad happens if we accidentally log more of the inode than we actually modify. Worst case is we consume a tiny bit more memory and log bandwidth.
However, if we want to do more complex state manipulations on the log item that requires updates at all three of these potential locations, we need to have some mechanism of serialising those operations. To do this, introduce a spinlock into the log item to serialise internal state.
This could be done via the
xfs_inodei_flags_lock, but this then leads to potential lock inversion issues where inode flag updates need to occur inside locks that best nest inside the inode log item locks (e.g. marking inodes stale during inode cluster freeing). Using a separate spinlock avoids these sorts of problems and simplifies future code.
Quoted 6 lines
This does not touch the use of
ili_fieldsin the item formatting code - that is entirely protected by theILOCK_EXCLat this point in time, so it remains untouched.Signed-off-by Dave Chinner <[email protected]>---
I think I have a better understanding of this now that I've reached the end of the series and poked around a bit more. AFAICT logging and flushing the inode are protected via the ILOCK, which means this protects logging the inode and completing (or aborting) an inode flush. Technically, it seems like we could get away with using the buffer lock with some tweaks in those cases since logging the inode now acquires it, but that's only the first time it's logged and I don't think it's worth reworking that just to avoid the need for a new spinlock...
Quoted 258 lines
fs/xfs/libxfs/xfs_trans_inode.c | 52 ++++++++++++++++----------------- fs/xfs/xfs_file.c | 9 ++++-- fs/xfs/xfs_inode.c | 20 ++++++++----- fs/xfs/xfs_inode_item.c | 7 +++++ fs/xfs/xfs_inode_item.h | 18 ++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-)fs/xfs/libxfs/xfs_trans_inode.c +26
-26@@ -82,16 +82,20 @@ xfs_trans_ichgtime(*/ void xfs_trans_log_inode ( -xfs_trans_t *tp, -xfs_inode_t *ip, -uint flags) +struct xfs_trans *tp, +struct xfs_inode *ip, +uint flags) { -struct inode *inode =VFS_I (ip); +struct xfs_inode_log_item *iip = ip->i_itemp; +struct inode *inode =VFS_I (ip); +uint iversion_flags =0 ; -ASSERT (ip->i_itemp !=NULL ); +ASSERT (iip);ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));ASSERT (!xfs_iflags_test (ip,XFS_ISTALE )); + tp->t_flags |=XFS_TRANS_DIRTY ; +/* * Don't bother with i_lock for the I_DIRTY_TIME check here, as races * don't matter - we either will need an extra transaction in 24 hours @@ -104,15 +108,6 @@ xfs_trans_log_inode(spin_unlock (&inode->i_lock); } -/* -* Record the specific change for fdatasync optimisation. This -* allows fdatasync to skip log forces for inodes that are only -* timestamp dirty. We do this before the change count so that -* the core being logged in this case does not impact on fdatasync -* behaviour. -*/ - ip->i_itemp->ili_fsync_fields |= flags; -/* * First time we log the inode in a transaction, bump the inode change * counter if it is configured for this to occur. While we have the @@ -122,23 +117,28 @@ xfs_trans_log_inode(* set however, then go ahead and bump the i_version counter * unconditionally. */ -if (!test_and_set_bit (XFS_LI_DIRTY , &ip->i_itemp->ili_item.li_flags) && -IS_I_VERSION (VFS_I (ip))) { -if (inode_maybe_inc_iversion (VFS_I (ip), flags &XFS_ILOG_CORE )) - flags |=XFS_ILOG_CORE ; +if (!test_and_set_bit (XFS_LI_DIRTY , &iip->ili_item.li_flags)) { +if (IS_I_VERSION (inode) && +inode_maybe_inc_iversion (inode, flags &XFS_ILOG_CORE )) + iversion_flags =XFS_ILOG_CORE ; } - tp->t_flags |=XFS_TRANS_DIRTY ; +/* +* Record the specific change for fdatasync optimisation. This allows +* fdatasync to skip log forces for inodes that are only timestamp +* dirty. +*/ +spin_lock (&iip->ili_lock); + iip->ili_fsync_fields |= flags;/* -* Always OR in the bits from the ili_last_fields field. -* This is to coordinate with the xfs_iflush() and xfs_iflush_done() -* routines in the eventual clearing of the ili_fields bits. -* See the big comment in xfs_iflush() for an explanation of -* this coordination mechanism. +* Always OR in the bits from the ili_last_fields field. This is to +* coordinate with the xfs_iflush() and xfs_iflush_done() routines in +* the eventual clearing of the ili_fields bits. See the big comment in +* xfs_iflush() for an explanation of this coordination mechanism. */ - flags |= ip->i_itemp->ili_last_fields; - ip->i_itemp->ili_fields |= flags; + iip->ili_fields |= (flags | iip->ili_last_fields | iversion_flags); +spin_unlock (&iip->ili_lock); }int fs/xfs/xfs_file.c +6
-3@@ -94,6 +94,7 @@ xfs_file_fsync( {struct inode *inode = file->f_mapping->host;struct xfs_inode *ip =XFS_I (inode); +struct xfs_inode_log_item *iip = ip->i_itemp;struct xfs_mount *mp = ip->i_mount;int error =0 ;int log_flushed =0 ;@@ -137,13 +138,15 @@ xfs_file_fsync(xfs_ilock (ip,XFS_ILOCK_SHARED );if (xfs_ipincount (ip)) {if (!datasync || - (ip->i_itemp->ili_fsync_fields & ~XFS_ILOG_TIMESTAMP )) - lsn = ip->i_itemp->ili_last_lsn; + (iip->ili_fsync_fields & ~XFS_ILOG_TIMESTAMP )) + lsn = iip->ili_last_lsn; }if (lsn) { error =xfs_log_force_lsn (mp, lsn,XFS_LOG_SYNC , &log_flushed); - ip->i_itemp->ili_fsync_fields =0 ; +spin_lock (&iip->ili_lock); + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); }xfs_iunlock (ip,XFS_ILOCK_SHARED );fs/xfs/xfs_inode.c +12
-8@@ -2702,9 +2702,11 @@ xfs_ifree_cluster(continue ; iip = ip->i_itemp; +spin_lock (&iip->ili_lock); iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock);xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);@@ -2740,6 +2742,7 @@ xfs_ifree( {int error;struct xfs_icluster xic = {0 }; +struct xfs_inode_log_item *iip = ip->i_itemp;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL ));ASSERT (VFS_I (ip)->i_nlink ==0 );@@ -2777,7 +2780,9 @@ xfs_ifree( ip->i_df.if_format =XFS_DINODE_FMT_EXTENTS ;/* Don't attempt to replay owner changes for a deleted inode */ - ip->i_itemp->ili_fields &= ~(XFS_ILOG_AOWNER |XFS_ILOG_DOWNER ); +spin_lock (&iip->ili_lock); + iip->ili_fields &= ~(XFS_ILOG_AOWNER |XFS_ILOG_DOWNER ); +spin_unlock (&iip->ili_lock);/* * Bump the generation count so no one will be confused @@ -3833,20 +3838,19 @@ xfs_iflush_int(* know that the information those bits represent is permanently on * disk. As long as the flush completes before the inode is logged * again, then both ili_fields and ili_last_fields will be cleared. -* -* We can play with the ili_fields bits here, because the inode lock -* must be held exclusively in order to set bits there and the flush -* lock protects the ili_last_fields bits. Store the current LSN of the -* inode so that we can tell whether the item has moved in the AIL from -* xfs_iflush_done(). In order to read the lsn we need the AIL lock, -* because it is a 64 bit value that cannot be read atomically. */ error =0 ; flush_out: +spin_lock (&iip->ili_lock); iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +/* +* Store the current LSN of the inode so that we can tell whether the +* item has moved in the AIL from xfs_iflush_done(). +*/ xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);fs/xfs/xfs_inode_item.c +7
-0@@ -637,6 +637,7 @@ xfs_inode_item_init( iip = ip->i_itemp =kmem_zone_zalloc (xfs_ili_zone,0 ); iip->ili_inode = ip; +spin_lock_init (&iip->ili_lock);xfs_log_item_init (mp, &iip->ili_item,XFS_LI_INODE , &xfs_inode_item_ops); }@@ -738,7 +739,11 @@ xfs_iflush_done(list_for_each_entry_safe (blip, n, &tmp, li_bio_list) {list_del_init (&blip->li_bio_list); iip =INODE_ITEM (blip); + +spin_lock (&iip->ili_lock); iip->ili_last_fields =0 ; +spin_unlock (&iip->ili_lock); +xfs_ifunlock (iip->ili_inode); }list_del (&tmp);@@ -762,9 +767,11 @@ xfs_iflush_abort(* Clear the inode logging fields so no more flushes are * attempted. */ +spin_lock (&iip->ili_lock); iip->ili_last_fields =0 ; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); }/* * Release the inode's flush lock since we're done with it. fs/xfs/xfs_inode_item.h +15
-3@@ -16,12 +16,24 @@ struct xfs_mount;struct xfs_inode_log_item {struct xfs_log_item ili_item;/* common portion */ struct xfs_inode *ili_inode;/* inode ptr */ -xfs_lsn_t ili_flush_lsn;/* lsn at last flush */ -xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ -unsigned short ili_lock_flags;/* lock flags */ +unsigned short ili_lock_flags;/* inode lock flags */ +/* +* The ili_lock protects the interactions between the dirty state and +* the flush state of the inode log item. This allows us to do atomic +* modifications of multiple state fields without having to hold a +* specific inode lock to serialise them. +* +* We need atomic changes between inode dirtying, inode flushing and +* inode completion, but these all hold different combinations of +* ILOCK and iflock and hence we need some other method of serialising +* updates to the flush state. +*/ +spinlock_t ili_lock;/* flush state lock */ unsigned int ili_last_fields;/* fields when flushed */ unsigned int ili_fields;/* fields to be logged */ unsigned int ili_fsync_fields;/* logged since last fsync */ +xfs_lsn_t ili_flush_lsn;/* lsn at last flush */ +xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ };static inline int xfs_inode_clean (xfs_inode_t *ip)Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 4/30 xfs: mark inode buffers in cache
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Inode buffers always have write IO callbacks, so by marking them directly we can avoid needing to attach ->b_iodone functions to them. This avoids an indirect call, and makes future modifications much simpler.
While this is largely a refactor of existing functionality, we broaden the scope of the flag to beyond where inodes are explicitly attached because future changes need to know what type of log items are attached to the buffer. Adding this buffer flag may invoke the inode iodone callback in cases where it wouldn't have been previously, but this is not a functional change because the callback is identical to the normal buffer write iodone callback when inodes are not attached.
--- fs/xfs/xfs_buf.c | 21 ++++++++++++++++----- fs/xfs/xfs_buf.h | 38 +++++++++++++++++++++++++------------- fs/xfs/xfs_buf_item.c | 42 +++++++++++++++++++++++++++++++----------- fs/xfs/xfs_buf_item.h | 1 + fs/xfs/xfs_inode.c | 2 +- fs/xfs/xfs_trans_buf.c | 3 +++ 6 files changed, 77 insertions(+), 30 deletions(-)
fs/xfs/xfs_buf.c +16 -5
@@ -14,6 +14,8 @@#include "xfs_mount.h" #include "xfs_trace.h" #include "xfs_log.h" +#include "xfs_trans.h" +#include "xfs_buf_item.h" #include "xfs_errortag.h" #include "xfs_error.h"
@@ -1202,12 +1204,21 @@ xfs_buf_ioend( bp->b_flags |=XBF_DONE ; } -if (bp->b_iodone) +if (read) + goto out_finish; + +if (bp->b_flags & _XBF_INODES) { +xfs_buf_inode_iodone (bp); +return ; + } + +if (bp->b_iodone) { (*(bp->b_iodone))(bp); -else if (bp->b_flags &XBF_ASYNC ) -xfs_buf_relse (bp); -else -complete (&bp->b_iowait); +return ; + } + +out_finish: +xfs_buf_ioend_finish (bp); } static void
fs/xfs/xfs_buf.h +25 -13
@@ -30,15 +30,18 @@#define XBF_STALE (1 << 6)/* buffer has been staled, do not find it */ #define XBF_WRITE_FAIL (1 << 7)/* async writes have failed on this buffer */ -/* flags used only as arguments to access routines */ -#define XBF_TRYLOCK (1 << 16)/* lock requested, but do not wait */ -#define XBF_UNMAPPED (1 << 17)/* do not map the buffer */ +/* buffer type flags for write callbacks */ +#define _XBF_INODES (1 << 16)/* inode buffer */ /* flags used only internally */ #define _XBF_PAGES (1 << 20)/* backed by refcounted pages */ #define _XBF_KMEM (1 << 21)/* backed by heap memory */ #define _XBF_DELWRI_Q (1 << 22)/* buffer on a delwri queue */ +/* flags used only as arguments to access routines */ +#define XBF_TRYLOCK (1 << 30)/* lock requested, but do not wait */ +#define XBF_UNMAPPED (1 << 31)/* do not map the buffer */ +typedef unsigned int xfs_buf_flags_t ;#define XFS_BUF_FLAGS \
@@ -50,12 +53,13 @@ typedef unsigned int xfs_buf_flags_t; { XBF_DONE, "DONE" }, \ { XBF_STALE, "STALE" }, \ { XBF_WRITE_FAIL, "WRITE_FAIL" }, \ - { XBF_TRYLOCK, "TRYLOCK" }, /* should never be set */\ - { XBF_UNMAPPED, "UNMAPPED" }, /* ditto */\ + { _XBF_INODES, "INODES" }, \ { _XBF_PAGES, "PAGES" }, \ { _XBF_KMEM, "KMEM" }, \ - { _XBF_DELWRI_Q, "DELWRI_Q" } - + { _XBF_DELWRI_Q, "DELWRI_Q" }, \ +/* The following interface flags should never be set */ \ + { XBF_TRYLOCK, "TRYLOCK" }, \ + { XBF_UNMAPPED, "UNMAPPED" }/* * Internal state flags.
@@ -257,9 +261,23 @@ extern void xfs_buf_unlock(xfs_buf_t *);#define xfs_buf_islocked (bp) \ ((bp)->b_sema.count <= 0) +static inline void xfs_buf_relse (xfs_buf_t *bp) +{ +xfs_buf_unlock (bp); +xfs_buf_rele (bp); +} +/* Buffer Read and Write Routines */ extern int xfs_bwrite (struct xfs_buf *bp);extern void xfs_buf_ioend (struct xfs_buf *bp); +static inline void xfs_buf_ioend_finish (struct xfs_buf *bp) +{ +if (bp->b_flags &XBF_ASYNC ) +xfs_buf_relse (bp); +else +complete (&bp->b_iowait); +} +extern void __xfs_buf_ioerror (struct xfs_buf *bp,int error,xfs_failaddr_t failaddr);#define xfs_buf_ioerror (bp, err) __xfs_buf_ioerror((bp), (err), __this_address)
@@ -324,12 +342,6 @@ static inline int xfs_buf_ispinned(struct xfs_buf *bp)return atomic_read (&bp->b_pin_count); } -static inline void xfs_buf_relse (xfs_buf_t *bp) -{ -xfs_buf_unlock (bp); -xfs_buf_rele (bp); -} -static inline int xfs_buf_verify_cksum (struct xfs_buf *bp,unsigned long cksum_offset) {
fs/xfs/xfs_buf_item.c +31 -11
60 lines not on this page · raw message
fs/xfs/xfs_buf_item.h +1 -0
@@ -59,6 +59,7 @@ void xfs_buf_attach_iodone(struct xfs_buf *, struct xfs_log_item *);void xfs_buf_iodone_callbacks (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *,struct xfs_log_item *); +void xfs_buf_inode_iodone (struct xfs_buf *);bool xfs_buf_log_check_iovec (struct xfs_log_iovec *iovec);extern kmem_zone_t *xfs_buf_item_zone;
fs/xfs/xfs_inode.c +1 -1
@@ -3860,13 +3860,13 @@ xfs_iflush_int(* completion on the buffer to remove the inode from the AIL and release * the flush lock. */ + bp->b_flags |= _XBF_INODES;xfs_buf_attach_iodone (bp, xfs_iflush_done, &iip->ili_item);/* generate the checksum. */ xfs_dinode_calc_crc (mp, dip);ASSERT (!list_empty (&bp->b_li_list)); -ASSERT (bp->b_iodone !=NULL );return error; }
fs/xfs/xfs_trans_buf.c +3 -0
@@ -626,6 +626,7 @@ xfs_trans_inode_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_INODE_BUF ; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }
@@ -651,6 +652,7 @@ xfs_trans_stale_inode_buf( bip->bli_flags |=XFS_BLI_STALE_INODE ; bip->bli_item.li_cb = xfs_buf_iodone; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }
@@ -675,6 +677,7 @@ xfs_trans_inode_alloc_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_INODE_ALLOC_BUF ; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 04/30] xfs: mark inode buffers in cache
Quoted 7 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Inode buffers always have write IO callbacks, so by marking them directly we can avoid needing to attach ->b_iodone functions to them. This avoids an indirect call, and makes future modifications much simpler.
Quoted 9 lines
While this is largely a refactor of existing functionality, we broaden the scope of the flag to beyond where inodes are explicitly attached because future changes need to know what type of log items are attached to the buffer. Adding this buffer flag may invoke the inode iodone callback in cases where it wouldn't have been previously, but this is not a functional change because the callback is identical to the normal buffer write iodone callback when inodes are not attached.
Better.. I'm still not clear on why the association with where the bli flags are set, but perhaps that's just convenience and/or it will make more sense later. Code seems fine, regardless:
Quoted 259 lines
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf.c | 21 ++++++++++++++++----- fs/xfs/xfs_buf.h | 38 +++++++++++++++++++++++++------------- fs/xfs/xfs_buf_item.c | 42 +++++++++++++++++++++++++++++++----------- fs/xfs/xfs_buf_item.h | 1 + fs/xfs/xfs_inode.c | 2 +- fs/xfs/xfs_trans_buf.c | 3 +++ 6 files changed, 77 insertions(+), 30 deletions(-)fs/xfs/xfs_buf.c +16
-5@@ -14,6 +14,8 @@#include "xfs_mount.h" #include "xfs_trace.h" #include "xfs_log.h" +#include "xfs_trans.h" +#include "xfs_buf_item.h" #include "xfs_errortag.h" #include "xfs_error.h" @@ -1202,12 +1204,21 @@ xfs_buf_ioend( bp->b_flags |=XBF_DONE ; } -if (bp->b_iodone) +if (read) + goto out_finish; + +if (bp->b_flags & _XBF_INODES) { +xfs_buf_inode_iodone (bp); +return ; + } + +if (bp->b_iodone) { (*(bp->b_iodone))(bp); -else if (bp->b_flags &XBF_ASYNC ) -xfs_buf_relse (bp); -else -complete (&bp->b_iowait); +return ; + } + +out_finish: +xfs_buf_ioend_finish (bp); } static voidfs/xfs/xfs_buf.h +25
-13@@ -30,15 +30,18 @@#define XBF_STALE (1 << 6)/* buffer has been staled, do not find it */ #define XBF_WRITE_FAIL (1 << 7)/* async writes have failed on this buffer */ -/* flags used only as arguments to access routines */ -#define XBF_TRYLOCK (1 << 16)/* lock requested, but do not wait */ -#define XBF_UNMAPPED (1 << 17)/* do not map the buffer */ +/* buffer type flags for write callbacks */ +#define _XBF_INODES (1 << 16)/* inode buffer */ /* flags used only internally */ #define _XBF_PAGES (1 << 20)/* backed by refcounted pages */ #define _XBF_KMEM (1 << 21)/* backed by heap memory */ #define _XBF_DELWRI_Q (1 << 22)/* buffer on a delwri queue */ +/* flags used only as arguments to access routines */ +#define XBF_TRYLOCK (1 << 30)/* lock requested, but do not wait */ +#define XBF_UNMAPPED (1 << 31)/* do not map the buffer */ +typedef unsigned int xfs_buf_flags_t ;#define XFS_BUF_FLAGS \@@ -50,12 +53,13 @@ typedef unsigned int xfs_buf_flags_t; { XBF_DONE, "DONE" }, \ { XBF_STALE, "STALE" }, \ { XBF_WRITE_FAIL, "WRITE_FAIL" }, \ - { XBF_TRYLOCK, "TRYLOCK" }, /* should never be set */\ - { XBF_UNMAPPED, "UNMAPPED" }, /* ditto */\ + { _XBF_INODES, "INODES" }, \ { _XBF_PAGES, "PAGES" }, \ { _XBF_KMEM, "KMEM" }, \ - { _XBF_DELWRI_Q, "DELWRI_Q" } - + { _XBF_DELWRI_Q, "DELWRI_Q" }, \ +/* The following interface flags should never be set */ \ + { XBF_TRYLOCK, "TRYLOCK" }, \ + { XBF_UNMAPPED, "UNMAPPED" }/* * Internal state flags. @@ -257,9 +261,23 @@ extern void xfs_buf_unlock(xfs_buf_t *);#define xfs_buf_islocked (bp) \ ((bp)->b_sema.count <= 0) +static inline void xfs_buf_relse (xfs_buf_t *bp) +{ +xfs_buf_unlock (bp); +xfs_buf_rele (bp); +} +/* Buffer Read and Write Routines */ extern int xfs_bwrite (struct xfs_buf *bp);extern void xfs_buf_ioend (struct xfs_buf *bp); +static inline void xfs_buf_ioend_finish (struct xfs_buf *bp) +{ +if (bp->b_flags &XBF_ASYNC ) +xfs_buf_relse (bp); +else +complete (&bp->b_iowait); +} +extern void __xfs_buf_ioerror (struct xfs_buf *bp,int error,xfs_failaddr_t failaddr);#define xfs_buf_ioerror (bp, err) __xfs_buf_ioerror((bp), (err), __this_address)@@ -324,12 +342,6 @@ static inline int xfs_buf_ispinned(struct xfs_buf *bp)return atomic_read (&bp->b_pin_count); } -static inline void xfs_buf_relse (xfs_buf_t *bp) -{ -xfs_buf_unlock (bp); -xfs_buf_rele (bp); -} -static inline int xfs_buf_verify_cksum (struct xfs_buf *bp,unsigned long cksum_offset) {fs/xfs/xfs_buf_item.c +31
-11@@ -1158,20 +1158,15 @@ xfs_buf_iodone_callback_error(return false; } -/* -* This is the iodone() function for buffers which have had callbacks attached -* to them by xfs_buf_attach_iodone(). We need to iterate the items on the -* callback list, mark the buffer as having no more callbacks and then push the -* buffer through IO completion processing. -*/ -void -xfs_buf_iodone_callbacks ( +static void +xfs_buf_run_callbacks (struct xfs_buf *bp) { +/* -* If there is an error, process it. Some errors require us -* to run callbacks after failure processing is done so we -* detect that and take appropriate action. +* If there is an error, process it. Some errors require us to run +* callbacks after failure processing is done so we detect that and take +* appropriate action. */ if (bp->b_error &&xfs_buf_iodone_callback_error (bp))return ;@@ -1188,9 +1183,34 @@ xfs_buf_iodone_callbacks( bp->b_log_item =NULL ;list_del_init (&bp->b_li_list); bp->b_iodone =NULL ; +} + +/* +* This is the iodone() function for buffers which have had callbacks attached +* to them by xfs_buf_attach_iodone(). We need to iterate the items on the +* callback list, mark the buffer as having no more callbacks and then push the +* buffer through IO completion processing. +*/ +void +xfs_buf_iodone_callbacks ( +struct xfs_buf *bp) +{ +xfs_buf_run_callbacks (bp);xfs_buf_ioend (bp); } +/* +* Inode buffer iodone callback function. +*/ +void +xfs_buf_inode_iodone ( +struct xfs_buf *bp) +{ +xfs_buf_run_callbacks (bp); +xfs_buf_ioend_finish (bp); +} + +/* * This is the iodone() function for buffers which have been * logged. It is called when they are eventually flushed out. fs/xfs/xfs_buf_item.h +1
-0@@ -59,6 +59,7 @@ void xfs_buf_attach_iodone(struct xfs_buf *, struct xfs_log_item *);void xfs_buf_iodone_callbacks (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *,struct xfs_log_item *); +void xfs_buf_inode_iodone (struct xfs_buf *);bool xfs_buf_log_check_iovec (struct xfs_log_iovec *iovec);extern kmem_zone_t *xfs_buf_item_zone;fs/xfs/xfs_inode.c +1
-1@@ -3860,13 +3860,13 @@ xfs_iflush_int(* completion on the buffer to remove the inode from the AIL and release * the flush lock. */ + bp->b_flags |= _XBF_INODES;xfs_buf_attach_iodone (bp, xfs_iflush_done, &iip->ili_item);/* generate the checksum. */ xfs_dinode_calc_crc (mp, dip);ASSERT (!list_empty (&bp->b_li_list)); -ASSERT (bp->b_iodone !=NULL );return error; }fs/xfs/xfs_trans_buf.c +3
-0@@ -626,6 +626,7 @@ xfs_trans_inode_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_INODE_BUF ; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }@@ -651,6 +652,7 @@ xfs_trans_stale_inode_buf( bip->bli_flags |=XFS_BLI_STALE_INODE ; bip->bli_item.li_cb = xfs_buf_iodone; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }@@ -675,6 +677,7 @@ xfs_trans_inode_alloc_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_INODE_ALLOC_BUF ; + bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 5/30 xfs: mark dquot buffers in cache
In reply to Dave Chinnerdquot buffers always have write IO callbacks, so by marking them directly we can avoid needing to attach ->b_iodone functions to them. This avoids an indirect call, and makes future modifications much simpler.
This is largely a rearrangement of the code at this point - no IO completion functionality changes at this point, just how the code is run is modified.
--- fs/xfs/xfs_buf.c | 5 +++++ fs/xfs/xfs_buf.h | 2 ++ fs/xfs/xfs_buf_item.c | 10 ++++++++++ fs/xfs/xfs_buf_item.h | 1 + fs/xfs/xfs_dquot.c | 1 + fs/xfs/xfs_trans_buf.c | 1 + 6 files changed, 20 insertions(+)
fs/xfs/xfs_buf.c +5 -0
@@ -1212,6 +1212,11 @@ xfs_buf_ioend(return ; } +if (bp->b_flags & _XBF_DQUOTS) { +xfs_buf_dquot_iodone (bp); +return ; + } +if (bp->b_iodone) { (*(bp->b_iodone))(bp);return ;
fs/xfs/xfs_buf.h +2 -0
@@ -32,6 +32,7 @@/* buffer type flags for write callbacks */ #define _XBF_INODES (1 << 16)/* inode buffer */ +#define _XBF_DQUOTS (1 << 17)/* dquot buffer */ /* flags used only internally */ #define _XBF_PAGES (1 << 20)/* backed by refcounted pages */
@@ -54,6 +55,7 @@ typedef unsigned int xfs_buf_flags_t; { XBF_STALE, "STALE" }, \ { XBF_WRITE_FAIL, "WRITE_FAIL" }, \ { _XBF_INODES, "INODES" }, \ + { _XBF_DQUOTS, "DQUOTS" }, \ { _XBF_PAGES, "PAGES" }, \ { _XBF_KMEM, "KMEM" }, \ { _XBF_DELWRI_Q, "DELWRI_Q" }, \
fs/xfs/xfs_buf_item.c +10 -0
@@ -1210,6 +1210,16 @@ xfs_buf_inode_iodone(xfs_buf_ioend_finish (bp); } +/* +* Dquot buffer iodone callback function. +*/ +void +xfs_buf_dquot_iodone ( +struct xfs_buf *bp) +{ +xfs_buf_run_callbacks (bp); +xfs_buf_ioend_finish (bp); +}/* * This is the iodone() function for buffers which have been
fs/xfs/xfs_buf_item.h +1 -0
@@ -60,6 +60,7 @@ void xfs_buf_attach_iodone(struct xfs_buf *,void xfs_buf_iodone_callbacks (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *,struct xfs_log_item *);void xfs_buf_inode_iodone (struct xfs_buf *); +void xfs_buf_dquot_iodone (struct xfs_buf *);bool xfs_buf_log_check_iovec (struct xfs_log_iovec *iovec);extern kmem_zone_t *xfs_buf_item_zone;
fs/xfs/xfs_dquot.c +1 -0
@@ -1179,6 +1179,7 @@ xfs_qm_dqflush(* Attach an iodone routine so that we can remove this dquot from the * AIL and release the flush lock once the dquot is synced to disk. */ + bp->b_flags |= _XBF_DQUOTS;xfs_buf_attach_iodone (bp, xfs_qm_dqflush_done, &dqp->q_logitem.qli_item);
fs/xfs/xfs_trans_buf.c +1 -0
@@ -788,5 +788,6 @@ xfs_trans_dquot_buf(break ; } + bp->b_flags |= _XBF_DQUOTS;xfs_trans_buf_set_type (tp, bp, type); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 6/30 xfs: mark log recovery buffers for completion
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Log recovery has it's own buffer write completion handler for buffers that it directly recovers. Convert these to direct calls by flagging these buffers as being log recovery buffers. The flag will get cleared by the log recovery IO completion routine, so it will never leak out of log recovery.
--- fs/xfs/xfs_buf.c | 10 ++++++++++ fs/xfs/xfs_buf.h | 2 ++ fs/xfs/xfs_buf_item_recover.c | 5 ++--- fs/xfs/xfs_dquot_item_recover.c | 2 +- fs/xfs/xfs_inode_item_recover.c | 2 +- fs/xfs/xfs_log_recover.c | 5 ++--- 6 files changed, 18 insertions(+), 8 deletions(-)
fs/xfs/xfs_buf.c +10 -0
@@ -14,6 +14,7 @@#include "xfs_mount.h" #include "xfs_trace.h" #include "xfs_log.h" +#include "xfs_log_recover.h" #include "xfs_trans.h" #include "xfs_buf_item.h" #include "xfs_errortag.h"
@@ -1207,6 +1208,15 @@ xfs_buf_ioend(if (read) goto out_finish; +/* +* If this is a log recovery buffer, we aren't doing transactional IO +* yet so we need to let it handle IO completions. +*/ +if (bp->b_flags & _XBF_LOGRECOVERY) { +xlog_recover_iodone (bp); +return ; + } +if (bp->b_flags & _XBF_INODES) {xfs_buf_inode_iodone (bp);return ;
fs/xfs/xfs_buf.h +2 -0
@@ -33,6 +33,7 @@/* buffer type flags for write callbacks */ #define _XBF_INODES (1 << 16)/* inode buffer */ #define _XBF_DQUOTS (1 << 17)/* dquot buffer */ +#define _XBF_LOGRECOVERY (1 << 18)/* log recovery buffer */ /* flags used only internally */ #define _XBF_PAGES (1 << 20)/* backed by refcounted pages */
@@ -56,6 +57,7 @@ typedef unsigned int xfs_buf_flags_t; { XBF_WRITE_FAIL, "WRITE_FAIL" }, \ { _XBF_INODES, "INODES" }, \ { _XBF_DQUOTS, "DQUOTS" }, \ + { _XBF_LOGRECOVERY, "LOG_RECOVERY" }, \ { _XBF_PAGES, "PAGES" }, \ { _XBF_KMEM, "KMEM" }, \ { _XBF_DELWRI_Q, "DELWRI_Q" }, \
fs/xfs/xfs_buf_item_recover.c +2 -3
@@ -419,8 +419,7 @@ xlog_recover_validate_buf_type(if (bp->b_ops) {struct xfs_buf_log_item *bip; -ASSERT (!bp->b_iodone || bp->b_iodone == xlog_recover_iodone); - bp->b_iodone = xlog_recover_iodone; + bp->b_flags |= _XBF_LOGRECOVERY;xfs_buf_item_init (bp, mp); bip = bp->b_log_item; bip->bli_item.li_lsn = current_lsn;
@@ -963,7 +962,7 @@ xlog_recover_buf_commit_pass2( error =xfs_bwrite (bp); }else {ASSERT (bp->b_mount == mp); - bp->b_iodone = xlog_recover_iodone; + bp->b_flags |= _XBF_LOGRECOVERY;xfs_buf_delwri_queue (bp, buffer_list); }
fs/xfs/xfs_dquot_item_recover.c +1 -1
@@ -153,7 +153,7 @@ xlog_recover_dquot_commit_pass2(ASSERT (dq_f->qlf_size ==2 );ASSERT (bp->b_mount == mp); - bp->b_iodone = xlog_recover_iodone; + bp->b_flags |= _XBF_LOGRECOVERY;xfs_buf_delwri_queue (bp, buffer_list); out_release:
fs/xfs/xfs_inode_item_recover.c +1 -1
@@ -376,7 +376,7 @@ xlog_recover_inode_commit_pass2(xfs_dinode_calc_crc (log->l_mp, dip);ASSERT (bp->b_mount == mp); - bp->b_iodone = xlog_recover_iodone; + bp->b_flags |= _XBF_LOGRECOVERY;xfs_buf_delwri_queue (bp, buffer_list); out_release:
fs/xfs/xfs_log_recover.c +2 -3
@@ -287,9 +287,8 @@ xlog_recover_iodone(if (bp->b_log_item)xfs_buf_item_relse (bp);ASSERT (bp->b_log_item ==NULL ); - - bp->b_iodone =NULL ; -xfs_buf_ioend (bp); + bp->b_flags &= ~_XBF_LOGRECOVERY; +xfs_buf_ioend_finish (bp); }/*
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 7/30 xfs: call xfs_buf_iodone directly
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
All unmarked dirty buffers should be in the AIL and have log items attached to them. Hence when they are written, we will run a callback to remove the item from the AIL if appropriate. Now that we've handled inode and dquot buffers, all remaining calls are to xfs_buf_iodone() and so we can hard code this rather than use an indirect call.
--- fs/xfs/xfs_buf.c | 24 ++++++++---------------- fs/xfs/xfs_buf.h | 6 +----- fs/xfs/xfs_buf_item.c | 40 ++++++++++------------------------------ fs/xfs/xfs_buf_item.h | 4 ++-- fs/xfs/xfs_trans_buf.c | 13 +++---------- 5 files changed, 24 insertions(+), 63 deletions(-)
fs/xfs/xfs_buf.c +8 -16
@@ -658,7 +658,6 @@ xfs_buf_find(*/ if (bp->b_flags &XBF_STALE ) {ASSERT ((bp->b_flags & _XBF_DELWRI_Q) ==0 ); -ASSERT (bp->b_iodone ==NULL ); bp->b_flags &= _XBF_KMEM | _XBF_PAGES; bp->b_ops =NULL ; }
@@ -1194,10 +1193,13 @@ xfs_buf_ioend(if (!bp->b_error && bp->b_io_error)xfs_buf_ioerror (bp, bp->b_io_error); -/* Only validate buffers that were read without errors */ -if (read && !bp->b_error && bp->b_ops) { -ASSERT (!bp->b_iodone); - bp->b_ops->verify_read (bp); +if (read) { +if (!bp->b_error && bp->b_ops) + bp->b_ops->verify_read (bp); +if (!bp->b_error) + bp->b_flags |=XBF_DONE ; +xfs_buf_ioend_finish (bp); +return ; }if (!bp->b_error) {
@@ -1205,9 +1207,6 @@ xfs_buf_ioend( bp->b_flags |=XBF_DONE ; } -if (read) - goto out_finish; -/* * If this is a log recovery buffer, we aren't doing transactional IO * yet so we need to let it handle IO completions.
@@ -1226,14 +1225,7 @@ xfs_buf_ioend(xfs_buf_dquot_iodone (bp);return ; } - -if (bp->b_iodone) { - (*(bp->b_iodone))(bp); -return ; - } - -out_finish: -xfs_buf_ioend_finish (bp); +xfs_buf_iodone (bp); } static void
fs/xfs/xfs_buf.h +1 -5
@@ -18,6 +18,7 @@/* * Base types */ +struct xfs_buf ;#define XFS_BUF_DADDR_NULL ((xfs_daddr_t) (-1LL))
@@ -102,10 +103,6 @@ typedef struct xfs_buftarg {struct ratelimit_state bt_ioerror_rl; } xfs_buftarg_t; -struct xfs_buf ; -typedef void (*xfs_buf_iodone_t )(struct xfs_buf *); - -#define XB_PAGES 2struct xfs_buf_map {
@@ -158,7 +155,6 @@ typedef struct xfs_buf {xfs_buftarg_t *b_target;/* buffer target (device) */ void *b_addr;/* virtual address of buffer */ struct work_struct b_ioend_work; -xfs_buf_iodone_t b_iodone;/* I/O completion function */ struct completion b_iowait;/* queue for I/O waiters */ struct xfs_buf_log_item *b_log_item;struct list_head b_li_list;/* Log items list head */
fs/xfs/xfs_buf_item.c +10 -30
@@ -460,7 +460,6 @@ xfs_buf_item_unpin(xfs_buf_do_callbacks (bp); bp->b_log_item =NULL ;list_del_init (&bp->b_li_list); - bp->b_iodone =NULL ; }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bp);
@@ -936,11 +935,7 @@ xfs_buf_item_free( }/* -* This is called when the buf log item is no longer needed. It should -* free the buf log item associated with the given buffer and clear -* the buffer's pointer to the buf log item. If there are no more -* items in the list, clear the b_iodone field of the buffer (see -* xfs_buf_attach_iodone() below). +* xfs_buf_item_relse() is called when the buf log item is no longer needed. */ void xfs_buf_item_relse(
@@ -952,9 +947,6 @@ xfs_buf_item_relse(ASSERT (!test_bit (XFS_LI_IN_AIL , &bip->bli_item.li_flags)); bp->b_log_item =NULL ; -if (list_empty (&bp->b_li_list)) - bp->b_iodone =NULL ; -xfs_buf_rele (bp);xfs_buf_item_free (bip); }
@@ -962,10 +954,7 @@ xfs_buf_item_relse(/* * Add the given log item with its callback to the list of callbacks -* to be called when the buffer's I/O completes. If it is not set -* already, set the buffer's b_iodone() routine to be -* xfs_buf_iodone_callbacks() and link the log item into the list of -* items rooted at b_li_list. +* to be called when the buffer's I/O completes. */ void xfs_buf_attach_iodone(
@@ -977,10 +966,6 @@ xfs_buf_attach_iodone( lip->li_cb = cb;list_add_tail (&lip->li_bio_list, &bp->b_li_list); - -ASSERT (bp->b_iodone ==NULL || - bp->b_iodone == xfs_buf_iodone_callbacks); - bp->b_iodone = xfs_buf_iodone_callbacks; }/*
@@ -1096,7 +1081,6 @@ xfs_buf_iodone_callback_error( goto out_stale;trace_xfs_buf_item_iodone_async (bp, _RET_IP_); -ASSERT (bp->b_iodone !=NULL ); cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error);
@@ -1182,28 +1166,24 @@ xfs_buf_run_callbacks(xfs_buf_do_callbacks (bp); bp->b_log_item =NULL ;list_del_init (&bp->b_li_list); - bp->b_iodone =NULL ; }/* -* This is the iodone() function for buffers which have had callbacks attached -* to them by xfs_buf_attach_iodone(). We need to iterate the items on the -* callback list, mark the buffer as having no more callbacks and then push the -* buffer through IO completion processing. +* Inode buffer iodone callback function. */ void -xfs_buf_iodone_callbacks ( +xfs_buf_inode_iodone (struct xfs_buf *bp) {xfs_buf_run_callbacks (bp); -xfs_buf_ioend (bp); +xfs_buf_ioend_finish (bp); }/* -* Inode buffer iodone callback function. +* Dquot buffer iodone callback function. */ void -xfs_buf_inode_iodone ( +xfs_buf_dquot_iodone (struct xfs_buf *bp) {xfs_buf_run_callbacks (bp);
@@ -1211,10 +1191,10 @@ xfs_buf_inode_iodone( }/* -* Dquot buffer iodone callback function. +* Dirty buffer iodone callback function. */ void -xfs_buf_dquot_iodone ( +xfs_buf_iodone (struct xfs_buf *bp) {xfs_buf_run_callbacks (bp);
@@ -1229,7 +1209,7 @@ xfs_buf_dquot_iodone(* care of cleaning up the buffer itself. */ void -xfs_buf_iodone ( +xfs_buf_item_iodone (struct xfs_buf *bp,struct xfs_log_item *lip) {
fs/xfs/xfs_buf_item.h +2 -2
@@ -57,10 +57,10 @@ bool xfs_buf_item_dirty_format(struct xfs_buf_log_item *);void xfs_buf_attach_iodone (struct xfs_buf *,void (*)(struct xfs_buf *,struct xfs_log_item *),struct xfs_log_item *); -void xfs_buf_iodone_callbacks (struct xfs_buf *); -void xfs_buf_iodone (struct xfs_buf *,struct xfs_log_item *); +void xfs_buf_item_iodone (struct xfs_buf *,struct xfs_log_item *);void xfs_buf_inode_iodone (struct xfs_buf *);void xfs_buf_dquot_iodone (struct xfs_buf *); +void xfs_buf_iodone (struct xfs_buf *);bool xfs_buf_log_check_iovec (struct xfs_log_iovec *iovec);extern kmem_zone_t *xfs_buf_item_zone;
fs/xfs/xfs_trans_buf.c +3 -10
@@ -465,24 +465,17 @@ xfs_trans_dirty_buf(ASSERT (bp->b_transp == tp);ASSERT (bip !=NULL ); -ASSERT (bp->b_iodone ==NULL || - bp->b_iodone == xfs_buf_iodone_callbacks);/* * Mark the buffer as needing to be written out eventually, * and set its iodone function to remove the buffer's buf log * item from the AIL and free it when the buffer is flushed -* to disk. See xfs_buf_attach_iodone() for more details -* on li_cb and xfs_buf_iodone_callbacks(). -* If we end up aborting this transaction, we trap this buffer -* inside the b_bdstrat callback so that this won't get written to -* disk. +* to disk. */ bp->b_flags |=XBF_DONE ;ASSERT (atomic_read (&bip->bli_refcount) >0 ); - bp->b_iodone = xfs_buf_iodone_callbacks; - bip->bli_item.li_cb = xfs_buf_iodone; + bip->bli_item.li_cb = xfs_buf_item_iodone;/* * If we invalidated the buffer within this transaction, then
@@ -651,7 +644,7 @@ xfs_trans_stale_inode_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_STALE_INODE ; - bip->bli_item.li_cb = xfs_buf_iodone; + bip->bli_item.li_cb = xfs_buf_item_iodone; bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 8/30 xfs: clean up whacky buffer log item list reinit
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When we've emptied the buffer log item list, it does a list_del_init on itself to reset it's pointers to itself. This is unnecessary as the list is already empty at this point - it was a left-over fragment from the list_head conversion of the buffer log item list. Remove them.
--- fs/xfs/xfs_buf_item.c | 2 -- 1 file changed, 2 deletions(-)
fs/xfs/xfs_buf_item.c +0 -2
@@ -459,7 +459,6 @@ xfs_buf_item_unpin(if (bip->bli_flags &XFS_BLI_STALE_INODE ) {xfs_buf_do_callbacks (bp); bp->b_log_item =NULL ; -list_del_init (&bp->b_li_list); }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bp);
@@ -1165,7 +1164,6 @@ xfs_buf_run_callbacks(xfs_buf_do_callbacks (bp); bp->b_log_item =NULL ; -list_del_init (&bp->b_li_list); }/*
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 9/30 xfs: make inode IO completion buffer centric
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Having different io completion callbacks for different inode states makes things complex. We can detect if the inode is stale via the XFS_ISTALE flag in IO completion, so we don't need a special callback just for this.
This means inodes only have a single iodone callback, and inode IO completion is entirely buffer centric at this point. Hence we no longer need to use a log item callback at all as we can just call xfs_iflush_done() directly from the buffer completions and walk the buffer log item list to complete the all inodes under IO.
--- fs/xfs/xfs_buf_item.c | 35 ++++++++++++++++++---- fs/xfs/xfs_inode.c | 6 ++-- fs/xfs/xfs_inode_item.c | 65 ++++++++++++++--------------------------- fs/xfs/xfs_inode_item.h | 5 ++-- 4 files changed, 56 insertions(+), 55 deletions(-)
fs/xfs/xfs_buf_item.c +30 -5
@@ -13,6 +13,8 @@#include "xfs_mount.h" #include "xfs_trans.h" #include "xfs_buf_item.h" +#include "xfs_inode.h" +#include "xfs_inode_item.h" #include "xfs_trans_priv.h" #include "xfs_trace.h" #include "xfs_log.h"
@@ -457,7 +459,8 @@ xfs_buf_item_unpin(* the AIL lock. */ if (bip->bli_flags &XFS_BLI_STALE_INODE ) { -xfs_buf_do_callbacks (bp); + lip->li_cb (bp, lip); +xfs_iflush_done (bp); bp->b_log_item =NULL ; }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );
@@ -1141,8 +1144,8 @@ xfs_buf_iodone_callback_error(return false; } -static void -xfs_buf_run_callbacks ( +static inline bool +xfs_buf_had_callback_errors (struct xfs_buf *bp) {
@@ -1152,7 +1155,7 @@ xfs_buf_run_callbacks(* appropriate action. */ if (bp->b_error &&xfs_buf_iodone_callback_error (bp)) -return ; +return true;/* * Successful IO or permanent error. Either way, we can clear the
@@ -1161,7 +1164,16 @@ xfs_buf_run_callbacks( bp->b_last_error =0 ; bp->b_retries =0 ; bp->b_first_retry_time =0 ; +return false; +} +static void +xfs_buf_run_callbacks ( +struct xfs_buf *bp) +{ + +if (xfs_buf_had_callback_errors (bp)) +return ;xfs_buf_do_callbacks (bp); bp->b_log_item =NULL ; }
@@ -1173,7 +1185,20 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -xfs_buf_run_callbacks (bp); +struct xfs_buf_log_item *blip = bp->b_log_item; +struct xfs_log_item *lip; + +if (xfs_buf_had_callback_errors (bp)) +return ; + +/* If there is a buf_log_item attached, run its callback */ +if (blip) { + lip = &blip->bli_item; + lip->li_cb (bp, lip); + bp->b_log_item =NULL ; + } + +xfs_iflush_done (bp);xfs_buf_ioend_finish (bp); }
fs/xfs/xfs_inode.c +2 -4
@@ -2677,7 +2677,6 @@ xfs_ifree_cluster(list_for_each_entry (lip, &bp->b_li_list, li_bio_list) {if (lip->li_type ==XFS_LI_INODE ) { iip = (struct xfs_inode_log_item *)lip; - lip->li_cb = xfs_istale_done;xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn);
@@ -2710,8 +2709,7 @@ xfs_ifree_cluster(xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn); -xfs_buf_attach_iodone (bp, xfs_istale_done, - &iip->ili_item); +xfs_buf_attach_iodone (bp,NULL , &iip->ili_item);if (ip != free_ip)xfs_iunlock (ip,XFS_ILOCK_EXCL );
@@ -3861,7 +3859,7 @@ xfs_iflush_int(* the flush lock. */ bp->b_flags |= _XBF_INODES; -xfs_buf_attach_iodone (bp, xfs_iflush_done, &iip->ili_item); +xfs_buf_attach_iodone (bp,NULL , &iip->ili_item);/* generate the checksum. */ xfs_dinode_calc_crc (mp, dip);
fs/xfs/xfs_inode_item.c +22 -43
111 lines not on this page · raw message
fs/xfs/xfs_inode_item.h +2 -3
@@ -36,15 +36,14 @@ struct xfs_inode_log_item {xfs_lsn_t ili_last_lsn;/* lsn at last transaction */ }; -static inline int xfs_inode_clean (xfs_inode_t *ip) +static inline int xfs_inode_clean (struct xfs_inode *ip) {return !ip->i_itemp || !(ip->i_itemp->ili_fields &XFS_ILOG_ALL ); }extern void xfs_inode_item_init (struct xfs_inode *,struct xfs_mount *);extern void xfs_inode_item_destroy (struct xfs_inode *); -extern void xfs_iflush_done (struct xfs_buf *,struct xfs_log_item *); -extern void xfs_istale_done (struct xfs_buf *,struct xfs_log_item *); +extern void xfs_iflush_done (struct xfs_buf *);extern void xfs_iflush_abort (struct xfs_inode *);extern int xfs_inode_item_format_convert (xfs_log_iovec_t *,struct xfs_inode_log_format *);
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 10/30 xfs: use direct calls for dquot IO completion
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Similar to inodes, we can call the dquot IO completion functions directly from the buffer completion code, removing another user of log item callbacks for IO completion processing.
--- fs/xfs/xfs_buf_item.c | 18 +++++++++++++++++- fs/xfs/xfs_dquot.c | 18 ++++++++++++++---- fs/xfs/xfs_dquot.h | 1 + 3 files changed, 32 insertions(+), 5 deletions(-)
fs/xfs/xfs_buf_item.c +17 -1
@@ -15,6 +15,9 @@#include "xfs_buf_item.h" #include "xfs_inode.h" #include "xfs_inode_item.h" +#include "xfs_quota.h" +#include "xfs_dquot_item.h" +#include "xfs_dquot.h" #include "xfs_trans_priv.h" #include "xfs_trace.h" #include "xfs_log.h"
@@ -1209,7 +1212,20 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -xfs_buf_run_callbacks (bp); +struct xfs_buf_log_item *blip = bp->b_log_item; +struct xfs_log_item *lip; + +if (xfs_buf_had_callback_errors (bp)) +return ; + +/* a newly allocated dquot buffer might have a log item attached */ +if (blip) { + lip = &blip->bli_item; + lip->li_cb (bp, lip); + bp->b_log_item =NULL ; + } + +xfs_dquot_done (bp);xfs_buf_ioend_finish (bp); }
fs/xfs/xfs_dquot.c +14 -4
@@ -1048,9 +1048,8 @@ xfs_qm_dqrele(* from the AIL if it has not been re-logged, and unlocking the dquot's * flush lock. This behavior is very similar to that of inodes.. */ -STATIC void +static void xfs_qm_dqflush_done ( -struct xfs_buf *bp,struct xfs_log_item *lip) {struct xfs_dq_logitem *qip = (struct xfs_dq_logitem *)lip;
@@ -1091,6 +1090,18 @@ xfs_qm_dqflush_done(xfs_dqfunlock (dqp); } +void +xfs_dquot_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; + +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +list_del_init (&lip->li_bio_list); +xfs_qm_dqflush_done (lip); + } +} +/* * Write a modified dquot to disk. * The dquot must be locked and the flush lock too taken by caller.
@@ -1180,8 +1191,7 @@ xfs_qm_dqflush(* AIL and release the flush lock once the dquot is synced to disk. */ bp->b_flags |= _XBF_DQUOTS; -xfs_buf_attach_iodone (bp, xfs_qm_dqflush_done, - &dqp->q_logitem.qli_item); +xfs_buf_attach_iodone (bp,NULL , &dqp->q_logitem.qli_item);/* * If the buffer is pinned then push on the log so we won't
fs/xfs/xfs_dquot.h +1 -0
@@ -174,6 +174,7 @@ void xfs_qm_dqput(struct xfs_dquot *dqp);void xfs_dqlock2 (struct xfs_dquot *,struct xfs_dquot *);void xfs_dquot_set_prealloc_limits (struct xfs_dquot *); +void xfs_dquot_done (struct xfs_buf *);static inline struct xfs_dquot *xfs_qm_dqhold (struct xfs_dquot *dqp) {
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 11/30 xfs: clean up the buffer iodone callback functions
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we've sorted inode and dquot buffers, we can apply the same cleanups to dirty buffers with buffer log items. They only have one callback, too, so we don't need the log item callback. Collapse the iodone functions and remove all the now unnecessary infrastructure around callback processing.
--- fs/xfs/xfs_buf_item.c | 140 +++++++++-------------------------------- fs/xfs/xfs_buf_item.h | 1 - fs/xfs/xfs_trans_buf.c | 2 - 3 files changed, 29 insertions(+), 114 deletions(-)
fs/xfs/xfs_buf_item.c +29 -111
@@ -30,7 +30,7 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip) return container_of(lip, struct xfs_buf_log_item, bli_item); } -STATIC void xfs_buf_do_callbacks(struct xfs_buf *bp); +static void xfs_buf_item_done (struct xfs_buf *bp);/* Is this log iovec plausibly large enough to contain the buffer log format? */ bool
@@ -462,9 +462,8 @@ xfs_buf_item_unpin(* the AIL lock. */ if (bip->bli_flags &XFS_BLI_STALE_INODE ) { - lip->li_cb (bp, lip); +xfs_buf_item_done (bp);xfs_iflush_done (bp); - bp->b_log_item =NULL ; }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bp);
@@ -973,46 +972,6 @@ xfs_buf_attach_iodone(list_add_tail (&lip->li_bio_list, &bp->b_li_list); } -/* -* We can have many callbacks on a buffer. Running the callbacks individually -* can cause a lot of contention on the AIL lock, so we allow for a single -* callback to be able to scan the remaining items in bp->b_li_list for other -* items of the same type and callback to be processed in the first call. -* -* As a result, the loop walking the callback list below will also modify the -* list. it removes the first item from the list and then runs the callback. -* The loop then restarts from the new first item int the list. This allows the -* callback to scan and modify the list attached to the buffer and we don't -* have to care about maintaining a next item pointer. -*/ -STATIC void -xfs_buf_do_callbacks ( -struct xfs_buf *bp) -{ -struct xfs_buf_log_item *blip = bp->b_log_item; -struct xfs_log_item *lip; - -/* If there is a buf_log_item attached, run its callback */ -if (blip) { - lip = &blip->bli_item; - lip->li_cb (bp, lip); - } - -while (!list_empty (&bp->b_li_list)) { - lip = list_first_entry(&bp->b_li_list, struct xfs_log_item, - li_bio_list); - -/* -* Remove the item from the list, so we don't have any -* confusion if the item is added to another buf. -* Don't touch the log item after calling its -* callback, because it could have freed itself. -*/ -list_del_init (&lip->li_bio_list); - lip->li_cb (bp, lip); - } -} -/* * Invoke the error state callback for each log item affected by the failed I/O. *
@@ -1025,8 +984,8 @@ STATIC voidxfs_buf_do_callbacks_fail ( struct xfs_buf *bp) { +struct xfs_ail *ailp = bp->b_mount->m_ail;struct xfs_log_item *lip; -struct xfs_ail *ailp;/* * Buffer log item errors are handled directly by xfs_buf_item_push()
@@ -1036,9 +995,6 @@ xfs_buf_do_callbacks_fail(if (list_empty (&bp->b_li_list))return ; - lip = list_first_entry(&bp->b_li_list, struct xfs_log_item, - li_bio_list); - ailp = lip->li_ailp;spin_lock (&ailp->ail_lock);list_for_each_entry (lip, &bp->b_li_list, li_bio_list) {if (lip->li_ops->iop_error)
@@ -1051,22 +1007,11 @@ static boolxfs_buf_iodone_callback_error ( struct xfs_buf *bp) { -struct xfs_buf_log_item *bip = bp->b_log_item; -struct xfs_log_item *lip; -struct xfs_mount *mp; +struct xfs_mount *mp = bp->b_mount;static ulong lasttime;static xfs_buftarg_t *lasttarg;struct xfs_error_cfg *cfg; -/* -* The failed buffer might not have a buf_log_item attached or the -* log_item list might be empty. Get the mp from the available -* xfs_log_item -*/ - lip = list_first_entry_or_null(&bp->b_li_list, struct xfs_log_item, - li_bio_list); - mp = lip ? lip->li_mountp : bip->bli_item.li_mountp; -/* * If we've already decided to shutdown the filesystem because of * I/O errors, there's no point in giving this a retry.
@@ -1171,14 +1116,27 @@ xfs_buf_had_callback_errors( }static void -xfs_buf_run_callbacks ( +xfs_buf_item_done (struct xfs_buf *bp) { +struct xfs_buf_log_item *bip = bp->b_log_item; -if (xfs_buf_had_callback_errors (bp)) +if (!bip)return ; -xfs_buf_do_callbacks (bp); + +/* +* If we are forcibly shutting down, this may well be off the AIL +* already. That's because we simulate the log-committed callbacks to +* unpin these buffers. Or we may never have put this item on AIL +* because of the transaction was aborted forcibly. +* xfs_trans_ail_delete() takes care of these. +* +* Either way, AIL is useless if we're forcing a shutdown. +*/ +xfs_trans_ail_delete (&bip->bli_item,SHUTDOWN_CORRUPT_INCORE ); bp->b_log_item =NULL ; +xfs_buf_item_free (bip); +xfs_buf_rele (bp); }/*
@@ -1188,19 +1146,10 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -struct xfs_buf_log_item *blip = bp->b_log_item; -struct xfs_log_item *lip; -if (xfs_buf_had_callback_errors (bp))return ; -/* If there is a buf_log_item attached, run its callback */ -if (blip) { - lip = &blip->bli_item; - lip->li_cb (bp, lip); - bp->b_log_item =NULL ; - } - +xfs_buf_item_done (bp);xfs_iflush_done (bp);xfs_buf_ioend_finish (bp); }
@@ -1212,59 +1161,28 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -struct xfs_buf_log_item *blip = bp->b_log_item; -struct xfs_log_item *lip; -if (xfs_buf_had_callback_errors (bp))return ;/* a newly allocated dquot buffer might have a log item attached */ -if (blip) { - lip = &blip->bli_item; - lip->li_cb (bp, lip); - bp->b_log_item =NULL ; - } - +xfs_buf_item_done (bp);xfs_dquot_done (bp);xfs_buf_ioend_finish (bp); }/* * Dirty buffer iodone callback function. +* +* Note that for things like remote attribute buffers, there may not be a buffer +* log item here, so processing the buffer log item must remain be optional. */ void xfs_buf_iodone (struct xfs_buf *bp) { -xfs_buf_run_callbacks (bp); -xfs_buf_ioend_finish (bp); -} - -/* -* This is the iodone() function for buffers which have been -* logged. It is called when they are eventually flushed out. -* It should remove the buf item from the AIL, and free the buf item. -* It is called by xfs_buf_iodone_callbacks() above which will take -* care of cleaning up the buffer itself. -*/ -void -xfs_buf_item_iodone ( -struct xfs_buf *bp, -struct xfs_log_item *lip) -{ -ASSERT (BUF_ITEM (lip)->bli_buf == bp); - -xfs_buf_rele (bp); +if (xfs_buf_had_callback_errors (bp)) +return ; -/* -* If we are forcibly shutting down, this may well be off the AIL -* already. That's because we simulate the log-committed callbacks to -* unpin these buffers. Or we may never have put this item on AIL -* because of the transaction was aborted forcibly. -* xfs_trans_ail_delete() takes care of these. -* -* Either way, AIL is useless if we're forcing a shutdown. -*/ -xfs_trans_ail_delete (lip,SHUTDOWN_CORRUPT_INCORE ); -xfs_buf_item_free (BUF_ITEM (lip)); +xfs_buf_item_done (bp); +xfs_buf_ioend_finish (bp); }
fs/xfs/xfs_buf_item.h +0 -1
@@ -57,7 +57,6 @@ bool xfs_buf_item_dirty_format(struct xfs_buf_log_item *);void xfs_buf_attach_iodone (struct xfs_buf *,void (*)(struct xfs_buf *,struct xfs_log_item *),struct xfs_log_item *); -void xfs_buf_item_iodone (struct xfs_buf *,struct xfs_log_item *);void xfs_buf_inode_iodone (struct xfs_buf *);void xfs_buf_dquot_iodone (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *);
fs/xfs/xfs_trans_buf.c +0 -2
@@ -475,7 +475,6 @@ xfs_trans_dirty_buf( bp->b_flags |=XBF_DONE ;ASSERT (atomic_read (&bip->bli_refcount) >0 ); - bip->bli_item.li_cb = xfs_buf_item_iodone;/* * If we invalidated the buffer within this transaction, then
@@ -644,7 +643,6 @@ xfs_trans_stale_inode_buf(ASSERT (atomic_read (&bip->bli_refcount) >0 ); bip->bli_flags |=XFS_BLI_STALE_INODE ; - bip->bli_item.li_cb = xfs_buf_item_iodone; bp->b_flags |= _XBF_INODES;xfs_trans_buf_set_type (tp, bp,XFS_BLFT_DINO_BUF ); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 12/30 xfs: get rid of log item callbacks
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
They are not used anymore, so remove them from the log item and the buffer iodone attachment interfaces.
--- fs/xfs/xfs_buf_item.c | 17 ----------------- fs/xfs/xfs_buf_item.h | 3 --- fs/xfs/xfs_dquot.c | 6 +++--- fs/xfs/xfs_inode.c | 5 +++-- fs/xfs/xfs_trans.h | 4 ---- 5 files changed, 6 insertions(+), 29 deletions(-)
fs/xfs/xfs_buf_item.c +0 -17
@@ -955,23 +955,6 @@ xfs_buf_item_relse(xfs_buf_item_free (bip); } - -/* -* Add the given log item with its callback to the list of callbacks -* to be called when the buffer's I/O completes. -*/ -void -xfs_buf_attach_iodone ( -struct xfs_buf *bp, -void (*cb)(struct xfs_buf *,struct xfs_log_item *), -struct xfs_log_item *lip) -{ -ASSERT (xfs_buf_islocked (bp)); - - lip->li_cb = cb; -list_add_tail (&lip->li_bio_list, &bp->b_li_list); -} -/* * Invoke the error state callback for each log item affected by the failed I/O. *
fs/xfs/xfs_buf_item.h +0 -3
@@ -54,9 +54,6 @@ void xfs_buf_item_relse(struct xfs_buf *);bool xfs_buf_item_put (struct xfs_buf_log_item *);void xfs_buf_item_log (struct xfs_buf_log_item *,uint ,uint );bool xfs_buf_item_dirty_format (struct xfs_buf_log_item *); -void xfs_buf_attach_iodone (struct xfs_buf *, -void (*)(struct xfs_buf *,struct xfs_log_item *), -struct xfs_log_item *);void xfs_buf_inode_iodone (struct xfs_buf *);void xfs_buf_dquot_iodone (struct xfs_buf *);void xfs_buf_iodone (struct xfs_buf *);
fs/xfs/xfs_dquot.c +3 -3
@@ -1187,11 +1187,11 @@ xfs_qm_dqflush( }/* -* Attach an iodone routine so that we can remove this dquot from the -* AIL and release the flush lock once the dquot is synced to disk. +* Attach the dquot to the buffer so that we can remove this dquot from +* the AIL and release the flush lock once the dquot is synced to disk. */ bp->b_flags |= _XBF_DQUOTS; -xfs_buf_attach_iodone (bp,NULL , &dqp->q_logitem.qli_item); +list_add_tail (&dqp->q_logitem.qli_item.li_bio_list, &bp->b_li_list);/* * If the buffer is pinned then push on the log so we won't
fs/xfs/xfs_inode.c +3 -2
@@ -2709,7 +2709,8 @@ xfs_ifree_cluster(xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn); -xfs_buf_attach_iodone (bp,NULL , &iip->ili_item); +list_add_tail (&iip->ili_item.li_bio_list, + &bp->b_li_list);if (ip != free_ip)xfs_iunlock (ip,XFS_ILOCK_EXCL );
@@ -3859,7 +3860,7 @@ xfs_iflush_int(* the flush lock. */ bp->b_flags |= _XBF_INODES; -xfs_buf_attach_iodone (bp,NULL , &iip->ili_item); +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list);/* generate the checksum. */ xfs_dinode_calc_crc (mp, dip);
fs/xfs/xfs_trans.h +0 -4
@@ -37,10 +37,6 @@ struct xfs_log_item {unsigned long li_flags;/* misc flags */ struct xfs_buf *li_buf;/* real buffer pointer */ struct list_head li_bio_list;/* buffer item list */ -void (*li_cb)(struct xfs_buf *, -struct xfs_log_item *); -/* buffer item iodone */ -/* callback func */ const struct xfs_item_ops *li_ops;/* function list */ /* delayed logging */
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 13/30 xfs: handle buffer log item IO errors directly
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Currently when a buffer with attached log items has an IO error it called ->iop_error for each attched log item. These all call xfs_set_li_failed() to handle the error, but we are about to change the way log items manage buffers. hence we first need to remove the per-item dependency on buffer handling done by xfs_set_li_failed().
We already have specific buffer type IO completion routines, so move the log item error handling out of the generic error handling and into the log item specific functions so we can implement per-type error handling easily.
This requires a more complex return value from the error handling code so that we can take the correct action the failure handling requires. This results in some repeated boilerplate in the functions, but that can be cleaned up later once all the changes cascade through this code.
--- fs/xfs/xfs_buf_item.c | 215 ++++++++++++++++++++++++++++-------------- 1 file changed, 145 insertions(+), 70 deletions(-)
fs/xfs/xfs_buf_item.c +145 -70
@@ -986,21 +986,24 @@ xfs_buf_do_callbacks_fail(spin_unlock (&ailp->ail_lock); } +/* +* Decide if we're going to retry the write after a failure, and prepare +* the buffer for retrying the write. +*/ static bool -xfs_buf_iodone_callback_error ( +xfs_buf_ioerror_fail_without_retry (struct xfs_buf *bp) {struct xfs_mount *mp = bp->b_mount;static ulong lasttime;static xfs_buftarg_t *lasttarg; -struct xfs_error_cfg *cfg;/* * If we've already decided to shutdown the filesystem because of * I/O errors, there's no point in giving this a retry. */ if (XFS_FORCED_SHUTDOWN (mp)) - goto out_stale; +return true;if (bp->b_target != lasttarg ||time_after (jiffies, (lasttime +5 *HZ ))) {
@@ -1011,91 +1014,115 @@ xfs_buf_iodone_callback_error(/* synchronous writes will have callers process the error */ if (!(bp->b_flags &XBF_ASYNC )) - goto out_stale; - -trace_xfs_buf_item_iodone_async (bp, _RET_IP_); - - cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +return true; +return false; +} -/* -* If the write was asynchronous then no one will be looking for the -* error. If this is the first failure of this type, clear the error -* state and write the buffer out again. This means we always retry an -* async write failure at least once, but we also need to set the buffer -* up to behave correctly now for repeated failures. -*/ -if (!(bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) || - bp->b_last_error != bp->b_error) { - bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); - bp->b_last_error = bp->b_error; -if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && - !bp->b_first_retry_time) - bp->b_first_retry_time = jiffies; +static bool +xfs_buf_ioerror_retry ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +if (bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) +return false; +if (bp->b_last_error == bp->b_error) +return false; -xfs_buf_ioerror (bp,0 ); -xfs_buf_submit (bp); -return true; - } + bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); + bp->b_last_error = bp->b_error; +if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && + !bp->b_first_retry_time) + bp->b_first_retry_time = jiffies; +return true; +} -/* -* Repeated failure on an async write. Take action according to the -* error configuration we have been set up to use. -*/ +/* +* Account for this latest trip around the retry handler, and decide if +* we've failed enough times to constitute a permanent failure. +*/ +static bool +xfs_buf_ioerror_permanent ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +struct xfs_mount *mp = bp->b_mount;if (cfg->max_retries !=XFS_ERR_RETRY_FOREVER && ++bp->b_retries > cfg->max_retries) - goto permanent_error; +return true;if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER &&time_after (jiffies, cfg->retry_timeout + bp->b_first_retry_time)) - goto permanent_error; +return true;/* At unmount we may treat errors differently */ if ((mp->m_flags &XFS_MOUNT_UNMOUNTING ) && mp->m_fail_unmount) - goto permanent_error; - -/* -* Still a transient error, run IO completion failure callbacks and let -* the higher layers retry the buffer. -*/ -xfs_buf_do_callbacks_fail (bp); -xfs_buf_ioerror (bp,0 ); -xfs_buf_relse (bp); -return true; +return true; -/* -* Permanent error - we need to trigger a shutdown if we haven't already -* to indicate that inconsistency will result from this action. -*/ -permanent_error: -xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); -out_stale: -xfs_buf_stale (bp); - bp->b_flags |=XBF_DONE ; -trace_xfs_buf_error_relse (bp, _RET_IP_);return false; } -static inline bool -xfs_buf_had_callback_errors ( +/* +* On a sync write or shutdown we just want to stale the buffer and let the +* caller handle the error in bp->b_error appropriately. +* +* If the write was asynchronous then no one will be looking for the error. If +* this is the first failure of this type, clear the error state and write the +* buffer out again. This means we always retry an async write failure at least +* once, but we also need to set the buffer up to behave correctly now for +* repeated failures. +* +* If we get repeated async write failures, then we take action according to the +* error configuration we have been set up to use. +* +* Multi-state return value: +* +* XBF_IOERROR_FINISH: clear IO error retry state and run callback completions +* XBF_IOERROR_DONE: resubmitted immediately, do not run any completions +* XBF_IOERROR_FAIL: transient error, run failure callback completions and then +* release the buffer +*/ +enum { +XBF_IOERROR_FINISH , +XBF_IOERROR_DONE , +XBF_IOERROR_FAIL , +}; + +static int +xfs_buf_iodone_error (struct xfs_buf *bp) { +struct xfs_mount *mp = bp->b_mount; +struct xfs_error_cfg *cfg; -/* -* If there is an error, process it. Some errors require us to run -* callbacks after failure processing is done so we detect that and take -* appropriate action. -*/ -if (bp->b_error &&xfs_buf_iodone_callback_error (bp)) -return true; +if (xfs_buf_ioerror_fail_without_retry (bp)) + goto out_stale; + +trace_xfs_buf_item_iodone_async (bp, _RET_IP_); + + cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +if (xfs_buf_ioerror_retry (bp, cfg)) { +xfs_buf_ioerror (bp,0 ); +xfs_buf_submit (bp); +return XBF_IOERROR_DONE ; + }/* -* Successful IO or permanent error. Either way, we can clear the -* retry state here in preparation for the next error that may occur. +* Permanent error - we need to trigger a shutdown if we haven't already +* to indicate that inconsistency will result from this action. */ - bp->b_last_error =0 ; - bp->b_retries =0 ; - bp->b_first_retry_time =0 ; -return false; +if (xfs_buf_ioerror_permanent (bp, cfg)) { +xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); + goto out_stale; + } + +/* Still considered a transient error. Caller will schedule retries. */ +return XBF_IOERROR_FAIL ; + +out_stale: +xfs_buf_stale (bp); + bp->b_flags |=XBF_DONE ; +trace_xfs_buf_error_relse (bp, _RET_IP_); +return XBF_IOERROR_FINISH ; } static void
@@ -1122,6 +1149,15 @@ xfs_buf_item_done(xfs_buf_rele (bp); } +static inline void +xfs_buf_clear_ioerror_retry_state ( +struct xfs_buf *bp) +{ + bp->b_last_error =0 ; + bp->b_retries =0 ; + bp->b_first_retry_time =0 ; +} +/* * Inode buffer iodone callback function. */
@@ -1129,9 +1165,22 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_iflush_done (bp);xfs_buf_ioend_finish (bp);
@@ -1144,9 +1193,22 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);/* a newly allocated dquot buffer might have a log item attached */ xfs_buf_item_done (bp);xfs_dquot_done (bp);
@@ -1163,9 +1225,22 @@ voidxfs_buf_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_buf_ioend_finish (bp); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 13/30] xfs: handle buffer log item IO errors directly
Quoted 28 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Currently when a buffer with attached log items has an IO error it called ->iop_error for each attched log item. These all call
xfs_set_li_failed()to handle the error, but we are about to change the way log items manage buffers. hence we first need to remove the per-item dependency on buffer handling done byxfs_set_li_failed().We already have specific buffer type IO completion routines, so move the log item error handling out of the generic error handling and into the log item specific functions so we can implement per-type error handling easily.
This requires a more complex return value from the error handling code so that we can take the correct action the failure handling requires. This results in some repeated boilerplate in the functions, but that can be cleaned up later once all the changes cascade through this code.
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf_item.c | 215 ++++++++++++++++++++++++++++-------------- 1 file changed, 145 insertions(+), 70 deletions(-)fs/xfs/xfs_buf_item.c +0
-0
...
Quoted 26 lines
fs/xfs/xfs_buf_item.c +3
-18@@ -1011,91 +1014,115 @@ xfs_buf_iodone_callback_error(/* synchronous writes will have callers process the error */ if (!(bp->b_flags &XBF_ASYNC )) - goto out_stale; - -trace_xfs_buf_item_iodone_async (bp, _RET_IP_); - - cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +return true; +return false; +} -/* -* If the write was asynchronous then no one will be looking for the -* error. If this is the first failure of this type, clear the error -* state and write the buffer out again. This means we always retry an -* async write failure at least once, but we also need to set the buffer -* up to behave correctly now for repeated failures. -*/ -if (!(bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) || - bp->b_last_error != bp->b_error) { - bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); - bp->b_last_error = bp->b_error; -if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && - !bp->b_first_retry_time)
fs/xfs/xfs_buf_item.c xfs_buf_iodone_callback_error( · line 1029
Quoted 10 lines
fs/xfs/xfs_buf_item.c +9
-1- bp->b_first_retry_time = jiffies; +static bool +xfs_buf_ioerror_retry ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +if (bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) +return false; +if (bp->b_last_error == bp->b_error) +return false;
This looks like a subtle logic change. The current code issues the internal retry if the flag isn't set (i.e. first ioerror in the sequence) or if the current error code differs from the previous. This code only looks at ->b_last_error if the fail flag wasn't set.
Quoted 13 lines
fs/xfs/xfs_buf_item.c +7
-4-xfs_buf_ioerror (bp,0 ); -xfs_buf_submit (bp); -return true; - } + bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); + bp->b_last_error = bp->b_error; +if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && + !bp->b_first_retry_time) + bp->b_first_retry_time = jiffies; +return true; +}
...
Quoted 15 lines
fs/xfs/xfs_buf_item.c +13
-2-static inline bool -xfs_buf_had_callback_errors( +/* +* On a sync write or shutdown we just want to stale the buffer and let the +* caller handle the error in bp->b_error appropriately. +* +* If the write was asynchronous then no one will be looking for the error. If +* this is the first failure of this type, clear the error state and write the +* buffer out again. This means we always retry an async write failure at least +* once, but we also need to set the buffer up to behave correctly now for +* repeated failures. +* +* If we get repeated async write failures, then we take action according to the +* error configuration we have been set up to use. +*
fs/xfs/xfs_buf_item.c xfs_buf_iodone_callback_error( · line 1086
Quoted 10 lines
fs/xfs/xfs_buf_item.c +10
-0+* Multi-state return value: +* +* XBF_IOERROR_FINISH: clear IO error retry state and run callback completions +* XBF_IOERROR_DONE: resubmitted immediately, do not run any completions +* XBF_IOERROR_FAIL: transient error, run failure callback completions and then +* release the buffer +*/ +enum { +XBF_IOERROR_FINISH , +XBF_IOERROR_DONE ,
I like the enum, but I have a hard time distinguishing what the difference is between FINISH and DONE based on the naming. I think RESUBMIT would be more clear than DONE and perhaps have the resubmit in the caller, but then we'd have to duplicate that as well. Eh, perhaps it makes sense to defer such potential cleanups to the end.
Brian
Quoted 146 lines
fs/xfs/xfs_buf_item.c +84
-16+ XBF_IOERROR_FAIL, +}; + +static int +xfs_buf_iodone_error (struct xfs_buf *bp) { +struct xfs_mount *mp = bp->b_mount; +struct xfs_error_cfg *cfg; -/* -* If there is an error, process it. Some errors require us to run -* callbacks after failure processing is done so we detect that and take -* appropriate action. -*/ -if (bp->b_error &&xfs_buf_iodone_callback_error (bp)) -return true; +if (xfs_buf_ioerror_fail_without_retry (bp)) + goto out_stale; + +trace_xfs_buf_item_iodone_async (bp, _RET_IP_); + + cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +if (xfs_buf_ioerror_retry (bp, cfg)) { +xfs_buf_ioerror (bp,0 ); +xfs_buf_submit (bp); +return XBF_IOERROR_DONE ; + }/* -* Successful IO or permanent error. Either way, we can clear the -* retry state here in preparation for the next error that may occur. +* Permanent error - we need to trigger a shutdown if we haven't already +* to indicate that inconsistency will result from this action. */ - bp->b_last_error =0 ; - bp->b_retries =0 ; - bp->b_first_retry_time =0 ; -return false; +if (xfs_buf_ioerror_permanent (bp, cfg)) { +xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); + goto out_stale; + } + +/* Still considered a transient error. Caller will schedule retries. */ +return XBF_IOERROR_FAIL ; + +out_stale: +xfs_buf_stale (bp); + bp->b_flags |=XBF_DONE ; +trace_xfs_buf_error_relse (bp, _RET_IP_); +return XBF_IOERROR_FINISH ; } static void@@ -1122,6 +1149,15 @@ xfs_buf_item_done(xfs_buf_rele (bp); } +static inline void +xfs_buf_clear_ioerror_retry_state ( +struct xfs_buf *bp) +{ + bp->b_last_error =0 ; + bp->b_retries =0 ; + bp->b_first_retry_time =0 ; +} +/* * Inode buffer iodone callback function. */ @@ -1129,9 +1165,22 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_iflush_done (bp);xfs_buf_ioend_finish (bp);@@ -1144,9 +1193,22 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);/* a newly allocated dquot buffer might have a log item attached */ xfs_buf_item_done (bp);xfs_dquot_done (bp);@@ -1163,9 +1225,22 @@ voidxfs_buf_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_buf_ioend_finish (bp); }Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 13/30] xfs: handle buffer log item IO errors directly
Quoted 63 lines · Brian Foster
Quoted 28 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Currently when a buffer with attached log items has an IO error it called ->iop_error for each attched log item. These all call
xfs_set_li_failed()to handle the error, but we are about to change the way log items manage buffers. hence we first need to remove the per-item dependency on buffer handling done byxfs_set_li_failed().We already have specific buffer type IO completion routines, so move the log item error handling out of the generic error handling and into the log item specific functions so we can implement per-type error handling easily.
This requires a more complex return value from the error handling code so that we can take the correct action the failure handling requires. This results in some repeated boilerplate in the functions, but that can be cleaned up later once all the changes cascade through this code.
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf_item.c | 215 ++++++++++++++++++++++++++++-------------- 1 file changed, 145 insertions(+), 70 deletions(-)fs/xfs/xfs_buf_item.c +0
-0...
Quoted 33 lines
fs/xfs/xfs_buf_item.c +9
-19@@ -1011,91 +1014,115 @@ xfs_buf_iodone_callback_error(/* synchronous writes will have callers process the error */ if (!(bp->b_flags &XBF_ASYNC )) - goto out_stale; - -trace_xfs_buf_item_iodone_async (bp, _RET_IP_); - - cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +return true; +return false; +} -/* -* If the write was asynchronous then no one will be looking for the -* error. If this is the first failure of this type, clear the error -* state and write the buffer out again. This means we always retry an -* async write failure at least once, but we also need to set the buffer -* up to behave correctly now for repeated failures. -*/ -if (!(bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) || - bp->b_last_error != bp->b_error) { - bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); - bp->b_last_error = bp->b_error; -if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && - !bp->b_first_retry_time) - bp->b_first_retry_time = jiffies; +static bool +xfs_buf_ioerror_retry ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +if (bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL ))
Quoted 8 lines
fs/xfs/xfs_buf_item.c xfs_buf_iodone_callback_error( · line 1029
Quoted 3 lines
fs/xfs/xfs_buf_item.c +3
-0+return false; +if (bp->b_last_error == bp->b_error) +return false;This looks like a subtle logic change. The current code issues the internal retry if the flag isn't set (i.e. first ioerror in the sequence) or if the current error code differs from the previous. This code only looks at ->b_last_error if the fail flag wasn't set.
Yeah, well spotted. Brain fart: !A||!B == !(A && B). It should be:
if ((bp->b_flags & (XBF_STALE | XBF_WRITE_FAIL)) && bp->b_last_error == bp->b_error) return false;
Quoted 34 lines
Quoted 13 lines
fs/xfs/xfs_buf_item.c +7
-4-xfs_buf_ioerror (bp,0 ); -xfs_buf_submit (bp); -return true; - } + bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); + bp->b_last_error = bp->b_error; +if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && + !bp->b_first_retry_time) + bp->b_first_retry_time = jiffies; +return true; +}...
Quoted 20 lines
fs/xfs/xfs_buf_item.c +18
-2-static inline bool -xfs_buf_had_callback_errors( +/* +* On a sync write or shutdown we just want to stale the buffer and let the +* caller handle the error in bp->b_error appropriately. +* +* If the write was asynchronous then no one will be looking for the error. If +* this is the first failure of this type, clear the error state and write the +* buffer out again. This means we always retry an async write failure at least +* once, but we also need to set the buffer up to behave correctly now for +* repeated failures. +* +* If we get repeated async write failures, then we take action according to the +* error configuration we have been set up to use. +* +* Multi-state return value: +* +* XBF_IOERROR_FINISH: clear IO error retry state and run callback completions +* XBF_IOERROR_DONE: resubmitted immediately, do not run any completions +* XBF_IOERROR_FAIL: transient error, run failure callback completions and then
Quoted 8 lines
fs/xfs/xfs_buf_item.c xfs_buf_iodone_callback_error( · line 1086
Quoted 5 lines
+ * release the buffer
+ */fs/xfs/xfs_buf_item.c +3
-0+enum { +XBF_IOERROR_FINISH , +XBF_IOERROR_DONE ,I like the enum, but I have a hard time distinguishing what the difference is between FINISH and DONE based on the naming. I think
It was really just describing the action the caller needs to take. i.e. "buffer IO still needs finishing" vs "buffer IO is done, nothing more to do" vs "Buffer IO needs failure completion".
Quoted 3 lines
RESUBMIT would be more clear than DONE and perhaps have the resubmit in the caller, but then we'd have to duplicate that as well. Eh, perhaps it makes sense to defer such potential cleanups to the end.
Yeah, renaming just cascades the rename through multiple patches at this point. I'll take a note for later.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 13/30 xfs: handle buffer log item IO errors directly
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Currently when a buffer with attached log items has an IO error it called ->iop_error for each attched log item. These all call xfs_set_li_failed() to handle the error, but we are about to change the way log items manage buffers. hence we first need to remove the per-item dependency on buffer handling done by xfs_set_li_failed().
We already have specific buffer type IO completion routines, so move the log item error handling out of the generic error handling and into the log item specific functions so we can implement per-type error handling easily.
This requires a more complex return value from the error handling code so that we can take the correct action the failure handling requires. This results in some repeated boilerplate in the functions, but that can be cleaned up later once all the changes cascade through this code.
---
V2: fix buffer retry selection logic braino.
fs/xfs/xfs_buf_item.c | 214 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 144 insertions(+), 70 deletions(-)
fs/xfs/xfs_buf_item.c +144 -70
@@ -986,21 +986,24 @@ xfs_buf_do_callbacks_fail(spin_unlock (&ailp->ail_lock); } +/* +* Decide if we're going to retry the write after a failure, and prepare +* the buffer for retrying the write. +*/ static bool -xfs_buf_iodone_callback_error ( +xfs_buf_ioerror_fail_without_retry (struct xfs_buf *bp) {struct xfs_mount *mp = bp->b_mount;static ulong lasttime;static xfs_buftarg_t *lasttarg; -struct xfs_error_cfg *cfg;/* * If we've already decided to shutdown the filesystem because of * I/O errors, there's no point in giving this a retry. */ if (XFS_FORCED_SHUTDOWN (mp)) - goto out_stale; +return true;if (bp->b_target != lasttarg ||time_after (jiffies, (lasttime +5 *HZ ))) {
@@ -1011,91 +1014,114 @@ xfs_buf_iodone_callback_error(/* synchronous writes will have callers process the error */ if (!(bp->b_flags &XBF_ASYNC )) - goto out_stale; - -trace_xfs_buf_item_iodone_async (bp, _RET_IP_); - - cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +return true; +return false; +} -/* -* If the write was asynchronous then no one will be looking for the -* error. If this is the first failure of this type, clear the error -* state and write the buffer out again. This means we always retry an -* async write failure at least once, but we also need to set the buffer -* up to behave correctly now for repeated failures. -*/ -if (!(bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) || - bp->b_last_error != bp->b_error) { - bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); - bp->b_last_error = bp->b_error; -if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && - !bp->b_first_retry_time) - bp->b_first_retry_time = jiffies; +static bool +xfs_buf_ioerror_retry ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +if ((bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) && + bp->b_last_error == bp->b_error) +return false; -xfs_buf_ioerror (bp,0 ); -xfs_buf_submit (bp); -return true; - } + bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); + bp->b_last_error = bp->b_error; +if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && + !bp->b_first_retry_time) + bp->b_first_retry_time = jiffies; +return true; +} -/* -* Repeated failure on an async write. Take action according to the -* error configuration we have been set up to use. -*/ +/* +* Account for this latest trip around the retry handler, and decide if +* we've failed enough times to constitute a permanent failure. +*/ +static bool +xfs_buf_ioerror_permanent ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +struct xfs_mount *mp = bp->b_mount;if (cfg->max_retries !=XFS_ERR_RETRY_FOREVER && ++bp->b_retries > cfg->max_retries) - goto permanent_error; +return true;if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER &&time_after (jiffies, cfg->retry_timeout + bp->b_first_retry_time)) - goto permanent_error; +return true;/* At unmount we may treat errors differently */ if ((mp->m_flags &XFS_MOUNT_UNMOUNTING ) && mp->m_fail_unmount) - goto permanent_error; - -/* -* Still a transient error, run IO completion failure callbacks and let -* the higher layers retry the buffer. -*/ -xfs_buf_do_callbacks_fail (bp); -xfs_buf_ioerror (bp,0 ); -xfs_buf_relse (bp); -return true; +return true; -/* -* Permanent error - we need to trigger a shutdown if we haven't already -* to indicate that inconsistency will result from this action. -*/ -permanent_error: -xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); -out_stale: -xfs_buf_stale (bp); - bp->b_flags |=XBF_DONE ; -trace_xfs_buf_error_relse (bp, _RET_IP_);return false; } -static inline bool -xfs_buf_had_callback_errors ( +/* +* On a sync write or shutdown we just want to stale the buffer and let the +* caller handle the error in bp->b_error appropriately. +* +* If the write was asynchronous then no one will be looking for the error. If +* this is the first failure of this type, clear the error state and write the +* buffer out again. This means we always retry an async write failure at least +* once, but we also need to set the buffer up to behave correctly now for +* repeated failures. +* +* If we get repeated async write failures, then we take action according to the +* error configuration we have been set up to use. +* +* Multi-state return value: +* +* XBF_IOERROR_FINISH: clear IO error retry state and run callback completions +* XBF_IOERROR_DONE: resubmitted immediately, do not run any completions +* XBF_IOERROR_FAIL: transient error, run failure callback completions and then +* release the buffer +*/ +enum { +XBF_IOERROR_FINISH , +XBF_IOERROR_DONE , +XBF_IOERROR_FAIL , +}; + +static int +xfs_buf_iodone_error (struct xfs_buf *bp) { +struct xfs_mount *mp = bp->b_mount; +struct xfs_error_cfg *cfg; -/* -* If there is an error, process it. Some errors require us to run -* callbacks after failure processing is done so we detect that and take -* appropriate action. -*/ -if (bp->b_error &&xfs_buf_iodone_callback_error (bp)) -return true; +if (xfs_buf_ioerror_fail_without_retry (bp)) + goto out_stale; + +trace_xfs_buf_item_iodone_async (bp, _RET_IP_); + + cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +if (xfs_buf_ioerror_retry (bp, cfg)) { +xfs_buf_ioerror (bp,0 ); +xfs_buf_submit (bp); +return XBF_IOERROR_DONE ; + }/* -* Successful IO or permanent error. Either way, we can clear the -* retry state here in preparation for the next error that may occur. +* Permanent error - we need to trigger a shutdown if we haven't already +* to indicate that inconsistency will result from this action. */ - bp->b_last_error =0 ; - bp->b_retries =0 ; - bp->b_first_retry_time =0 ; -return false; +if (xfs_buf_ioerror_permanent (bp, cfg)) { +xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); + goto out_stale; + } + +/* Still considered a transient error. Caller will schedule retries. */ +return XBF_IOERROR_FAIL ; + +out_stale: +xfs_buf_stale (bp); + bp->b_flags |=XBF_DONE ; +trace_xfs_buf_error_relse (bp, _RET_IP_); +return XBF_IOERROR_FINISH ; } static void
@@ -1122,6 +1148,15 @@ xfs_buf_item_done(xfs_buf_rele (bp); } +static inline void +xfs_buf_clear_ioerror_retry_state ( +struct xfs_buf *bp) +{ + bp->b_last_error =0 ; + bp->b_retries =0 ; + bp->b_first_retry_time =0 ; +} +/* * Inode buffer iodone callback function. */
@@ -1129,9 +1164,22 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_iflush_done (bp);xfs_buf_ioend_finish (bp);
@@ -1144,9 +1192,22 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);/* a newly allocated dquot buffer might have a log item attached */ xfs_buf_item_done (bp);xfs_dquot_done (bp);
@@ -1163,9 +1224,22 @@ voidxfs_buf_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_buf_ioend_finish (bp); }
Re: [PATCH 13/30 V2] xfs: handle buffer log item IO errors directly
Quoted 20 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Currently when a buffer with attached log items has an IO error it called ->iop_error for each attched log item. These all call
xfs_set_li_failed()to handle the error, but we are about to change the way log items manage buffers. hence we first need to remove the per-item dependency on buffer handling done byxfs_set_li_failed().We already have specific buffer type IO completion routines, so move the log item error handling out of the generic error handling and into the log item specific functions so we can implement per-type error handling easily.
This requires a more complex return value from the error handling code so that we can take the correct action the failure handling requires. This results in some repeated boilerplate in the functions, but that can be cleaned up later once all the changes cascade through this code.
Quoted 4 lines
Signed-off-by Dave Chinner <[email protected]>---
V2: fix buffer retry selection logic braino.
Looks good:
Quoted 303 lines
fs/xfs/xfs_buf_item.c | 214 +++++++++++++++++++++++++++++++++----------------- 1 file changed, 144 insertions(+), 70 deletions(-)fs/xfs/xfs_buf_item.c +144
-70@@ -986,21 +986,24 @@ xfs_buf_do_callbacks_fail(spin_unlock (&ailp->ail_lock); } +/* +* Decide if we're going to retry the write after a failure, and prepare +* the buffer for retrying the write. +*/ static bool -xfs_buf_iodone_callback_error ( +xfs_buf_ioerror_fail_without_retry (struct xfs_buf *bp) {struct xfs_mount *mp = bp->b_mount;static ulong lasttime;static xfs_buftarg_t *lasttarg; -struct xfs_error_cfg *cfg;/* * If we've already decided to shutdown the filesystem because of * I/O errors, there's no point in giving this a retry. */ if (XFS_FORCED_SHUTDOWN (mp)) - goto out_stale; +return true;if (bp->b_target != lasttarg ||time_after (jiffies, (lasttime +5 *HZ ))) {@@ -1011,91 +1014,114 @@ xfs_buf_iodone_callback_error(/* synchronous writes will have callers process the error */ if (!(bp->b_flags &XBF_ASYNC )) - goto out_stale; - -trace_xfs_buf_item_iodone_async (bp, _RET_IP_); - - cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +return true; +return false; +} -/* -* If the write was asynchronous then no one will be looking for the -* error. If this is the first failure of this type, clear the error -* state and write the buffer out again. This means we always retry an -* async write failure at least once, but we also need to set the buffer -* up to behave correctly now for repeated failures. -*/ -if (!(bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) || - bp->b_last_error != bp->b_error) { - bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); - bp->b_last_error = bp->b_error; -if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && - !bp->b_first_retry_time) - bp->b_first_retry_time = jiffies; +static bool +xfs_buf_ioerror_retry ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +if ((bp->b_flags & (XBF_STALE |XBF_WRITE_FAIL )) && + bp->b_last_error == bp->b_error) +return false; -xfs_buf_ioerror (bp,0 ); -xfs_buf_submit (bp); -return true; - } + bp->b_flags |= (XBF_WRITE |XBF_DONE |XBF_WRITE_FAIL ); + bp->b_last_error = bp->b_error; +if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER && + !bp->b_first_retry_time) + bp->b_first_retry_time = jiffies; +return true; +} -/* -* Repeated failure on an async write. Take action according to the -* error configuration we have been set up to use. -*/ +/* +* Account for this latest trip around the retry handler, and decide if +* we've failed enough times to constitute a permanent failure. +*/ +static bool +xfs_buf_ioerror_permanent ( +struct xfs_buf *bp, +struct xfs_error_cfg *cfg) +{ +struct xfs_mount *mp = bp->b_mount;if (cfg->max_retries !=XFS_ERR_RETRY_FOREVER && ++bp->b_retries > cfg->max_retries) - goto permanent_error; +return true;if (cfg->retry_timeout !=XFS_ERR_RETRY_FOREVER &&time_after (jiffies, cfg->retry_timeout + bp->b_first_retry_time)) - goto permanent_error; +return true;/* At unmount we may treat errors differently */ if ((mp->m_flags &XFS_MOUNT_UNMOUNTING ) && mp->m_fail_unmount) - goto permanent_error; - -/* -* Still a transient error, run IO completion failure callbacks and let -* the higher layers retry the buffer. -*/ -xfs_buf_do_callbacks_fail (bp); -xfs_buf_ioerror (bp,0 ); -xfs_buf_relse (bp); -return true; +return true; -/* -* Permanent error - we need to trigger a shutdown if we haven't already -* to indicate that inconsistency will result from this action. -*/ -permanent_error: -xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); -out_stale: -xfs_buf_stale (bp); - bp->b_flags |=XBF_DONE ; -trace_xfs_buf_error_relse (bp, _RET_IP_);return false; } -static inline bool -xfs_buf_had_callback_errors ( +/* +* On a sync write or shutdown we just want to stale the buffer and let the +* caller handle the error in bp->b_error appropriately. +* +* If the write was asynchronous then no one will be looking for the error. If +* this is the first failure of this type, clear the error state and write the +* buffer out again. This means we always retry an async write failure at least +* once, but we also need to set the buffer up to behave correctly now for +* repeated failures. +* +* If we get repeated async write failures, then we take action according to the +* error configuration we have been set up to use. +* +* Multi-state return value: +* +* XBF_IOERROR_FINISH: clear IO error retry state and run callback completions +* XBF_IOERROR_DONE: resubmitted immediately, do not run any completions +* XBF_IOERROR_FAIL: transient error, run failure callback completions and then +* release the buffer +*/ +enum { +XBF_IOERROR_FINISH , +XBF_IOERROR_DONE , +XBF_IOERROR_FAIL , +}; + +static int +xfs_buf_iodone_error (struct xfs_buf *bp) { +struct xfs_mount *mp = bp->b_mount; +struct xfs_error_cfg *cfg; -/* -* If there is an error, process it. Some errors require us to run -* callbacks after failure processing is done so we detect that and take -* appropriate action. -*/ -if (bp->b_error &&xfs_buf_iodone_callback_error (bp)) -return true; +if (xfs_buf_ioerror_fail_without_retry (bp)) + goto out_stale; + +trace_xfs_buf_item_iodone_async (bp, _RET_IP_); + + cfg =xfs_error_get_cfg (mp,XFS_ERR_METADATA , bp->b_error); +if (xfs_buf_ioerror_retry (bp, cfg)) { +xfs_buf_ioerror (bp,0 ); +xfs_buf_submit (bp); +return XBF_IOERROR_DONE ; + }/* -* Successful IO or permanent error. Either way, we can clear the -* retry state here in preparation for the next error that may occur. +* Permanent error - we need to trigger a shutdown if we haven't already +* to indicate that inconsistency will result from this action. */ - bp->b_last_error =0 ; - bp->b_retries =0 ; - bp->b_first_retry_time =0 ; -return false; +if (xfs_buf_ioerror_permanent (bp, cfg)) { +xfs_force_shutdown (mp,SHUTDOWN_META_IO_ERROR ); + goto out_stale; + } + +/* Still considered a transient error. Caller will schedule retries. */ +return XBF_IOERROR_FAIL ; + +out_stale: +xfs_buf_stale (bp); + bp->b_flags |=XBF_DONE ; +trace_xfs_buf_error_relse (bp, _RET_IP_); +return XBF_IOERROR_FINISH ; } static void@@ -1122,6 +1148,15 @@ xfs_buf_item_done(xfs_buf_rele (bp); } +static inline void +xfs_buf_clear_ioerror_retry_state ( +struct xfs_buf *bp) +{ + bp->b_last_error =0 ; + bp->b_retries =0 ; + bp->b_first_retry_time =0 ; +} +/* * Inode buffer iodone callback function. */ @@ -1129,9 +1164,22 @@ voidxfs_buf_inode_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_iflush_done (bp);xfs_buf_ioend_finish (bp);@@ -1144,9 +1192,22 @@ voidxfs_buf_dquot_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);/* a newly allocated dquot buffer might have a log item attached */ xfs_buf_item_done (bp);xfs_dquot_done (bp);@@ -1163,9 +1224,22 @@ voidxfs_buf_iodone ( struct xfs_buf *bp) { -if (xfs_buf_had_callback_errors (bp)) +if (bp->b_error) { +int ret =xfs_buf_iodone_error (bp); + +if (ret ==XBF_IOERROR_FINISH ) + goto finish_iodone; +if (ret ==XBF_IOERROR_DONE ) +return ; +ASSERT (ret ==XBF_IOERROR_FAIL ); +xfs_buf_do_callbacks_fail (bp); +xfs_buf_ioerror (bp,0 ); +xfs_buf_relse (bp);return ; + } +finish_iodone: +xfs_buf_clear_ioerror_retry_state (bp);xfs_buf_item_done (bp);xfs_buf_ioend_finish (bp); }
PATCH 14/30 xfs: unwind log item error flagging
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When an buffer IO error occurs, we want to mark all
the log items attached to the buffer as failed. Open code the error handling loop so that we can modify the flagging for the different types of objects directly and independently of each other.
This also allows us to remove the ->iop_error method from the log item operations.
--- fs/xfs/xfs_buf_item.c | 48 ++++++++++++----------------------------- fs/xfs/xfs_dquot_item.c | 18 ---------------- fs/xfs/xfs_inode_item.c | 18 ---------------- fs/xfs/xfs_trans.h | 1 - 4 files changed, 14 insertions(+), 71 deletions(-)
fs/xfs/xfs_buf_item.c +14 -34
@@ -12,6 +12,7 @@#include "xfs_bit.h" #include "xfs_mount.h" #include "xfs_trans.h" +#include "xfs_trans_priv.h" #include "xfs_buf_item.h" #include "xfs_inode.h" #include "xfs_inode_item.h"
@@ -955,37 +956,6 @@ xfs_buf_item_relse(xfs_buf_item_free (bip); } -/* -* Invoke the error state callback for each log item affected by the failed I/O. -* -* If a metadata buffer write fails with a non-permanent error, the buffer is -* eventually resubmitted and so the completion callbacks are not run. The error -* state may need to be propagated to the log items attached to the buffer, -* however, so the next AIL push of the item knows hot to handle it correctly. -*/ -STATIC void -xfs_buf_do_callbacks_fail ( -struct xfs_buf *bp) -{ -struct xfs_ail *ailp = bp->b_mount->m_ail; -struct xfs_log_item *lip; - -/* -* Buffer log item errors are handled directly by xfs_buf_item_push() -* and xfs_buf_iodone_callback_error, and they have no IO error -* callbacks. Check only for items in b_li_list. -*/ -if (list_empty (&bp->b_li_list)) -return ; - -spin_lock (&ailp->ail_lock); -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { -if (lip->li_ops->iop_error) - lip->li_ops->iop_error (lip, bp); - } -spin_unlock (&ailp->ail_lock); -} -/* * Decide if we're going to retry the write after a failure, and prepare * the buffer for retrying the write.
@@ -1166,6 +1136,7 @@ xfs_buf_inode_iodone( struct xfs_buf *bp) {if (bp->b_error) { +struct xfs_log_item *lip;int ret =xfs_buf_iodone_error (bp);if (ret ==XBF_IOERROR_FINISH )
@@ -1173,7 +1144,11 @@ xfs_buf_inode_iodone(if (ret ==XBF_IOERROR_DONE )return ;ASSERT (ret ==XBF_IOERROR_FAIL ); -xfs_buf_do_callbacks_fail (bp); +spin_lock (&bp->b_mount->m_ail->ail_lock); +list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { +xfs_set_li_failed (lip, bp); + } +spin_unlock (&bp->b_mount->m_ail->ail_lock);xfs_buf_ioerror (bp,0 );xfs_buf_relse (bp);return ;
@@ -1194,6 +1169,7 @@ xfs_buf_dquot_iodone( struct xfs_buf *bp) {if (bp->b_error) { +struct xfs_log_item *lip;int ret =xfs_buf_iodone_error (bp);if (ret ==XBF_IOERROR_FINISH )
@@ -1201,7 +1177,11 @@ xfs_buf_dquot_iodone(if (ret ==XBF_IOERROR_DONE )return ;ASSERT (ret ==XBF_IOERROR_FAIL ); -xfs_buf_do_callbacks_fail (bp); +spin_lock (&bp->b_mount->m_ail->ail_lock); +list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { +xfs_set_li_failed (lip, bp); + } +spin_unlock (&bp->b_mount->m_ail->ail_lock);xfs_buf_ioerror (bp,0 );xfs_buf_relse (bp);return ;
@@ -1233,7 +1213,7 @@ xfs_buf_iodone(if (ret ==XBF_IOERROR_DONE )return ;ASSERT (ret ==XBF_IOERROR_FAIL ); -xfs_buf_do_callbacks_fail (bp); +ASSERT (list_empty (&bp->b_li_list));xfs_buf_ioerror (bp,0 );xfs_buf_relse (bp);return ;
fs/xfs/xfs_dquot_item.c +0 -18
@@ -113,23 +113,6 @@ xfs_qm_dqunpin_wait(wait_event (dqp->q_pinwait, (atomic_read (&dqp->q_pincount) ==0 )); } -/* -* Callback used to mark a buffer with XFS_LI_FAILED when items in the buffer -* have been failed during writeback -* -* this informs the AIL that the dquot is already flush locked on the next push, -* and acquires a hold on the buffer to ensure that it isn't reclaimed before -* dirty data makes it to disk. -*/ -STATIC void -xfs_dquot_item_error ( -struct xfs_log_item *lip, -struct xfs_buf *bp) -{ -ASSERT (!completion_done (&DQUOT_ITEM (lip)->qli_dquot->q_flush)); -xfs_set_li_failed (lip, bp); -} - STATIC uint xfs_qm_dquot_logitem_push( struct xfs_log_item *lip,
@@ -216,7 +199,6 @@ static const struct xfs_item_ops xfs_dquot_item_ops = { .iop_release = xfs_qm_dquot_logitem_release, .iop_committing = xfs_qm_dquot_logitem_committing, .iop_push = xfs_qm_dquot_logitem_push, - .iop_error = xfs_dquot_item_error };/*
fs/xfs/xfs_inode_item.c +0 -18
@@ -464,23 +464,6 @@ xfs_inode_item_unpin(wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT); } -/* -* Callback used to mark a buffer with XFS_LI_FAILED when items in the buffer -* have been failed during writeback -* -* This informs the AIL that the inode is already flush locked on the next push, -* and acquires a hold on the buffer to ensure that it isn't reclaimed before -* dirty data makes it to disk. -*/ -STATIC void -xfs_inode_item_error ( -struct xfs_log_item *lip, -struct xfs_buf *bp) -{ -ASSERT (xfs_isiflocked (INODE_ITEM (lip)->ili_inode)); -xfs_set_li_failed (lip, bp); -} - STATIC uint xfs_inode_item_push( struct xfs_log_item *lip,
@@ -619,7 +602,6 @@ static const struct xfs_item_ops xfs_inode_item_ops = { .iop_committed = xfs_inode_item_committed, .iop_push = xfs_inode_item_push, .iop_committing = xfs_inode_item_committing, - .iop_error = xfs_inode_item_error };
fs/xfs/xfs_trans.h +0 -1
@@ -74,7 +74,6 @@ struct xfs_item_ops {void (*iop_committing)(struct xfs_log_item *,xfs_lsn_t commit_lsn);void (*iop_release)(struct xfs_log_item *);xfs_lsn_t (*iop_committed)(struct xfs_log_item *,xfs_lsn_t ); -void (*iop_error)(struct xfs_log_item *,xfs_buf_t *);int (*iop_recover)(struct xfs_log_item *lip,struct xfs_trans *tp);bool (*iop_match)(struct xfs_log_item *item,uint64_t id); };
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 15/30 xfs: move xfs_clear_li_failed out of xfs_ail_delete_one()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
xfs_ail_delete_one() is called directly from dquot and inode IO completion, as well as from the generic xfs_trans_ail_delete() function. Inodes are about to have their own failure handling, and dquots will in future, too. Pull the clearing of the LI_FAILED flag up into the callers so we can customise the code appropriately.
--- fs/xfs/xfs_dquot.c | 6 +----- fs/xfs/xfs_inode_item.c | 3 +-- fs/xfs/xfs_trans_ail.c | 2 +- 3 files changed, 3 insertions(+), 8 deletions(-)
fs/xfs/xfs_dquot.c +1 -5
@@ -1070,16 +1070,12 @@ xfs_qm_dqflush_done( test_bit(XFS_LI_FAILED, &lip->li_flags))) {spin_lock (&ailp->ail_lock); +xfs_clear_li_failed (lip);if (lip->li_lsn == qip->qli_flush_lsn) {/* xfs_ail_update_finish() drops the AIL lock */ tail_lsn =xfs_ail_delete_one (ailp, lip);xfs_ail_update_finish (ailp, tail_lsn); }else { -/* -* Clear the failed state since we are about to drop the -* flush lock -*/ -xfs_clear_li_failed (lip);spin_unlock (&ailp->ail_lock); } }
fs/xfs/xfs_inode_item.c +1 -2
@@ -690,12 +690,11 @@ xfs_iflush_done(/* this is an opencoded batch version of xfs_trans_ail_delete */ spin_lock (&ailp->ail_lock);list_for_each_entry (lip, &tmp, li_bio_list) { +xfs_clear_li_failed (lip);if (lip->li_lsn ==INODE_ITEM (lip)->ili_flush_lsn) {xfs_lsn_t lsn =xfs_ail_delete_one (ailp, lip);if (!tail_lsn && lsn) tail_lsn = lsn; - }else { -xfs_clear_li_failed (lip); } }xfs_ail_update_finish (ailp, tail_lsn);
fs/xfs/xfs_trans_ail.c +1 -1
@@ -843,7 +843,6 @@ xfs_ail_delete_one(trace_xfs_ail_delete (lip, mlip->li_lsn, lip->li_lsn);xfs_ail_delete (ailp, lip); -xfs_clear_li_failed (lip);clear_bit (XFS_LI_IN_AIL , &lip->li_flags); lip->li_lsn =0 ;
@@ -874,6 +873,7 @@ xfs_trans_ail_delete( }/* xfs_ail_update_finish() drops the AIL lock */ +xfs_clear_li_failed (lip); tail_lsn =xfs_ail_delete_one (ailp, lip);xfs_ail_update_finish (ailp, tail_lsn); }
Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 16/30 xfs: pin inode backing buffer to the inode log item
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When we dirty an inode, we are going to have to write it disk at some point in the near future. This requires the inode cluster backing buffer to be present in memory. Unfortunately, under severe memory pressure we can reclaim the inode backing buffer while the inode is dirty in memory, resulting in stalling the AIL pushing because it has to do a read-modify-write cycle on the cluster buffer.
When we have no memory available, the read of the cluster buffer blocks the AIL pushing process, and this causes all sorts of issues for memory reclaim as it requires inode writeback to make forwards progress. Allocating a cluster buffer causes more memory pressure, and results in more cluster buffers to be reclaimed, resulting in more RMW cycles to be done in the AIL context and everything then backs up on AIL progress. Only the synchronous inode cluster writeback in the the inode reclaim code provides some level of forwards progress guarantees that prevent OOM-killer rampages in this situation.
Fix this by pinning the inode backing buffer to the inode log item when the inode is first dirtied (i.e. in xfs_trans_log_inode()). This may mean the first modification of an inode that has been held in cache for a long time may block on a cluster buffer read, but we can do that in transaction context and block safely until the buffer has been allocated and read.
Once we have the cluster buffer, the inode log item takes a reference to it, pinning it in memory, and attaches it to the log item for future reference. This means we can always grab the cluster buffer from the inode log item when we need it.
When the inode is finally cleaned and removed from the AIL, we can drop the reference the inode log item holds on the cluster buffer. Once all inodes on the cluster buffer are clean, the cluster buffer will be unpinned and it will be available for memory reclaim to reclaim again.
This avoids the issues with needing to do RMW cycles in the AIL pushing context, and hence allows complete non-blocking inode flushing to be performed by the AIL pushing context.
--- fs/xfs/libxfs/xfs_inode_buf.c | 3 +- fs/xfs/libxfs/xfs_trans_inode.c | 53 ++++++++++++++++++++++++---- fs/xfs/xfs_buf_item.c | 4 +-- fs/xfs/xfs_inode_item.c | 61 ++++++++++++++++++++++++++------- fs/xfs/xfs_trans_ail.c | 8 +++-- 5 files changed, 105 insertions(+), 24 deletions(-)
fs/xfs/libxfs/xfs_inode_buf.c +2 -1
@@ -176,7 +176,8 @@ xfs_imap_to_bp( } *bpp = bp; - *dipp =xfs_buf_offset (bp, imap->im_boffset); +if (dipp) + *dipp =xfs_buf_offset (bp, imap->im_boffset);return 0 ; }
fs/xfs/libxfs/xfs_trans_inode.c +47 -6
72 lines not on this page · raw message
fs/xfs/xfs_buf_item.c +1 -3
@@ -1144,11 +1144,9 @@ xfs_buf_inode_iodone(if (ret ==XBF_IOERROR_DONE )return ;ASSERT (ret ==XBF_IOERROR_FAIL ); -spin_lock (&bp->b_mount->m_ail->ail_lock);list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { -xfs_set_li_failed (lip, bp); +set_bit (XFS_LI_FAILED , &lip->li_flags); } -spin_unlock (&bp->b_mount->m_ail->ail_lock);xfs_buf_ioerror (bp,0 );xfs_buf_relse (bp);return ;
fs/xfs/xfs_inode_item.c +49 -12
133 lines not on this page · raw message
fs/xfs/xfs_trans_ail.c +6 -2
@@ -377,8 +377,12 @@ xfsaild_resubmit_item( }/* protected by ail_lock */ -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) -xfs_clear_li_failed (lip); +list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { +if (bp->b_flags & _XBF_INODES) +clear_bit (XFS_LI_FAILED , &lip->li_flags); +else +xfs_clear_li_failed (lip); + }xfs_buf_unlock (bp);return XFS_ITEM_SUCCESS ;
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 16/30] xfs: pin inode backing buffer to the inode log item
Quoted 39 lines · Dave Chinner
From: Dave Chinner <[email protected]>
When we dirty an inode, we are going to have to write it disk at some point in the near future. This requires the inode cluster backing buffer to be present in memory. Unfortunately, under severe memory pressure we can reclaim the inode backing buffer while the inode is dirty in memory, resulting in stalling the AIL pushing because it has to do a read-modify-write cycle on the cluster buffer.
When we have no memory available, the read of the cluster buffer blocks the AIL pushing process, and this causes all sorts of issues for memory reclaim as it requires inode writeback to make forwards progress. Allocating a cluster buffer causes more memory pressure, and results in more cluster buffers to be reclaimed, resulting in more RMW cycles to be done in the AIL context and everything then backs up on AIL progress. Only the synchronous inode cluster writeback in the the inode reclaim code provides some level of forwards progress guarantees that prevent OOM-killer rampages in this situation.
Fix this by pinning the inode backing buffer to the inode log item when the inode is first dirtied (i.e. in
xfs_trans_log_inode()). This may mean the first modification of an inode that has been held in cache for a long time may block on a cluster buffer read, but we can do that in transaction context and block safely until the buffer has been allocated and read.Once we have the cluster buffer, the inode log item takes a reference to it, pinning it in memory, and attaches it to the log item for future reference. This means we can always grab the cluster buffer from the inode log item when we need it.
When the inode is finally cleaned and removed from the AIL, we can drop the reference the inode log item holds on the cluster buffer. Once all inodes on the cluster buffer are clean, the cluster buffer will be unpinned and it will be available for memory reclaim to reclaim again.
Quoted 6 lines
This avoids the issues with needing to do RMW cycles in the AIL pushing context, and hence allows complete non-blocking inode flushing to be performed by the AIL pushing context.
Signed-off-by Dave Chinner <[email protected]>---
More clear with the spurious bits removed, thanks. I'm still not clear on the ili_flush_lsn reset bit. It seems a little strange that we'd clear it and reset it in cases where flush completions race with relogs (and thus AIL moving), but it shouldn't cause any issues from what I can tell:
Quoted 285 lines
fs/xfs/libxfs/xfs_inode_buf.c | 3 +- fs/xfs/libxfs/xfs_trans_inode.c | 53 ++++++++++++++++++++++++---- fs/xfs/xfs_buf_item.c | 4 +-- fs/xfs/xfs_inode_item.c | 61 ++++++++++++++++++++++++++------- fs/xfs/xfs_trans_ail.c | 8 +++-- 5 files changed, 105 insertions(+), 24 deletions(-)fs/xfs/libxfs/xfs_inode_buf.c +2
-1@@ -176,7 +176,8 @@ xfs_imap_to_bp( } *bpp = bp; - *dipp =xfs_buf_offset (bp, imap->im_boffset); +if (dipp) + *dipp =xfs_buf_offset (bp, imap->im_boffset);return 0 ; }fs/xfs/libxfs/xfs_trans_inode.c +47
-6@@ -8,6 +8,8 @@#include "xfs_shared.h" #include "xfs_format.h" #include "xfs_log_format.h" +#include "xfs_trans_resv.h" +#include "xfs_mount.h" #include "xfs_inode.h" #include "xfs_trans.h" #include "xfs_trans_priv.h" @@ -72,13 +74,19 @@ xfs_trans_ichgtime( }/* -* This is called to mark the fields indicated in fieldmask as needing -* to be logged when the transaction is committed. The inode must -* already be associated with the given transaction. +* This is called to mark the fields indicated in fieldmask as needing to be +* logged when the transaction is committed. The inode must already be +* associated with the given transaction. * -* The values for fieldmask are defined in xfs_inode_item.h. We always -* log all of the core inode if any of it has changed, and we always log -* all of the inline data/extents/b-tree root if any of them has changed. +* The values for fieldmask are defined in xfs_inode_item.h. We always log all +* of the core inode if any of it has changed, and we always log all of the +* inline data/extents/b-tree root if any of them has changed. +* +* Grab and pin the cluster buffer associated with this inode to avoid RMW +* cycles at inode writeback time. Avoid the need to add error handling to every +* xfs_trans_log_inode() call by shutting down on read error. This will cause +* transactions to fail and everything to error out, just like if we return a +* read error in a dirty transaction and cancel it. */ void xfs_trans_log_inode(@@ -131,6 +139,39 @@ xfs_trans_log_inode(spin_lock (&iip->ili_lock); iip->ili_fsync_fields |= flags; +if (!iip->ili_item.li_buf) { +struct xfs_buf *bp; +int error; + +/* +* We hold the ILOCK here, so this inode is not going to be +* flushed while we are here. Further, because there is no +* buffer attached to the item, we know that there is no IO in +* progress, so nothing will clear the ili_fields while we read +* in the buffer. Hence we can safely drop the spin lock and +* read the buffer knowing that the state will not change from +* here. +*/ +spin_unlock (&iip->ili_lock); + error =xfs_imap_to_bp (ip->i_mount, tp, &ip->i_imap,NULL , + &bp,0 ); +if (error) { +xfs_force_shutdown (ip->i_mount,SHUTDOWN_META_IO_ERROR ); +return ; + } + +/* +* We need an explicit buffer reference for the log item but +* don't want the buffer to remain attached to the transaction. +* Hold the buffer but release the transaction reference. +*/ +xfs_buf_hold (bp); +xfs_trans_brelse (tp, bp); + +spin_lock (&iip->ili_lock); + iip->ili_item.li_buf = bp; + } +/* * Always OR in the bits from the ili_last_fields field. This is to * coordinate with the xfs_iflush() and xfs_iflush_done() routines in fs/xfs/xfs_buf_item.c +1
-3@@ -1144,11 +1144,9 @@ xfs_buf_inode_iodone(if (ret ==XBF_IOERROR_DONE )return ;ASSERT (ret ==XBF_IOERROR_FAIL ); -spin_lock (&bp->b_mount->m_ail->ail_lock);list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { -xfs_set_li_failed (lip, bp); +set_bit (XFS_LI_FAILED , &lip->li_flags); } -spin_unlock (&bp->b_mount->m_ail->ail_lock);xfs_buf_ioerror (bp,0 );xfs_buf_relse (bp);return ;fs/xfs/xfs_inode_item.c +49
-12@@ -439,6 +439,7 @@ xfs_inode_item_pin(struct xfs_inode *ip =INODE_ITEM (lip)->ili_inode;ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL )); +ASSERT (lip->li_buf);trace_xfs_inode_pin (ip, _RET_IP_);atomic_inc (&ip->i_pincount);@@ -450,6 +451,12 @@ xfs_inode_item_pin(* item which was previously pinned with a call to xfs_inode_item_pin(). * * Also wake up anyone in xfs_iunpin_wait() if the count goes to 0. +* +* Note that unpin can race with inode cluster buffer freeing marking the buffer +* stale. In that case, flush completions are run from the buffer unpin call, +* which may happen before the inode is unpinned. If we lose the race, there +* will be no buffer attached to the log item, but the inode will be marked +* XFS_ISTALE. */ STATIC void xfs_inode_item_unpin(@@ -459,6 +466,7 @@ xfs_inode_item_unpin(struct xfs_inode *ip =INODE_ITEM (lip)->ili_inode;trace_xfs_inode_unpin (ip, _RET_IP_); +ASSERT (lip->li_buf ||xfs_iflags_test (ip,XFS_ISTALE ));ASSERT (atomic_read (&ip->i_pincount) >0 );if (atomic_dec_and_test (&ip->i_pincount))wake_up_bit (&ip->i_flags, __XFS_IPINNED_BIT);@@ -629,10 +637,15 @@ xfs_inode_item_init(*/ void xfs_inode_item_destroy ( -xfs_inode_t *ip) +struct xfs_inode *ip) { -kmem_free (ip->i_itemp->ili_item.li_lv_shadow); -kmem_cache_free (xfs_ili_zone, ip->i_itemp); +struct xfs_inode_log_item *iip = ip->i_itemp; + +ASSERT (iip->ili_item.li_buf ==NULL ); + + ip->i_itemp =NULL ; +kmem_free (iip->ili_item.li_lv_shadow); +kmem_cache_free (xfs_ili_zone, iip); }@@ -673,11 +686,10 @@ xfs_iflush_done(list_move_tail (&lip->li_bio_list, &tmp);/* Do an unlocked check for needing the AIL lock. */ -if (lip->li_lsn == iip->ili_flush_lsn || +if (iip->ili_flush_lsn == lip->li_lsn ||test_bit (XFS_LI_FAILED , &lip->li_flags)) need_ail++; } -ASSERT (list_empty (&bp->b_li_list));/* * We only want to pull the item from the AIL if it is actually there @@ -690,7 +702,7 @@ xfs_iflush_done(/* this is an opencoded batch version of xfs_trans_ail_delete */ spin_lock (&ailp->ail_lock);list_for_each_entry (lip, &tmp, li_bio_list) { -xfs_clear_li_failed (lip); +clear_bit (XFS_LI_FAILED , &lip->li_flags);if (lip->li_lsn ==INODE_ITEM (lip)->ili_flush_lsn) {xfs_lsn_t lsn =xfs_ail_delete_one (ailp, lip);if (!tail_lsn && lsn)@@ -706,14 +718,29 @@ xfs_iflush_done(* them is safely on disk. */ list_for_each_entry_safe (lip, n, &tmp, li_bio_list) { +bool drop_buffer = false; +list_del_init (&lip->li_bio_list); iip =INODE_ITEM (lip);spin_lock (&iip->ili_lock); + +/* +* Remove the reference to the cluster buffer if the inode is +* clean in memory. Drop the buffer reference once we've dropped +* the locks we hold. +*/ +ASSERT (iip->ili_item.li_buf == bp); +if (!iip->ili_fields) { + iip->ili_item.li_buf =NULL ; + drop_buffer = true; + } iip->ili_last_fields =0 ; + iip->ili_flush_lsn =0 ;spin_unlock (&iip->ili_lock); -xfs_ifunlock (iip->ili_inode); +if (drop_buffer) +xfs_buf_rele (bp); } }@@ -725,12 +752,20 @@ xfs_iflush_done(*/ void xfs_iflush_abort ( -struct xfs_inode *ip) +struct xfs_inode *ip) { -struct xfs_inode_log_item *iip = ip->i_itemp; +struct xfs_inode_log_item *iip = ip->i_itemp; +struct xfs_buf *bp =NULL ;if (iip) { +/* +* Clear the failed bit before removing the item from the AIL so +* xfs_trans_ail_delete() doesn't try to clear and release the +* buffer attached to the log item before we are done with it. +*/ +clear_bit (XFS_LI_FAILED , &iip->ili_item.li_flags);xfs_trans_ail_delete (&iip->ili_item,0 ); +/* * Clear the inode logging fields so no more flushes are * attempted. @@ -739,12 +774,14 @@ xfs_iflush_abort( iip->ili_last_fields =0 ; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ; + iip->ili_flush_lsn =0 ; + bp = iip->ili_item.li_buf; + iip->ili_item.li_buf =NULL ;spin_unlock (&iip->ili_lock); } -/* -* Release the inode's flush lock since we're done with it. -*/ xfs_ifunlock (ip); +if (bp) +xfs_buf_rele (bp); }/* fs/xfs/xfs_trans_ail.c +6
-2@@ -377,8 +377,12 @@ xfsaild_resubmit_item( }/* protected by ail_lock */ -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) -xfs_clear_li_failed (lip); +list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { +if (bp->b_flags & _XBF_INODES) +clear_bit (XFS_LI_FAILED , &lip->li_flags); +else +xfs_clear_li_failed (lip); + }xfs_buf_unlock (bp);return XFS_ITEM_SUCCESS ;Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 17/30 xfs: make inode reclaim almost non-blocking
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that dirty inode writeback doesn't cause read-modify-write cycles on the inode cluster buffer under memory pressure, the need to throttle memory reclaim to the rate at which we can clean dirty inodes goes away. That is due to the fact that we no longer thrash inode cluster buffers under memory pressure to clean dirty inodes.
This means inode writeback no longer stalls on memory allocation or read IO, and hence can be done asynchronously without generating memory pressure. As a result, blocking inode writeback in reclaim is no longer necessary to prevent reclaim priority windup as cleaning dirty inodes is no longer dependent on having memory reserves available for the filesystem to make progress reclaiming inodes.
Hence we can convert inode reclaim to be non-blocking for shrinker callouts, both for direct reclaim and kswapd.
On a vanilla kernel, running a 16-way fsmark create workload on a 4 node/16p/16GB RAM machine, I can reliably pin 14.75GB of RAM via userspace mlock(). The OOM killer gets invoked at 15GB of pinned RAM.
With this patch alone, pinning memory triggers premature OOM killer invocation, sometimes with as much as 45% of RAM being free. It's trivially easy to trigger the OOM killer when reclaim does not block.
With pinning inode clusters in RAM and then adding this patch, I can reliably pin 14.5GB of RAM and still have the fsmark workload run to completion. The OOM killer gets invoked 14.75GB of pinned RAM, which is only a small amount of memory less than the vanilla kernel. It is much more reliable than just with async reclaim alone.
simoops shows that allocation stalls go away when async reclaim is used. Vanilla kernel:
Run time: 1924 seconds
Read latency (p50: 3,305,472) (p95: 3,723,264) (p99: 4,001,792) Write latency (p50: 184,064) (p95: 553,984) (p99: 807,936) Allocation latency (p50: 2,641,920) (p95: 3,911,680) (p99: 4,464,640) work rate = 13.45/sec (avg 13.44/sec) (p50: 13.46) (p95: 13.58) (p99: 13.70) alloc stall rate = 3.80/sec (avg: 2.59) (p50: 2.54) (p95: 2.96) (p99: 3.02)
With inode cluster pinning and async reclaim:
Run time: 1924 seconds
Read latency (p50: 3,305,472) (p95: 3,715,072) (p99: 3,977,216) Write latency (p50: 187,648) (p95: 553,984) (p99: 789,504) Allocation latency (p50: 2,748,416) (p95: 3,919,872) (p99: 4,448,256) work rate = 13.28/sec (avg 13.32/sec) (p50: 13.26) (p95: 13.34) (p99: 13.34) alloc stall rate = 0.02/sec (avg: 0.02) (p50: 0.01) (p95: 0.03) (p99: 0.03)
Latencies don't really change much, nor does the work rate. However, allocation almost never stalls with these changes, whilst the vanilla kernel is sometimes reporting 20 stalls/s over a 60s sample period. This difference is due to inode reclaim being largely non-blocking now.
IOWs, once we have pinned inode cluster buffers, we can make inode reclaim non-blocking without a major risk of premature and/or spurious OOM killer invocation, and without any changes to memory reclaim infrastructure.
--- fs/xfs/xfs_icache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/xfs/xfs_icache.c +1 -1
@@ -1402,7 +1402,7 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK |SYNC_WAIT , &nr_to_scan); +return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); }/*
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 17/30] xfs: make inode reclaim almost non-blocking
Quoted 23 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that dirty inode writeback doesn't cause read-modify-write cycles on the inode cluster buffer under memory pressure, the need to throttle memory reclaim to the rate at which we can clean dirty inodes goes away. That is due to the fact that we no longer thrash inode cluster buffers under memory pressure to clean dirty inodes.
This means inode writeback no longer stalls on memory allocation or read IO, and hence can be done asynchronously without generating memory pressure. As a result, blocking inode writeback in reclaim is no longer necessary to prevent reclaim priority windup as cleaning dirty inodes is no longer dependent on having memory reserves available for the filesystem to make progress reclaiming inodes.
Hence we can convert inode reclaim to be non-blocking for shrinker callouts, both for direct reclaim and kswapd.
On a vanilla kernel, running a 16-way fsmark create workload on a 4 node/16p/16GB RAM machine, I can reliably pin 14.75GB of RAM via userspace
mlock(). The OOM killer gets invoked at 15GB of pinned RAM.
Quoted 5 lines
With this patch alone, pinning memory triggers premature OOM killer invocation, sometimes with as much as 45% of RAM being free. It's trivially easy to trigger the OOM killer when reclaim does not block.
Nit, but I had to reread this a couple times to grok what "With this patch alone ..." meant. I think it means "With non-blocking inode reclaim without cluster buffer pinning ...," based on the next paragraph. If so, I think the explicit wording is a little more clear.
Quoted 31 lines
With pinning inode clusters in RAM and then adding this patch, I can reliably pin 14.5GB of RAM and still have the fsmark workload run to completion. The OOM killer gets invoked 14.75GB of pinned RAM, which is only a small amount of memory less than the vanilla kernel. It is much more reliable than just with async reclaim alone.
simoops shows that allocation stalls go away when async reclaim is used. Vanilla kernel:
Run time: 1924 seconds
Read latency (p50: 3,305,472) (p95: 3,723,264) (p99: 4,001,792) Write latency (p50: 184,064) (p95: 553,984) (p99: 807,936) Allocation latency (p50: 2,641,920) (p95: 3,911,680) (p99: 4,464,640) work rate = 13.45/sec (avg 13.44/sec) (p50: 13.46) (p95: 13.58) (p99: 13.70) alloc stall rate = 3.80/sec (avg: 2.59) (p50: 2.54) (p95: 2.96) (p99: 3.02)With inode cluster pinning and async reclaim:
Run time: 1924 seconds
Read latency (p50: 3,305,472) (p95: 3,715,072) (p99: 3,977,216) Write latency (p50: 187,648) (p95: 553,984) (p99: 789,504) Allocation latency (p50: 2,748,416) (p95: 3,919,872) (p99: 4,448,256) work rate = 13.28/sec (avg 13.32/sec) (p50: 13.26) (p95: 13.34) (p99: 13.34) alloc stall rate = 0.02/sec (avg: 0.02) (p50: 0.01) (p95: 0.03) (p99: 0.03)Latencies don't really change much, nor does the work rate. However, allocation almost never stalls with these changes, whilst the vanilla kernel is sometimes reporting 20 stalls/s over a 60s sample period. This difference is due to inode reclaim being largely non-blocking now.
Quoted 5 lines
IOWs, once we have pinned inode cluster buffers, we can make inode reclaim non-blocking without a major risk of premature and/or spurious OOM killer invocation, and without any changes to memory reclaim infrastructure.
Very interesting result:
Quoted 23 lines
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Amir Goldstein <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/xfs/xfs_icache.c +1
-1@@ -1402,7 +1402,7 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK |SYNC_WAIT , &nr_to_scan); +return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); }/* Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 18/30 xfs: remove IO submission from xfs_reclaim_inode()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
We no longer need to issue IO from shrinker based inode reclaim to prevent spurious OOM killer invocation. This leaves only the global filesystem management operations such as unmount needing to writeback dirty inodes and reclaim them.
Instead of using the reclaim pass to write dirty inodes before reclaiming them, use the AIL to push all the dirty inodes before we try to reclaim them. This allows us to remove all the conditional SYNC_WAIT locking and the writeback code from xfs_reclaim_inode() and greatly simplify the checks we need to do to reclaim an inode.
--- fs/xfs/xfs_icache.c | 117 ++++++++++++-------------------------------- 1 file changed, 31 insertions(+), 86 deletions(-)
fs/xfs/xfs_icache.c +31 -86
@@ -1111,24 +1111,17 @@ xfs_reclaim_inode_grab(* dirty, async => requeue * dirty, sync => flush, wait and reclaim */ -STATIC int +static bool xfs_reclaim_inode (struct xfs_inode *ip,struct xfs_perag *pag,int sync_mode) { -struct xfs_buf *bp =NULL ;xfs_ino_t ino = ip->i_ino;/* for radix_tree_delete */ -int error; -restart: - error =0 ;xfs_ilock (ip,XFS_ILOCK_EXCL ); -if (!xfs_iflock_nowait (ip)) { -if (!(sync_mode &SYNC_WAIT )) - goto out; -xfs_iflock (ip); - } +if (!xfs_iflock_nowait (ip)) + goto out;if (XFS_FORCED_SHUTDOWN (ip->i_mount)) {xfs_iunpin_wait (ip);
@@ -1136,52 +1129,12 @@ xfs_reclaim_inode(xfs_iflush_abort (ip); goto reclaim; } -if (xfs_ipincount (ip)) { -if (!(sync_mode &SYNC_WAIT )) - goto out_ifunlock; -xfs_iunpin_wait (ip); - } -if (xfs_inode_clean (ip)) { -xfs_ifunlock (ip); - goto reclaim; - } - -/* -* Never flush out dirty data during non-blocking reclaim, as it would -* just contend with AIL pushing trying to do the same job. -*/ -if (!(sync_mode &SYNC_WAIT )) +if (xfs_ipincount (ip)) + goto out_ifunlock; +if (!xfs_inode_clean (ip)) goto out_ifunlock; -/* -* Now we have an inode that needs flushing. -* -* Note that xfs_iflush will never block on the inode buffer lock, as -* xfs_ifree_cluster() can lock the inode buffer before it locks the -* ip->i_lock, and we are doing the exact opposite here. As a result, -* doing a blocking xfs_imap_to_bp() to get the cluster buffer would -* result in an ABBA deadlock with xfs_ifree_cluster(). -* -* As xfs_ifree_cluser() must gather all inodes that are active in the -* cache to mark them stale, if we hit this case we don't actually want -* to do IO here - we want the inode marked stale so we can simply -* reclaim it. Hence if we get an EAGAIN error here, just unlock the -* inode, back off and try again. Hopefully the next pass through will -* see the stale flag set on the inode. -*/ - error =xfs_iflush (ip, &bp); -if (error == -EAGAIN ) { -xfs_iunlock (ip,XFS_ILOCK_EXCL ); -/* backoff longer than in xfs_ifree_cluster */ -delay (2 ); - goto restart; - } - -if (!error) { - error =xfs_bwrite (bp); -xfs_buf_relse (bp); - } - +xfs_ifunlock (ip); reclaim:ASSERT (!xfs_isiflocked (ip));
@@ -1231,21 +1184,14 @@ xfs_reclaim_inode(ASSERT (xfs_inode_clean (ip));__xfs_inode_free (ip); -return error; +return true; out_ifunlock:xfs_ifunlock (ip); out: -xfs_iflags_clear (ip,XFS_IRECLAIM );xfs_iunlock (ip,XFS_ILOCK_EXCL ); -/* -* We could return -EAGAIN here to make reclaim rescan the inode tree in -* a short while. However, this just burns CPU time scanning the tree -* waiting for IO to complete and the reclaim work never goes back to -* the idle state. Instead, return 0 to let the next scheduled -* background reclaim attempt to reclaim the inode again. -*/ -return 0 ; +xfs_iflags_clear (ip,XFS_IRECLAIM ); +return false; }/*
@@ -1253,21 +1199,22 @@ xfs_reclaim_inode(* corrupted, we still want to try to reclaim all the inodes. If we don't, * then a shut down during filesystem unmount reclaim walk leak all the * unreclaimed inodes. +* +* Returns non-zero if any AGs or inodes were skipped in the reclaim pass +* so that callers that want to block until all dirty inodes are written back +* and reclaimed can sanely loop. */ -STATIC int +static int xfs_reclaim_inodes_ag (struct xfs_mount *mp,int flags,int *nr_to_scan) {struct xfs_perag *pag; -int error =0 ; -int last_error =0 ;xfs_agnumber_t ag;int trylock = flags &SYNC_TRYLOCK ;int skipped; -restart: ag =0 ; skipped =0 ;while ((pag =xfs_perag_get_tag (mp, ag,XFS_ICI_RECLAIM_TAG ))) {
@@ -1341,9 +1288,8 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; - error =xfs_reclaim_inode (batch[i], pag, flags); -if (error && last_error != -EFSCORRUPTED ) - last_error = error; +if (!xfs_reclaim_inode (batch[i], pag, flags)) + skipped++; } *nr_to_scan -=XFS_LOOKUP_BATCH ;
@@ -1359,19 +1305,7 @@ xfs_reclaim_inodes_ag(mutex_unlock (&pag->pag_ici_reclaim_lock);xfs_perag_put (pag); } - -/* -* if we skipped any AG, and we still have scan count remaining, do -* another pass this time using blocking reclaim semantics (i.e -* waiting on the reclaim locks and ignoring the reclaim cursors). This -* ensure that when we get more reclaimers than AGs we block rather -* than spin trying to execute reclaim. -*/ -if (skipped && (flags &SYNC_WAIT ) && *nr_to_scan >0 ) { - trylock =0 ; - goto restart; - } -return last_error; +return skipped; }int
@@ -1380,8 +1314,18 @@ xfs_reclaim_inodes( int mode) {int nr_to_scan =INT_MAX ; +int skipped; -return xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +if (!(mode &SYNC_WAIT )) +return 0 ; + +do { +xfs_ail_push_all_sync (mp->m_ail); + skipped =xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); + }while (skipped >0 ); + +return 0 ; }/*
@@ -1402,7 +1346,8 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +return 0 ; }/*
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 18/30] xfs: remove IO submission from xfs_reclaim_inode()
Quoted 23 lines · Dave Chinner
From: Dave Chinner <[email protected]>
We no longer need to issue IO from shrinker based inode reclaim to prevent spurious OOM killer invocation. This leaves only the global filesystem management operations such as unmount needing to writeback dirty inodes and reclaim them.
Instead of using the reclaim pass to write dirty inodes before reclaiming them, use the AIL to push all the dirty inodes before we try to reclaim them. This allows us to remove all the conditional
SYNC_WAITlocking and the writeback code fromxfs_reclaim_inode()and greatly simplify the checks we need to do to reclaim an inode.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 117 ++++++++++++-------------------------------- 1 file changed, 31 insertions(+), 86 deletions(-)fs/xfs/xfs_icache.c +0
-0
...
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1292
Quoted 9 lines
fs/xfs/xfs_icache.c +2
-3@@ -1341,9 +1288,8 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; - error =xfs_reclaim_inode (batch[i], pag, flags); -if (error && last_error != -EFSCORRUPTED ) - last_error = error; +if (!xfs_reclaim_inode (batch[i], pag, flags)) + skipped++;
Just a note that I find it a little bit of a landmine that skipped is bumped on trylock failure of the perag reclaim lock when the xfs_reclaim_inodes() caller can now spin on that. It doesn't appear to be an issue with current users, though (xfs_reclaim_workers() passes SYNC_TRYLOCK but not SYNC_WAIT).
Quoted 3 lines
fs/xfs/xfs_icache.c +0
-0} *nr_to_scan -=XFS_LOOKUP_BATCH ;
...
fs/xfs/xfs_icache.c xfs_reclaim_inodes( · line 1322
Quoted 11 lines
fs/xfs/xfs_icache.c +5
-1@@ -1380,8 +1314,18 @@ xfs_reclaim_inodes( int mode) {int nr_to_scan =INT_MAX ; +int skipped; -return xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +if (!(mode &SYNC_WAIT )) +return 0 ; +
Any reason we fall into the loop below if SYNC_WAIT was passed but the above xfs_reclaim_inodes_ag() call would have returned 0?
Looks reasonable other than that inefficiency:
Quoted 22 lines
fs/xfs/xfs_icache.c +8
-1+do { +xfs_ail_push_all_sync (mp->m_ail); + skipped =xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); + }while (skipped >0 ); + +return 0 ; }/* @@ -1402,7 +1346,8 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -return xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +return 0 ; }/* Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 18/30] xfs: remove IO submission from xfs_reclaim_inode()
Quoted 25 lines · Brian Foster
Quoted 23 lines · Dave Chinner
From: Dave Chinner <[email protected]>
We no longer need to issue IO from shrinker based inode reclaim to prevent spurious OOM killer invocation. This leaves only the global filesystem management operations such as unmount needing to writeback dirty inodes and reclaim them.
Instead of using the reclaim pass to write dirty inodes before reclaiming them, use the AIL to push all the dirty inodes before we try to reclaim them. This allows us to remove all the conditional
SYNC_WAITlocking and the writeback code fromxfs_reclaim_inode()and greatly simplify the checks we need to do to reclaim an inode.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 117 ++++++++++++-------------------------------- 1 file changed, 31 insertions(+), 86 deletions(-)fs/xfs/xfs_icache.c +0
-0...
Quoted 13 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1292
Quoted 9 lines
fs/xfs/xfs_icache.c +2
-3@@ -1341,9 +1288,8 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; - error =xfs_reclaim_inode (batch[i], pag, flags); -if (error && last_error != -EFSCORRUPTED ) - last_error = error; +if (!xfs_reclaim_inode (batch[i], pag, flags)) + skipped++;Just a note that I find it a little bit of a landmine that skipped is bumped on trylock failure of the perag reclaim lock when the
xfs_reclaim_inodes()caller can now spin on that.
Intentional, because without bumping skipped on perag reclaim lock failure we can silently skip entire AGs when doing blocking reclaim and xfs_reclaim_inodes() fails to reclaim all inodes in the cache.
It's only necessary to work around fatal bugs this patch exposes for the brief period that this infrastructure is being torn down by this patchset....
Quoted 3 lines
It doesn't appear to
be an issue with current users, though (xfs_reclaim_workers()passesSYNC_TRYLOCKbut notSYNC_WAIT).
xfs_reclaim_workers() is optimisitic, background reclaim, so we just don't care if it skips over things. We just don't want it to block.
Quoted 10 lines
Quoted 3 lines
fs/xfs/xfs_icache.c +0
-0} *nr_to_scan -=XFS_LOOKUP_BATCH ;...
Quoted 6 lines
fs/xfs/xfs_icache.c +1
-0@@ -1380,8 +1314,18 @@ xfs_reclaim_inodes( int mode) {int nr_to_scan =INT_MAX ; +int skipped;
Quoted 8 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes( · line 1322
Quoted 5 lines
fs/xfs/xfs_icache.c +4
-1-return xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +if (!(mode &SYNC_WAIT )) +return 0 ; +Any reason we fall into the loop below if
SYNC_WAITwas passed but the abovexfs_reclaim_inodes_ag()call would have returned 0?
Same thing again. It's temporary to maintain correctness while one thing at time is removed from the reclaim code. This code goes away in the same patch that makes SYNC_WAIT go away.
Quoted 3 lines
Looks reasonable other than that inefficiency:
Reviewed-by Brian Foster <[email protected]>
Thanks!
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 18/30] xfs: remove IO submission from xfs_reclaim_inode()
Quoted 48 lines · Dave Chinner
Quoted 25 lines · Brian Foster
Quoted 23 lines · Dave Chinner
From: Dave Chinner <[email protected]>
We no longer need to issue IO from shrinker based inode reclaim to prevent spurious OOM killer invocation. This leaves only the global filesystem management operations such as unmount needing to writeback dirty inodes and reclaim them.
Instead of using the reclaim pass to write dirty inodes before reclaiming them, use the AIL to push all the dirty inodes before we try to reclaim them. This allows us to remove all the conditional
SYNC_WAITlocking and the writeback code fromxfs_reclaim_inode()and greatly simplify the checks we need to do to reclaim an inode.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 117 ++++++++++++-------------------------------- 1 file changed, 31 insertions(+), 86 deletions(-)fs/xfs/xfs_icache.c +0
-0...
Quoted 13 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1292
Quoted 9 lines
fs/xfs/xfs_icache.c +2
-3@@ -1341,9 +1288,8 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; - error =xfs_reclaim_inode (batch[i], pag, flags); -if (error && last_error != -EFSCORRUPTED ) - last_error = error; +if (!xfs_reclaim_inode (batch[i], pag, flags)) + skipped++;Just a note that I find it a little bit of a landmine that skipped is bumped on trylock failure of the perag reclaim lock when the
xfs_reclaim_inodes()caller can now spin on that.Intentional, because without bumping skipped on perag reclaim lock failure we can silently skip entire AGs when doing blocking reclaim and
xfs_reclaim_inodes()fails to reclaim all inodes in the cache.It's only necessary to work around fatal bugs this patch exposes for the brief period that this infrastructure is being torn down by this patchset....
Quoted 8 lines
Quoted 3 lines
It doesn't appear to
be an issue with current users, though (xfs_reclaim_workers()passesSYNC_TRYLOCKbut notSYNC_WAIT).
xfs_reclaim_workers()is optimisitic, background reclaim, so we just don't care if it skips over things. We just don't want it to block.
Sure, I'm just warning that this puts in place infrastructure that is easily possible to misuse in a way that could lead to livelocks. If it's torn done by the end of the series then it's not a problem. If not, we should probably consider hardening it somehow or another after the series so we don't leave ourselves a landmine to step on.
Brian
Quoted 34 lines
Quoted 10 lines
Quoted 3 lines
fs/xfs/xfs_icache.c +0
-0} *nr_to_scan -=XFS_LOOKUP_BATCH ;...
Quoted 6 lines
fs/xfs/xfs_icache.c +1
-0@@ -1380,8 +1314,18 @@ xfs_reclaim_inodes( int mode) {int nr_to_scan =INT_MAX ; +int skipped;Quoted 8 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes( · line 1322
Quoted 5 lines
fs/xfs/xfs_icache.c +4
-1-return xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +if (!(mode &SYNC_WAIT )) +return 0 ; +Any reason we fall into the loop below if
SYNC_WAITwas passed but the abovexfs_reclaim_inodes_ag()call would have returned 0?Same thing again. It's temporary to maintain correctness while one thing at time is removed from the reclaim code. This code goes away in the same patch that makes
SYNC_WAITgo away.Quoted 3 lines
Looks reasonable other than that inefficiency:
Reviewed-by Brian Foster <[email protected]>Thanks!
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 19/30 xfs: allow multiple reclaimers per AG
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Inode reclaim will still throttle direct reclaim on the per-ag reclaim locks. This is no longer necessary as reclaim can run non-blocking now. Hence we can remove these locks so that we don't arbitrarily block reclaimers just because there are more direct reclaimers than there are AGs.
This can result in multiple reclaimers working on the same range of an AG, but this doesn't cause any apparent issues. Optimising the spread of concurrent reclaimers for best efficiency can be done in a future patchset.
--- fs/xfs/xfs_icache.c | 31 ++++++++++++------------------- fs/xfs/xfs_mount.c | 4 ---- fs/xfs/xfs_mount.h | 1 - 3 files changed, 12 insertions(+), 24 deletions(-)
fs/xfs/xfs_icache.c +12 -19
@@ -1211,12 +1211,9 @@ xfs_reclaim_inodes_ag( int *nr_to_scan) {struct xfs_perag *pag; -xfs_agnumber_t ag; -int trylock = flags &SYNC_TRYLOCK ; -int skipped; +xfs_agnumber_t ag =0 ; +int skipped =0 ; - ag =0 ; - skipped =0 ;while ((pag =xfs_perag_get_tag (mp, ag,XFS_ICI_RECLAIM_TAG ))) {unsigned long first_index =0 ;int done =0 ;
@@ -1224,15 +1221,13 @@ xfs_reclaim_inodes_ag( ag = pag->pag_agno +1 ; -if (trylock) { -if (!mutex_trylock (&pag->pag_ici_reclaim_lock)) { - skipped++; -xfs_perag_put (pag); -continue ; - } - first_index = pag->pag_ici_reclaim_cursor; - }else -mutex_lock (&pag->pag_ici_reclaim_lock); +/* +* If the cursor is not zero, we haven't scanned the whole AG +* so we might have skipped inodes here. +*/ + first_index =READ_ONCE (pag->pag_ici_reclaim_cursor); +if (first_index) + skipped++; do { struct xfs_inode *batch[XFS_LOOKUP_BATCH];
@@ -1298,11 +1293,9 @@ xfs_reclaim_inodes_ag( }while (nr_found && !done && *nr_to_scan >0 ); -if (trylock && !done) - pag->pag_ici_reclaim_cursor = first_index; -else - pag->pag_ici_reclaim_cursor =0 ; -mutex_unlock (&pag->pag_ici_reclaim_lock); +if (done) + first_index =0 ; +WRITE_ONCE (pag->pag_ici_reclaim_cursor, first_index);xfs_perag_put (pag); }return skipped;
fs/xfs/xfs_mount.c +0 -4
@@ -148,7 +148,6 @@ xfs_free_perag(ASSERT (atomic_read (&pag->pag_ref) ==0 );xfs_iunlink_destroy (pag);xfs_buf_hash_destroy (pag); -mutex_destroy (&pag->pag_ici_reclaim_lock);call_rcu (&pag->rcu_head, __xfs_free_perag); } }
@@ -200,7 +199,6 @@ xfs_initialize_perag( pag->pag_agno = index; pag->pag_mount = mp;spin_lock_init (&pag->pag_ici_lock); -mutex_init (&pag->pag_ici_reclaim_lock);INIT_RADIX_TREE (&pag->pag_ici_root,GFP_ATOMIC );if (xfs_buf_hash_init (pag)) goto out_free_pag;
@@ -242,7 +240,6 @@ xfs_initialize_perag( out_hash_destroy:xfs_buf_hash_destroy (pag); out_free_pag: -mutex_destroy (&pag->pag_ici_reclaim_lock);kmem_free (pag); out_unwind_new_pags:/* unwind any prior newly initialized pags */
@@ -252,7 +249,6 @@ xfs_initialize_perag(break ;xfs_buf_hash_destroy (pag);xfs_iunlink_destroy (pag); -mutex_destroy (&pag->pag_ici_reclaim_lock);kmem_free (pag); }return error;
fs/xfs/xfs_mount.h +0 -1
@@ -354,7 +354,6 @@ typedef struct xfs_perag {spinlock_t pag_ici_lock;/* incore inode cache lock */ struct radix_tree_root pag_ici_root;/* incore inode cache root */ int pag_ici_reclaimable;/* reclaimable inodes */ -struct mutex pag_ici_reclaim_lock;/* serialisation point */ unsigned long pag_ici_reclaim_cursor;/* reclaim restart point */ /* buffer cache index */
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 19/30] xfs: allow multiple reclaimers per AG
Quoted 25 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Inode reclaim will still throttle direct reclaim on the per-ag reclaim locks. This is no longer necessary as reclaim can run non-blocking now. Hence we can remove these locks so that we don't arbitrarily block reclaimers just because there are more direct reclaimers than there are AGs.
This can result in multiple reclaimers working on the same range of an AG, but this doesn't cause any apparent issues. Optimising the spread of concurrent reclaimers for best efficiency can be done in a future patchset.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 31 ++++++++++++------------------- fs/xfs/xfs_mount.c | 4 ---- fs/xfs/xfs_mount.h | 1 - 3 files changed, 12 insertions(+), 24 deletions(-)fs/xfs/xfs_icache.c +0
-0
...
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1298
Quoted 12 lines
fs/xfs/xfs_icache.c +3
-5@@ -1298,11 +1293,9 @@ xfs_reclaim_inodes_ag( }while (nr_found && !done && *nr_to_scan >0 ); -if (trylock && !done) - pag->pag_ici_reclaim_cursor = first_index; -else - pag->pag_ici_reclaim_cursor =0 ; -mutex_unlock (&pag->pag_ici_reclaim_lock); +if (done) + first_index =0 ; +WRITE_ONCE (pag->pag_ici_reclaim_cursor, first_index);
I thought the [READ|WRITE]_ONCE() macros had to do with ordering, not necessarily atomicity. Is this write safe if we're running a 32-bit kernel, for example? Outside of that the broader functional change seems reasonable.
Brian
Quoted 54 lines
fs/xfs/xfs_icache.c +0
-0xfs_perag_put (pag); }return skipped;fs/xfs/xfs_mount.c +0
-4@@ -148,7 +148,6 @@ xfs_free_perag(ASSERT (atomic_read (&pag->pag_ref) ==0 );xfs_iunlink_destroy (pag);xfs_buf_hash_destroy (pag); -mutex_destroy (&pag->pag_ici_reclaim_lock);call_rcu (&pag->rcu_head, __xfs_free_perag); } }@@ -200,7 +199,6 @@ xfs_initialize_perag( pag->pag_agno = index; pag->pag_mount = mp;spin_lock_init (&pag->pag_ici_lock); -mutex_init (&pag->pag_ici_reclaim_lock);INIT_RADIX_TREE (&pag->pag_ici_root,GFP_ATOMIC );if (xfs_buf_hash_init (pag)) goto out_free_pag;@@ -242,7 +240,6 @@ xfs_initialize_perag( out_hash_destroy:xfs_buf_hash_destroy (pag); out_free_pag: -mutex_destroy (&pag->pag_ici_reclaim_lock);kmem_free (pag); out_unwind_new_pags:/* unwind any prior newly initialized pags */ @@ -252,7 +249,6 @@ xfs_initialize_perag(break ;xfs_buf_hash_destroy (pag);xfs_iunlink_destroy (pag); -mutex_destroy (&pag->pag_ici_reclaim_lock);kmem_free (pag); }return error;fs/xfs/xfs_mount.h +0
-1@@ -354,7 +354,6 @@ typedef struct xfs_perag {spinlock_t pag_ici_lock;/* incore inode cache lock */ struct radix_tree_root pag_ici_root;/* incore inode cache root */ int pag_ici_reclaimable;/* reclaimable inodes */ -struct mutex pag_ici_reclaim_lock;/* serialisation point */ unsigned long pag_ici_reclaim_cursor;/* reclaim restart point */ /* buffer cache index */ Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 19/30] xfs: allow multiple reclaimers per AG
Quoted 31 lines · Brian Foster
Quoted 25 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Inode reclaim will still throttle direct reclaim on the per-ag reclaim locks. This is no longer necessary as reclaim can run non-blocking now. Hence we can remove these locks so that we don't arbitrarily block reclaimers just because there are more direct reclaimers than there are AGs.
This can result in multiple reclaimers working on the same range of an AG, but this doesn't cause any apparent issues. Optimising the spread of concurrent reclaimers for best efficiency can be done in a future patchset.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 31 ++++++++++++------------------- fs/xfs/xfs_mount.c | 4 ---- fs/xfs/xfs_mount.h | 1 - 3 files changed, 12 insertions(+), 24 deletions(-)fs/xfs/xfs_icache.c +0
-0...
Quoted 4 lines
fs/xfs/xfs_icache.c +0
-0@@ -1298,11 +1293,9 @@ xfs_reclaim_inodes_ag( }while (nr_found && !done && *nr_to_scan >0 );
Quoted 13 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1298
Quoted 8 lines
fs/xfs/xfs_icache.c +3
-5-if (trylock && !done) - pag->pag_ici_reclaim_cursor = first_index; -else - pag->pag_ici_reclaim_cursor =0 ; -mutex_unlock (&pag->pag_ici_reclaim_lock); +if (done) + first_index =0 ; +WRITE_ONCE (pag->pag_ici_reclaim_cursor, first_index);I thought the [READ|WRITE]_ONCE() macros had to do with ordering, not necessarily atomicity. Is this write safe if we're running a 32-bit kernel, for example? Outside of that the broader functional change seems reasonable.
They are used for documenting intentional data races now, too. That's what these are - we don't care about serialisation, but there are static checkers that will now spew "data race" warnings because multiple threads can race reading and writing unserialised variables.
It is safe on 32 bit machines because these variables are 32 bit on 32 bit machines, and reads/writes of 32 bit variables on 32 bit machines are atomic (though not serialised).
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Brian Foster
Reviewed-byRe: [PATCH 19/30] xfs: allow multiple reclaimers per AG
Quoted 46 lines · Dave Chinner
Quoted 31 lines · Brian Foster
Quoted 25 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Inode reclaim will still throttle direct reclaim on the per-ag reclaim locks. This is no longer necessary as reclaim can run non-blocking now. Hence we can remove these locks so that we don't arbitrarily block reclaimers just because there are more direct reclaimers than there are AGs.
This can result in multiple reclaimers working on the same range of an AG, but this doesn't cause any apparent issues. Optimising the spread of concurrent reclaimers for best efficiency can be done in a future patchset.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 31 ++++++++++++------------------- fs/xfs/xfs_mount.c | 4 ---- fs/xfs/xfs_mount.h | 1 - 3 files changed, 12 insertions(+), 24 deletions(-)fs/xfs/xfs_icache.c +0
-0...
Quoted 4 lines
fs/xfs/xfs_icache.c +0
-0@@ -1298,11 +1293,9 @@ xfs_reclaim_inodes_ag( }while (nr_found && !done && *nr_to_scan >0 );Quoted 13 lines
fs/xfs/xfs_icache.c xfs_reclaim_inodes_ag( · line 1298
Quoted 8 lines
fs/xfs/xfs_icache.c +3
-5-if (trylock && !done) - pag->pag_ici_reclaim_cursor = first_index; -else - pag->pag_ici_reclaim_cursor =0 ; -mutex_unlock (&pag->pag_ici_reclaim_lock); +if (done) + first_index =0 ; +WRITE_ONCE (pag->pag_ici_reclaim_cursor, first_index);I thought the [READ|WRITE]_ONCE() macros had to do with ordering, not necessarily atomicity. Is this write safe if we're running a 32-bit kernel, for example? Outside of that the broader functional change seems reasonable.
Quoted 6 lines
They are used for documenting intentional data races now, too. That's what these are - we don't care about serialisation, but there are static checkers that will now spew "data race" warnings because multiple threads can race reading and writing unserialised variables.
I wasn't aware of that. I'm not sure how widely known that is so it might be worth a one liner comment to ensure these are preserved (if they survive the end of the series).
Quoted 4 lines
It is safe on 32 bit machines because these variables are 32 bit on 32 bit machines, and reads/writes of 32 bit variables on 32 bit machines are atomic (though not serialised).
Ah, right. I was thinking they were always 64-bit but that is not the case. With that:
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 20/30 xfs: don't block inode reclaim on the ILOCK
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
When we attempt to reclaim an inode, the first thing we do is take the inode lock. This is blocking right now, so if the inode being accessed by something else (e.g. being flushed to the cluster buffer) we will block here.
Change this to a trylock so that we do not block inode reclaim unnecessarily here.
--- fs/xfs/xfs_icache.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
fs/xfs/xfs_icache.c +5 -3
@@ -1119,9 +1119,10 @@ xfs_reclaim_inode( {xfs_ino_t ino = ip->i_ino;/* for radix_tree_delete */ -xfs_ilock (ip,XFS_ILOCK_EXCL ); -if (!xfs_iflock_nowait (ip)) +if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) goto out; +if (!xfs_iflock_nowait (ip)) + goto out_iunlock;if (XFS_FORCED_SHUTDOWN (ip->i_mount)) {xfs_iunpin_wait (ip);
@@ -1188,8 +1189,9 @@ xfs_reclaim_inode( out_ifunlock:xfs_ifunlock (ip); -out: +out_iunlock:xfs_iunlock (ip,XFS_ILOCK_EXCL ); +out:xfs_iflags_clear (ip,XFS_IRECLAIM );return false; }
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 20/30] xfs: don't block inode reclaim on the ILOCK
Quoted 7 lines · Dave Chinner
From: Dave Chinner <[email protected]>
When we attempt to reclaim an inode, the first thing we do is take the inode lock. This is blocking right now, so if the inode being accessed by something else (e.g. being flushed to the cluster buffer) we will block here.
Quoted 6 lines
Change this to a trylock so that we do not block inode reclaim unnecessarily here.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Quoted 34 lines
fs/xfs/xfs_icache.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)fs/xfs/xfs_icache.c +5
-3@@ -1119,9 +1119,10 @@ xfs_reclaim_inode( {xfs_ino_t ino = ip->i_ino;/* for radix_tree_delete */ -xfs_ilock (ip,XFS_ILOCK_EXCL ); -if (!xfs_iflock_nowait (ip)) +if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) goto out; +if (!xfs_iflock_nowait (ip)) + goto out_iunlock;if (XFS_FORCED_SHUTDOWN (ip->i_mount)) {xfs_iunpin_wait (ip);@@ -1188,8 +1189,9 @@ xfs_reclaim_inode( out_ifunlock:xfs_ifunlock (ip); -out: +out_iunlock:xfs_iunlock (ip,XFS_ILOCK_EXCL ); +out:xfs_iflags_clear (ip,XFS_IRECLAIM );return false; }Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 21/30 xfs: remove SYNC_TRYLOCK from inode reclaim
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
All background reclaim is SYNC_TRYLOCK already, and even blocking reclaim (SYNC_WAIT) can use trylock mechanisms as xfs_reclaim_inodes_ag() will keep cycling until there are no more reclaimable inodes. Hence we can kill SYNC_TRYLOCK from inode reclaim and make everything unconditionally non-blocking.
--- fs/xfs/xfs_icache.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-)
fs/xfs/xfs_icache.c +12 -17
@@ -174,7 +174,7 @@ xfs_reclaim_worker(struct xfs_mount *mp =container_of (to_delayed_work (work), struct xfs_mount, m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_TRYLOCK ); +xfs_reclaim_inodes (mp,0 );xfs_reclaim_work_queue (mp); }
@@ -1030,10 +1030,9 @@ xfs_cowblocks_worker(* Grab the inode for reclaim exclusively. * Return 0 if we grabbed it, non-zero otherwise. */ -STATIC int +static int xfs_reclaim_inode_grab ( -struct xfs_inode *ip, -int flags) +struct xfs_inode *ip) {ASSERT (rcu_read_lock_held ());
@@ -1042,12 +1041,10 @@ xfs_reclaim_inode_grab(return 1 ;/* -* If we are asked for non-blocking operation, do unlocked checks to -* see if the inode already is being flushed or in reclaim to avoid -* lock traffic. +* Do unlocked checks to see if the inode already is being flushed or in +* reclaim to avoid lock traffic. */ -if ((flags &SYNC_TRYLOCK ) && -__xfs_iflags_test (ip,XFS_IFLOCK |XFS_IRECLAIM )) +if (__xfs_iflags_test (ip,XFS_IFLOCK |XFS_IRECLAIM ))return 1 ;/*
@@ -1114,8 +1111,7 @@ xfs_reclaim_inode_grab(static bool xfs_reclaim_inode (struct xfs_inode *ip, -struct xfs_perag *pag, -int sync_mode) +struct xfs_perag *pag) {xfs_ino_t ino = ip->i_ino;/* for radix_tree_delete */
@@ -1209,7 +1205,6 @@ xfs_reclaim_inode(static int xfs_reclaim_inodes_ag (struct xfs_mount *mp, -int flags,int *nr_to_scan) {struct xfs_perag *pag;
@@ -1254,7 +1249,7 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {struct xfs_inode *ip = batch[i]; -if (done ||xfs_reclaim_inode_grab (ip, flags)) +if (done ||xfs_reclaim_inode_grab (ip)) batch[i] =NULL ;/*
@@ -1285,7 +1280,7 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; -if (!xfs_reclaim_inode (batch[i], pag, flags)) +if (!xfs_reclaim_inode (batch[i], pag)) skipped++; }
@@ -1311,13 +1306,13 @@ xfs_reclaim_inodes(int nr_to_scan =INT_MAX ;int skipped; -xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, &nr_to_scan);if (!(mode &SYNC_WAIT ))return 0 ;do {xfs_ail_push_all_sync (mp->m_ail); - skipped =xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); + skipped =xfs_reclaim_inodes_ag (mp, &nr_to_scan); }while (skipped >0 );return 0 ;
@@ -1341,7 +1336,7 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +xfs_reclaim_inodes_ag (mp, &nr_to_scan);return 0 ; }
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 21/30] xfs: remove SYNC_TRYLOCK from inode reclaim
Quoted 11 lines · Dave Chinner
From: Dave Chinner <[email protected]>
All background reclaim is
SYNC_TRYLOCKalready, and even blocking reclaim (SYNC_WAIT) can use trylock mechanisms asxfs_reclaim_inodes_ag()will keep cycling until there are no more reclaimable inodes. Hence we can killSYNC_TRYLOCKfrom inode reclaim and make everything unconditionally non-blocking.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Quoted 109 lines
fs/xfs/xfs_icache.c | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-)fs/xfs/xfs_icache.c +12
-17@@ -174,7 +174,7 @@ xfs_reclaim_worker(struct xfs_mount *mp =container_of (to_delayed_work (work), struct xfs_mount, m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_TRYLOCK ); +xfs_reclaim_inodes (mp,0 );xfs_reclaim_work_queue (mp); }@@ -1030,10 +1030,9 @@ xfs_cowblocks_worker(* Grab the inode for reclaim exclusively. * Return 0 if we grabbed it, non-zero otherwise. */ -STATIC int +static int xfs_reclaim_inode_grab ( -struct xfs_inode *ip, -int flags) +struct xfs_inode *ip) {ASSERT (rcu_read_lock_held ());@@ -1042,12 +1041,10 @@ xfs_reclaim_inode_grab(return 1 ;/* -* If we are asked for non-blocking operation, do unlocked checks to -* see if the inode already is being flushed or in reclaim to avoid -* lock traffic. +* Do unlocked checks to see if the inode already is being flushed or in +* reclaim to avoid lock traffic. */ -if ((flags &SYNC_TRYLOCK ) && -__xfs_iflags_test (ip,XFS_IFLOCK |XFS_IRECLAIM )) +if (__xfs_iflags_test (ip,XFS_IFLOCK |XFS_IRECLAIM ))return 1 ;/* @@ -1114,8 +1111,7 @@ xfs_reclaim_inode_grab(static bool xfs_reclaim_inode (struct xfs_inode *ip, -struct xfs_perag *pag, -int sync_mode) +struct xfs_perag *pag) {xfs_ino_t ino = ip->i_ino;/* for radix_tree_delete */ @@ -1209,7 +1205,6 @@ xfs_reclaim_inode(static int xfs_reclaim_inodes_ag (struct xfs_mount *mp, -int flags,int *nr_to_scan) {struct xfs_perag *pag;@@ -1254,7 +1249,7 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {struct xfs_inode *ip = batch[i]; -if (done ||xfs_reclaim_inode_grab (ip, flags)) +if (done ||xfs_reclaim_inode_grab (ip)) batch[i] =NULL ;/* @@ -1285,7 +1280,7 @@ xfs_reclaim_inodes_ag(for (i =0 ; i < nr_found; i++) {if (!batch[i])continue ; -if (!xfs_reclaim_inode (batch[i], pag, flags)) +if (!xfs_reclaim_inode (batch[i], pag)) skipped++; }@@ -1311,13 +1306,13 @@ xfs_reclaim_inodes(int nr_to_scan =INT_MAX ;int skipped; -xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); +xfs_reclaim_inodes_ag (mp, &nr_to_scan);if (!(mode &SYNC_WAIT ))return 0 ;do {xfs_ail_push_all_sync (mp->m_ail); - skipped =xfs_reclaim_inodes_ag (mp, mode, &nr_to_scan); + skipped =xfs_reclaim_inodes_ag (mp, &nr_to_scan); }while (skipped >0 );return 0 ;@@ -1341,7 +1336,7 @@ xfs_reclaim_inodes_nr(xfs_reclaim_work_queue (mp);xfs_ail_push_all (mp->m_ail); -xfs_reclaim_inodes_ag (mp,SYNC_TRYLOCK , &nr_to_scan); +xfs_reclaim_inodes_ag (mp, &nr_to_scan);return 0 ; }Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 22/30 xfs: remove SYNC_WAIT from xfs_reclaim_inodes()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Clean up xfs_reclaim_inodes() callers. Most callers want blocking behaviour, so just make the existing SYNC_WAIT behaviour the default.
For the xfs_reclaim_worker(), just call xfs_reclaim_inodes_ag() directly because we just want optimistic clean inode reclaim to be done in the background.
For xfs_quiesce_attr() we can just remove the inode reclaim calls as they are a historic relic that was required to flush dirty inodes that contained unlogged changes. We now log all changes to the inodes, so the sync AIL push from xfs_log_quiesce() called by xfs_quiesce_attr() will do all the required inode writeback for freeze.
--- fs/xfs/xfs_icache.c | 48 ++++++++++++++++++++------------------------- fs/xfs/xfs_icache.h | 2 +- fs/xfs/xfs_mount.c | 11 +++++------ fs/xfs/xfs_super.c | 3 --- 4 files changed, 27 insertions(+), 37 deletions(-)
fs/xfs/xfs_icache.c +21 -27
@@ -160,24 +160,6 @@ xfs_reclaim_work_queue(rcu_read_unlock (); } -/* -* This is a fast pass over the inode cache to try to get reclaim moving on as -* many inodes as possible in a short period of time. It kicks itself every few -* seconds, as well as being kicked by the inode cache shrinker when memory -* goes low. It scans as quickly as possible avoiding locked inodes or those -* already being flushed, and once done schedules a future pass. -*/ -void -xfs_reclaim_worker ( -struct work_struct *work) -{ -struct xfs_mount *mp =container_of (to_delayed_work (work), - struct xfs_mount, m_reclaim_work); - -xfs_reclaim_inodes (mp,0 ); -xfs_reclaim_work_queue (mp); -} -static void xfs_perag_set_reclaim_tag (struct xfs_perag *pag)
@@ -1298,24 +1280,17 @@ xfs_reclaim_inodes_ag(return skipped; } -int +void xfs_reclaim_inodes ( -xfs_mount_t *mp, -int mode) +struct xfs_mount *mp) {int nr_to_scan =INT_MAX ;int skipped; -xfs_reclaim_inodes_ag (mp, &nr_to_scan); -if (!(mode &SYNC_WAIT )) -return 0 ; -do {xfs_ail_push_all_sync (mp->m_ail); skipped =xfs_reclaim_inodes_ag (mp, &nr_to_scan); }while (skipped >0 ); - -return 0 ; }/*
@@ -1434,6 +1409,25 @@ xfs_inode_matches_eofb(return true; } +/* +* This is a fast pass over the inode cache to try to get reclaim moving on as +* many inodes as possible in a short period of time. It kicks itself every few +* seconds, as well as being kicked by the inode cache shrinker when memory +* goes low. It scans as quickly as possible avoiding locked inodes or those +* already being flushed, and once done schedules a future pass. +*/ +void +xfs_reclaim_worker ( +struct work_struct *work) +{ +struct xfs_mount *mp =container_of (to_delayed_work (work), + struct xfs_mount, m_reclaim_work); +int nr_to_scan =INT_MAX ; + +xfs_reclaim_inodes_ag (mp, &nr_to_scan); +xfs_reclaim_work_queue (mp); +} + STATIC int xfs_inode_free_eofblocks( struct xfs_inode *ip,
fs/xfs/xfs_icache.h +1 -1
@@ -51,7 +51,7 @@ void xfs_inode_free(struct xfs_inode *ip);void xfs_reclaim_worker (struct work_struct *work); -int xfs_reclaim_inodes (struct xfs_mount *mp,int mode); +void xfs_reclaim_inodes (struct xfs_mount *mp);int xfs_reclaim_inodes_count (struct xfs_mount *mp);long xfs_reclaim_inodes_nr (struct xfs_mount *mp,int nr_to_scan);
fs/xfs/xfs_mount.c +5 -6
@@ -1011,7 +1011,7 @@ xfs_mountfs(* quota inodes. */ cancel_delayed_work_sync (&mp->m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_WAIT ); +xfs_reclaim_inodes (mp);xfs_health_unmount (mp); out_log_dealloc: mp->m_flags |=XFS_MOUNT_UNMOUNTING ;
@@ -1088,13 +1088,12 @@ xfs_unmountfs(xfs_ail_push_all_sync (mp->m_ail);/* -* And reclaim all inodes. At this point there should be no dirty -* inodes and none should be pinned or locked, but use synchronous -* reclaim just to be sure. We can stop background inode reclaim -* here as well if it is still running. +* Reclaim all inodes. At this point there should be no dirty inodes and +* none should be pinned or locked. Stop background inode reclaim here +* if it is still running. */ cancel_delayed_work_sync (&mp->m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_WAIT ); +xfs_reclaim_inodes (mp);xfs_health_unmount (mp);xfs_qm_unmount (mp);
fs/xfs/xfs_super.c +0 -3
@@ -890,9 +890,6 @@ xfs_quiesce_attr(/* force the log to unpin objects from the now complete transactions */ xfs_log_force (mp,XFS_LOG_SYNC ); -/* reclaim inodes to do any IO before the freeze completes */ -xfs_reclaim_inodes (mp,0 ); -xfs_reclaim_inodes (mp,SYNC_WAIT );/* Push the superblock and write an unmount record */ error =xfs_log_sbcount (mp);
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 22/30] xfs: remove SYNC_WAIT from xfs_reclaim_inodes()
Quoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Clean up
xfs_reclaim_inodes()callers. Most callers want blocking behaviour, so just make the existingSYNC_WAITbehaviour the default.For the
xfs_reclaim_worker(), just callxfs_reclaim_inodes_ag()directly because we just want optimistic clean inode reclaim to be done in the background.
Quoted 7 lines
For
xfs_quiesce_attr()we can just remove the inode reclaim calls as they are a historic relic that was required to flush dirty inodes that contained unlogged changes. We now log all changes to the inodes, so the sync AIL push fromxfs_log_quiesce()called byxfs_quiesce_attr()will do all the required inode writeback for freeze.
The above change should probably be a standalone patch, but not worth changing at this point:
BTW, is there any reason we continue to drain the buffer lru for freeze as well?
Brian
Quoted 152 lines
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_icache.c | 48 ++++++++++++++++++++------------------------- fs/xfs/xfs_icache.h | 2 +- fs/xfs/xfs_mount.c | 11 +++++------ fs/xfs/xfs_super.c | 3 --- 4 files changed, 27 insertions(+), 37 deletions(-)fs/xfs/xfs_icache.c +21
-27@@ -160,24 +160,6 @@ xfs_reclaim_work_queue(rcu_read_unlock (); } -/* -* This is a fast pass over the inode cache to try to get reclaim moving on as -* many inodes as possible in a short period of time. It kicks itself every few -* seconds, as well as being kicked by the inode cache shrinker when memory -* goes low. It scans as quickly as possible avoiding locked inodes or those -* already being flushed, and once done schedules a future pass. -*/ -void -xfs_reclaim_worker ( -struct work_struct *work) -{ -struct xfs_mount *mp =container_of (to_delayed_work (work), - struct xfs_mount, m_reclaim_work); - -xfs_reclaim_inodes (mp,0 ); -xfs_reclaim_work_queue (mp); -} -static void xfs_perag_set_reclaim_tag (struct xfs_perag *pag)@@ -1298,24 +1280,17 @@ xfs_reclaim_inodes_ag(return skipped; } -int +void xfs_reclaim_inodes ( -xfs_mount_t *mp, -int mode) +struct xfs_mount *mp) {int nr_to_scan =INT_MAX ;int skipped; -xfs_reclaim_inodes_ag (mp, &nr_to_scan); -if (!(mode &SYNC_WAIT )) -return 0 ; -do {xfs_ail_push_all_sync (mp->m_ail); skipped =xfs_reclaim_inodes_ag (mp, &nr_to_scan); }while (skipped >0 ); - -return 0 ; }/* @@ -1434,6 +1409,25 @@ xfs_inode_matches_eofb(return true; } +/* +* This is a fast pass over the inode cache to try to get reclaim moving on as +* many inodes as possible in a short period of time. It kicks itself every few +* seconds, as well as being kicked by the inode cache shrinker when memory +* goes low. It scans as quickly as possible avoiding locked inodes or those +* already being flushed, and once done schedules a future pass. +*/ +void +xfs_reclaim_worker ( +struct work_struct *work) +{ +struct xfs_mount *mp =container_of (to_delayed_work (work), + struct xfs_mount, m_reclaim_work); +int nr_to_scan =INT_MAX ; + +xfs_reclaim_inodes_ag (mp, &nr_to_scan); +xfs_reclaim_work_queue (mp); +} + STATIC int xfs_inode_free_eofblocks( struct xfs_inode *ip,fs/xfs/xfs_icache.h +1
-1@@ -51,7 +51,7 @@ void xfs_inode_free(struct xfs_inode *ip);void xfs_reclaim_worker (struct work_struct *work); -int xfs_reclaim_inodes (struct xfs_mount *mp,int mode); +void xfs_reclaim_inodes (struct xfs_mount *mp);int xfs_reclaim_inodes_count (struct xfs_mount *mp);long xfs_reclaim_inodes_nr (struct xfs_mount *mp,int nr_to_scan);fs/xfs/xfs_mount.c +5
-6@@ -1011,7 +1011,7 @@ xfs_mountfs(* quota inodes. */ cancel_delayed_work_sync (&mp->m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_WAIT ); +xfs_reclaim_inodes (mp);xfs_health_unmount (mp); out_log_dealloc: mp->m_flags |=XFS_MOUNT_UNMOUNTING ;@@ -1088,13 +1088,12 @@ xfs_unmountfs(xfs_ail_push_all_sync (mp->m_ail);/* -* And reclaim all inodes. At this point there should be no dirty -* inodes and none should be pinned or locked, but use synchronous -* reclaim just to be sure. We can stop background inode reclaim -* here as well if it is still running. +* Reclaim all inodes. At this point there should be no dirty inodes and +* none should be pinned or locked. Stop background inode reclaim here +* if it is still running. */ cancel_delayed_work_sync (&mp->m_reclaim_work); -xfs_reclaim_inodes (mp,SYNC_WAIT ); +xfs_reclaim_inodes (mp);xfs_health_unmount (mp);xfs_qm_unmount (mp);fs/xfs/xfs_super.c +0
-3@@ -890,9 +890,6 @@ xfs_quiesce_attr(/* force the log to unpin objects from the now complete transactions */ xfs_log_force (mp,XFS_LOG_SYNC ); -/* reclaim inodes to do any IO before the freeze completes */ -xfs_reclaim_inodes (mp,0 ); -xfs_reclaim_inodes (mp,SYNC_WAIT );/* Push the superblock and write an unmount record */ error =xfs_log_sbcount (mp);Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 22/30] xfs: remove SYNC_WAIT from xfs_reclaim_inodes()
Quoted 19 lines · Brian Foster
Quoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Clean up
xfs_reclaim_inodes()callers. Most callers want blocking behaviour, so just make the existingSYNC_WAITbehaviour the default.For the
xfs_reclaim_worker(), just callxfs_reclaim_inodes_ag()directly because we just want optimistic clean inode reclaim to be done in the background.Quoted 7 lines
For
xfs_quiesce_attr()we can just remove the inode reclaim calls as they are a historic relic that was required to flush dirty inodes that contained unlogged changes. We now log all changes to the inodes, so the sync AIL push fromxfs_log_quiesce()called byxfs_quiesce_attr()will do all the required inode writeback for freeze.
Quoted 7 lines
The above change should probably be a standalone patch, but not worth changing at this point:
Reviewed-by Brian Foster <[email protected]>BTW, is there any reason we continue to drain the buffer lru for freeze as well?
Probably not - out of scope for this patch so I haven't really thought about it.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 23/30 xfs: clean up inode reclaim comments
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Inode reclaim is quite different now to the way described in various comments, so update all the comments explaining what it does and how it works.
--- fs/xfs/xfs_icache.c | 128 ++++++++++++-------------------------------- 1 file changed, 35 insertions(+), 93 deletions(-)
fs/xfs/xfs_icache.c +35 -93
@@ -141,11 +141,8 @@ xfs_inode_free( }/* -* Queue a new inode reclaim pass if there are reclaimable inodes and there -* isn't a reclaim pass already in progress. By default it runs every 5s based -* on the xfs periodic sync default of 30s. Perhaps this should have it's own -* tunable, but that can be done if this method proves to be ineffective or too -* aggressive. +* Queue background inode reclaim work if there are reclaimable inodes and there +* isn't reclaim work already scheduled or in progress. */ static void xfs_reclaim_work_queue(
@@ -600,48 +597,31 @@ xfs_iget_cache_miss( }/* -* Look up an inode by number in the given file system. -* The inode is looked up in the cache held in each AG. -* If the inode is found in the cache, initialise the vfs inode -* if necessary. +* Look up an inode by number in the given file system. The inode is looked up +* in the cache held in each AG. If the inode is found in the cache, initialise +* the vfs inode if necessary. * -* If it is not in core, read it in from the file system's device, -* add it to the cache and initialise the vfs inode. +* If it is not in core, read it in from the file system's device, add it to the +* cache and initialise the vfs inode. * * The inode is locked according to the value of the lock_flags parameter. -* This flag parameter indicates how and if the inode's IO lock and inode lock -* should be taken. -* -* mp -- the mount point structure for the current file system. It points -* to the inode hash table. -* tp -- a pointer to the current transaction if there is one. This is -* simply passed through to the xfs_iread() call. -* ino -- the number of the inode desired. This is the unique identifier -* within the file system for the inode being requested. -* lock_flags -- flags indicating how to lock the inode. See the comment -* for xfs_ilock() for a list of valid values. +* Inode lookup is only done during metadata operations and not as part of the +* data IO path. Hence we only allow locking of the XFS_ILOCK during lookup. */ int xfs_iget ( -xfs_mount_t *mp, -xfs_trans_t *tp, -xfs_ino_t ino, -uint flags, -uint lock_flags, -xfs_inode_t **ipp) +struct xfs_mount *mp, +struct xfs_trans *tp, +xfs_ino_t ino, +uint flags, +uint lock_flags, +struct xfs_inode **ipp) { -xfs_inode_t *ip; -int error; -xfs_perag_t *pag; -xfs_agino_t agino; +struct xfs_inode *ip; +struct xfs_perag *pag; +xfs_agino_t agino; +int error; -/* -* xfs_reclaim_inode() uses the ILOCK to ensure an inode -* doesn't get freed while it's being referenced during a -* radix tree traversal here. It assumes this function -* aqcuires only the ILOCK (and therefore it has no need to -* involve the IOLOCK in this synchronization). -*/ ASSERT ((lock_flags & (XFS_IOLOCK_EXCL |XFS_IOLOCK_SHARED )) ==0 );/* reject inode numbers outside existing AGs */
@@ -758,15 +738,7 @@ xfs_inode_walk_ag_grab(ASSERT (rcu_read_lock_held ()); -/* -* check for stale RCU freed inode -* -* If the inode has been reallocated, it doesn't matter if it's not in -* the AG we are walking - we are walking for writeback, so if it -* passes all the "valid inode" checks and is dirty, then we'll write -* it back anyway. If it has been reallocated and still being -* initialised, the XFS_INEW check below will catch it. -*/ +/* Check for stale RCU freed inode */ spin_lock (&ip->i_flags_lock);if (!ip->i_ino) goto out_unlock_noent;
@@ -1052,43 +1024,16 @@ xfs_reclaim_inode_grab( }/* -* Inodes in different states need to be treated differently. The following -* table lists the inode states and the reclaim actions necessary: -* -* inode state iflush ret required action -* --------------- ---------- --------------- -* bad - reclaim -* shutdown EIO unpin and reclaim -* clean, unpinned 0 reclaim -* stale, unpinned 0 reclaim -* clean, pinned(*) 0 requeue -* stale, pinned EAGAIN requeue -* dirty, async - requeue -* dirty, sync 0 reclaim +* Inode reclaim is non-blocking, so the default action if progress cannot be +* made is to "requeue" the inode for reclaim by unlocking it and clearing the +* XFS_IRECLAIM flag. If we are in a shutdown state, we don't care about +* blocking anymore and hence we can wait for the inode to be able to reclaim +* it. * -* (*) dgc: I don't think the clean, pinned state is possible but it gets -* handled anyway given the order of checks implemented. -* -* Also, because we get the flush lock first, we know that any inode that has -* been flushed delwri has had the flush completed by the time we check that -* the inode is clean. -* -* Note that because the inode is flushed delayed write by AIL pushing, the -* flush lock may already be held here and waiting on it can result in very -* long latencies. Hence for sync reclaims, where we wait on the flush lock, -* the caller should push the AIL first before trying to reclaim inodes to -* minimise the amount of time spent waiting. For background relaim, we only -* bother to reclaim clean inodes anyway. -* -* Hence the order of actions after gaining the locks should be: -* bad => reclaim -* shutdown => unpin and reclaim -* pinned, async => requeue -* pinned, sync => unpin -* stale => reclaim -* clean => reclaim -* dirty, async => requeue -* dirty, sync => flush, wait and reclaim +* We do no IO here - if callers require inodes to be cleaned they must push the +* AIL first to trigger writeback of dirty inodes. This enables writeback to be +* done in the background in a non-blocking manner, and enables memory reclaim +* to make progress without blocking. */ static bool xfs_reclaim_inode(
@@ -1294,13 +1239,11 @@ xfs_reclaim_inodes( }/* -* Scan a certain number of inodes for reclaim. -* -* When called we make sure that there is a background (fast) inode reclaim in -* progress, while we will throttle the speed of reclaim via doing synchronous -* reclaim of inodes. That means if we come across dirty inodes, we wait for -* them to be cleaned, which we hope will not be very long due to the -* background walker having already kicked the IO off on those dirty inodes. +* The shrinker infrastructure determines how many inodes we should scan for +* reclaim. We want as many clean inodes ready to reclaim as possible, so we +* push the AIL here. We also want to proactively free up memory if we can to +* minimise the amount of work memory reclaim has to do so we kick the +* background reclaim if it isn't already scheduled. */ long xfs_reclaim_inodes_nr(
@@ -1413,8 +1356,7 @@ xfs_inode_matches_eofb(* This is a fast pass over the inode cache to try to get reclaim moving on as * many inodes as possible in a short period of time. It kicks itself every few * seconds, as well as being kicked by the inode cache shrinker when memory -* goes low. It scans as quickly as possible avoiding locked inodes or those -* already being flushed, and once done schedules a future pass. +* goes low. */ void xfs_reclaim_worker(
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 23/30] xfs: clean up inode reclaim comments
Quoted 9 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Inode reclaim is quite different now to the way described in various comments, so update all the comments explaining what it does and how it works.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Quoted 189 lines
fs/xfs/xfs_icache.c | 128 ++++++++++++-------------------------------- 1 file changed, 35 insertions(+), 93 deletions(-)fs/xfs/xfs_icache.c +35
-93@@ -141,11 +141,8 @@ xfs_inode_free( }/* -* Queue a new inode reclaim pass if there are reclaimable inodes and there -* isn't a reclaim pass already in progress. By default it runs every 5s based -* on the xfs periodic sync default of 30s. Perhaps this should have it's own -* tunable, but that can be done if this method proves to be ineffective or too -* aggressive. +* Queue background inode reclaim work if there are reclaimable inodes and there +* isn't reclaim work already scheduled or in progress. */ static void xfs_reclaim_work_queue(@@ -600,48 +597,31 @@ xfs_iget_cache_miss( }/* -* Look up an inode by number in the given file system. -* The inode is looked up in the cache held in each AG. -* If the inode is found in the cache, initialise the vfs inode -* if necessary. +* Look up an inode by number in the given file system. The inode is looked up +* in the cache held in each AG. If the inode is found in the cache, initialise +* the vfs inode if necessary. * -* If it is not in core, read it in from the file system's device, -* add it to the cache and initialise the vfs inode. +* If it is not in core, read it in from the file system's device, add it to the +* cache and initialise the vfs inode. * * The inode is locked according to the value of the lock_flags parameter. -* This flag parameter indicates how and if the inode's IO lock and inode lock -* should be taken. -* -* mp -- the mount point structure for the current file system. It points -* to the inode hash table. -* tp -- a pointer to the current transaction if there is one. This is -* simply passed through to the xfs_iread() call. -* ino -- the number of the inode desired. This is the unique identifier -* within the file system for the inode being requested. -* lock_flags -- flags indicating how to lock the inode. See the comment -* for xfs_ilock() for a list of valid values. +* Inode lookup is only done during metadata operations and not as part of the +* data IO path. Hence we only allow locking of the XFS_ILOCK during lookup. */ int xfs_iget ( -xfs_mount_t *mp, -xfs_trans_t *tp, -xfs_ino_t ino, -uint flags, -uint lock_flags, -xfs_inode_t **ipp) +struct xfs_mount *mp, +struct xfs_trans *tp, +xfs_ino_t ino, +uint flags, +uint lock_flags, +struct xfs_inode **ipp) { -xfs_inode_t *ip; -int error; -xfs_perag_t *pag; -xfs_agino_t agino; +struct xfs_inode *ip; +struct xfs_perag *pag; +xfs_agino_t agino; +int error; -/* -* xfs_reclaim_inode() uses the ILOCK to ensure an inode -* doesn't get freed while it's being referenced during a -* radix tree traversal here. It assumes this function -* aqcuires only the ILOCK (and therefore it has no need to -* involve the IOLOCK in this synchronization). -*/ ASSERT ((lock_flags & (XFS_IOLOCK_EXCL |XFS_IOLOCK_SHARED )) ==0 );/* reject inode numbers outside existing AGs */ @@ -758,15 +738,7 @@ xfs_inode_walk_ag_grab(ASSERT (rcu_read_lock_held ()); -/* -* check for stale RCU freed inode -* -* If the inode has been reallocated, it doesn't matter if it's not in -* the AG we are walking - we are walking for writeback, so if it -* passes all the "valid inode" checks and is dirty, then we'll write -* it back anyway. If it has been reallocated and still being -* initialised, the XFS_INEW check below will catch it. -*/ +/* Check for stale RCU freed inode */ spin_lock (&ip->i_flags_lock);if (!ip->i_ino) goto out_unlock_noent;@@ -1052,43 +1024,16 @@ xfs_reclaim_inode_grab( }/* -* Inodes in different states need to be treated differently. The following -* table lists the inode states and the reclaim actions necessary: -* -* inode state iflush ret required action -* --------------- ---------- --------------- -* bad - reclaim -* shutdown EIO unpin and reclaim -* clean, unpinned 0 reclaim -* stale, unpinned 0 reclaim -* clean, pinned(*) 0 requeue -* stale, pinned EAGAIN requeue -* dirty, async - requeue -* dirty, sync 0 reclaim +* Inode reclaim is non-blocking, so the default action if progress cannot be +* made is to "requeue" the inode for reclaim by unlocking it and clearing the +* XFS_IRECLAIM flag. If we are in a shutdown state, we don't care about +* blocking anymore and hence we can wait for the inode to be able to reclaim +* it. * -* (*) dgc: I don't think the clean, pinned state is possible but it gets -* handled anyway given the order of checks implemented. -* -* Also, because we get the flush lock first, we know that any inode that has -* been flushed delwri has had the flush completed by the time we check that -* the inode is clean. -* -* Note that because the inode is flushed delayed write by AIL pushing, the -* flush lock may already be held here and waiting on it can result in very -* long latencies. Hence for sync reclaims, where we wait on the flush lock, -* the caller should push the AIL first before trying to reclaim inodes to -* minimise the amount of time spent waiting. For background relaim, we only -* bother to reclaim clean inodes anyway. -* -* Hence the order of actions after gaining the locks should be: -* bad => reclaim -* shutdown => unpin and reclaim -* pinned, async => requeue -* pinned, sync => unpin -* stale => reclaim -* clean => reclaim -* dirty, async => requeue -* dirty, sync => flush, wait and reclaim +* We do no IO here - if callers require inodes to be cleaned they must push the +* AIL first to trigger writeback of dirty inodes. This enables writeback to be +* done in the background in a non-blocking manner, and enables memory reclaim +* to make progress without blocking. */ static bool xfs_reclaim_inode(@@ -1294,13 +1239,11 @@ xfs_reclaim_inodes( }/* -* Scan a certain number of inodes for reclaim. -* -* When called we make sure that there is a background (fast) inode reclaim in -* progress, while we will throttle the speed of reclaim via doing synchronous -* reclaim of inodes. That means if we come across dirty inodes, we wait for -* them to be cleaned, which we hope will not be very long due to the -* background walker having already kicked the IO off on those dirty inodes. +* The shrinker infrastructure determines how many inodes we should scan for +* reclaim. We want as many clean inodes ready to reclaim as possible, so we +* push the AIL here. We also want to proactively free up memory if we can to +* minimise the amount of work memory reclaim has to do so we kick the +* background reclaim if it isn't already scheduled. */ long xfs_reclaim_inodes_nr(@@ -1413,8 +1356,7 @@ xfs_inode_matches_eofb(* This is a fast pass over the inode cache to try to get reclaim moving on as * many inodes as possible in a short period of time. It kicks itself every few * seconds, as well as being kicked by the inode cache shrinker when memory -* goes low. It scans as quickly as possible avoiding locked inodes or those -* already being flushed, and once done schedules a future pass. +* goes low. */ void xfs_reclaim_worker(Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 24/30 xfs: rework stale inodes in xfs_ifree_cluster
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Once we have inodes pinning the cluster buffer and attached whenever they are dirty, we no longer have a guarantee that the items are flush locked when we lock the cluster buffer. Hence we cannot just walk the buffer log item list and modify the attached inodes.
If the inode is not flush locked, we have to ILOCK it first and then flush lock it to do all the prerequisite checks needed to avoid races with other code. This is already handled by xfs_ifree_get_one_inode(), so rework the inode iteration loop and function to update all inodes in cache whether they are attached to the buffer or not.
Note: we also remove the copying of the log item lsn to the ili_flush_lsn as xfs_iflush_done() now uses the XFS_ISTALE flag to trigger aborts and so flush lsn matching is not needed in IO completion for processing freed inodes.
--- fs/xfs/xfs_inode.c | 158 ++++++++++++++++++--------------------------- 1 file changed, 62 insertions(+), 96 deletions(-)
fs/xfs/xfs_inode.c +62 -96
@@ -2517,17 +2517,19 @@ xfs_iunlink_remove( }/* -* Look up the inode number specified and mark it stale if it is found. If it is -* dirty, return the inode so it can be attached to the cluster buffer so it can -* be processed appropriately when the cluster free transaction completes. +* Look up the inode number specified and if it is not already marked XFS_ISTALE +* mark it stale. We should only find clean inodes in this lookup that aren't +* already stale. */ -static struct xfs_inode * -xfs_ifree_get_one_inode ( -struct xfs_perag *pag, +static void +xfs_ifree_mark_inode_stale ( +struct xfs_buf *bp,struct xfs_inode *free_ip,xfs_ino_t inum) { -struct xfs_mount *mp = pag->pag_mount; +struct xfs_mount *mp = bp->b_mount; +struct xfs_perag *pag = bp->b_pag; +struct xfs_inode_log_item *iip;struct xfs_inode *ip; retry:
@@ -2535,8 +2537,10 @@ xfs_ifree_get_one_inode( ip =radix_tree_lookup (&pag->pag_ici_root,XFS_INO_TO_AGINO (mp, inum));/* Inode not in memory, nothing to do */ -if (!ip) - goto out_rcu_unlock; +if (!ip) { +rcu_read_unlock (); +return ; + }/* * because this is an RCU protected lookup, we could find a recently
@@ -2547,9 +2551,9 @@ xfs_ifree_get_one_inode(spin_lock (&ip->i_flags_lock);if (ip->i_ino != inum ||__xfs_iflags_test (ip,XFS_ISTALE )) {spin_unlock (&ip->i_flags_lock); - goto out_rcu_unlock; +rcu_read_unlock (); +return ; } -spin_unlock (&ip->i_flags_lock);/* * Don't try to lock/unlock the current inode, but we _cannot_ skip the
@@ -2559,43 +2563,53 @@ xfs_ifree_get_one_inode(*/ if (ip != free_ip) {if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) { +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();delay (1 ); goto retry; } - -/* -* Check the inode number again in case we're racing with -* freeing in xfs_reclaim_inode(). See the comments in that -* function for more information as to why the initial check is -* not sufficient. -*/ -if (ip->i_ino != inum) { -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_rcu_unlock; - } } + ip->i_flags |=XFS_ISTALE ; +spin_unlock (&ip->i_flags_lock);rcu_read_unlock (); -xfs_iflock (ip); -xfs_iflags_set (ip,XFS_ISTALE ); +/* +* If we can't get the flush lock, the inode is already attached. All +* we needed to do here is mark the inode stale so buffer IO completion +* will remove it from the AIL. +*/ + iip = ip->i_itemp; +if (!xfs_iflock_nowait (ip)) { +ASSERT (!list_empty (&iip->ili_item.li_bio_list)); +ASSERT (iip->ili_last_fields); + goto out_iunlock; + } +ASSERT (!iip ||list_empty (&iip->ili_item.li_bio_list));/* -* We don't need to attach clean inodes or those only with unlogged -* changes (which we throw away, anyway). +* Clean inodes can be released immediately. Everything else has to go +* through xfs_iflush_abort() on journal commit as the flock +* synchronises removal of the inode from the cluster buffer against +* inode reclaim. */ -if (!ip->i_itemp ||xfs_inode_clean (ip)) { -ASSERT (ip != free_ip); +if (xfs_inode_clean (ip)) {xfs_ifunlock (ip); -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_no_inode; + goto out_iunlock; } -return ip; -out_rcu_unlock: -rcu_read_unlock (); -out_no_inode: -return NULL ; +/* we have a dirty inode in memory that has not yet been flushed. */ +ASSERT (iip->ili_fields); +spin_lock (&iip->ili_lock); + iip->ili_last_fields = iip->ili_fields; + iip->ili_fields =0 ; + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); +ASSERT (iip->ili_last_fields); + +out_iunlock: +if (ip != free_ip) +xfs_iunlock (ip,XFS_ILOCK_EXCL ); }/*
@@ -2605,26 +2619,20 @@ xfs_ifree_get_one_inode(*/ STATIC intxfs_ifree_cluster ( -xfs_inode_t *free_ip, -xfs_trans_t *tp, +struct xfs_inode *free_ip, +struct xfs_trans *tp,struct xfs_icluster *xic) { -xfs_mount_t *mp = free_ip->i_mount; +struct xfs_mount *mp = free_ip->i_mount; +struct xfs_ino_geometry *igeo =M_IGEO (mp); +struct xfs_buf *bp; +xfs_daddr_t blkno; +xfs_ino_t inum = xic->first_ino;int nbufs;int i, j;int ioffset; -xfs_daddr_t blkno; -xfs_buf_t *bp; -xfs_inode_t *ip; -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip; -struct xfs_perag *pag; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -xfs_ino_t inum;int error; - inum = xic->first_ino; - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, inum)); nbufs = igeo->ialloc_blks / igeo->blocks_per_cluster;for (j =0 ; j < nbufs; j++, inum += igeo->inodes_per_cluster) {
@@ -2668,59 +2676,16 @@ xfs_ifree_cluster( bp->b_ops = &xfs_inode_buf_ops;/* -* Walk the inodes already attached to the buffer and mark them -* stale. These will all have the flush locks held, so an -* in-memory inode walk can't lock them. By marking them all -* stale first, we will not attempt to lock them in the loop -* below as the XFS_ISTALE flag will be set. -*/ -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { -if (lip->li_type ==XFS_LI_INODE ) { - iip = (struct xfs_inode_log_item *)lip; -xfs_trans_ail_copy_lsn (mp->m_ail, - &iip->ili_flush_lsn, - &iip->ili_item.li_lsn); -xfs_iflags_set (iip->ili_inode,XFS_ISTALE ); - } - } - - -/* -* For each inode in memory attempt to add it to the inode -* buffer and set it up for being staled on buffer IO -* completion. This is safe as we've locked out tail pushing -* and flushing by locking the buffer. -* -* We have already marked every inode that was part of a -* transaction stale above, which means there is no point in -* even trying to lock them. +* Now we need to set all the cached clean inodes as XFS_ISTALE, +* too. This requires lookups, and will skip inodes that we've +* already marked XFS_ISTALE. */ -for (i =0 ; i < igeo->inodes_per_cluster; i++) { - ip =xfs_ifree_get_one_inode (pag, free_ip, inum + i); -if (!ip) -continue ; - - iip = ip->i_itemp; -spin_lock (&iip->ili_lock); - iip->ili_last_fields = iip->ili_fields; - iip->ili_fields =0 ; - iip->ili_fsync_fields =0 ; -spin_unlock (&iip->ili_lock); -xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, - &iip->ili_item.li_lsn); - -list_add_tail (&iip->ili_item.li_bio_list, - &bp->b_li_list); - -if (ip != free_ip) -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - } +for (i =0 ; i < igeo->inodes_per_cluster; i++) +xfs_ifree_mark_inode_stale (bp, free_ip, inum + i);xfs_trans_stale_inode_buf (tp, bp);xfs_trans_binval (tp, bp); } - -xfs_perag_put (pag);return 0 ; }
@@ -3845,6 +3810,7 @@ xfs_iflush_int( iip->ili_fields =0 ; iip->ili_fsync_fields =0 ;spin_unlock (&iip->ili_lock); +ASSERT (iip->ili_last_fields);/* * Store the current LSN of the inode so that we can tell whether the
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 24/30] xfs: rework stale inodes in xfs_ifree_cluster
Quoted 29 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Once we have inodes pinning the cluster buffer and attached whenever they are dirty, we no longer have a guarantee that the items are flush locked when we lock the cluster buffer. Hence we cannot just walk the buffer log item list and modify the attached inodes.
If the inode is not flush locked, we have to ILOCK it first and then flush lock it to do all the prerequisite checks needed to avoid races with other code. This is already handled by
xfs_ifree_get_one_inode(), so rework the inode iteration loop and function to update all inodes in cache whether they are attached to the buffer or not.Note: we also remove the copying of the log item lsn to the
ili_flush_lsnasxfs_iflush_done()now uses theXFS_ISTALEflag to trigger aborts and so flush lsn matching is not needed in IO completion for processing freed inodes.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 158 ++++++++++++++++++--------------------------- 1 file changed, 62 insertions(+), 96 deletions(-)fs/xfs/xfs_inode.c +0
-0
...
Quoted 10 lines
fs/xfs/xfs_inode.c +1
-1@@ -2559,43 +2563,53 @@ xfs_ifree_get_one_inode(*/ if (ip != free_ip) {if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) { +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();delay (1 ); goto retry; } -
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2576
Quoted 10 lines
fs/xfs/xfs_inode.c +0
-10-/* -* Check the inode number again in case we're racing with -* freeing in xfs_reclaim_inode(). See the comments in that -* function for more information as to why the initial check is -* not sufficient. -*/ -if (ip->i_ino != inum) { -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_rcu_unlock; - }
Why is the recheck under ILOCK_EXCL no longer necessary? It looks like reclaim decides whether to proceed or not under the ilock and doesn't acquire the spinlock until it decides to reclaim. Hm?
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2580
Quoted 12 lines
fs/xfs/xfs_inode.c +7
-2} + ip->i_flags |=XFS_ISTALE ; +spin_unlock (&ip->i_flags_lock);rcu_read_unlock (); -xfs_iflock (ip); -xfs_iflags_set (ip,XFS_ISTALE ); +/* +* If we can't get the flush lock, the inode is already attached. All +* we needed to do here is mark the inode stale so buffer IO completion +* will remove it from the AIL. +*/
To make sure I'm following this correctly, we can assume the inode is attached based on an iflock_nowait() failure because we hold the ilock, right? IOW, any other task doing a similar iflock check would have to do so under ilock and release the flush lock first if the inode didn't end up flushed, for whatever reason.
Quoted 29 lines
fs/xfs/xfs_inode.c +13
-10+ iip = ip->i_itemp; +if (!xfs_iflock_nowait (ip)) { +ASSERT (!list_empty (&iip->ili_item.li_bio_list)); +ASSERT (iip->ili_last_fields); + goto out_iunlock; + } +ASSERT (!iip ||list_empty (&iip->ili_item.li_bio_list));/* -* We don't need to attach clean inodes or those only with unlogged -* changes (which we throw away, anyway). +* Clean inodes can be released immediately. Everything else has to go +* through xfs_iflush_abort() on journal commit as the flock +* synchronises removal of the inode from the cluster buffer against +* inode reclaim. */ -if (!ip->i_itemp ||xfs_inode_clean (ip)) { -ASSERT (ip != free_ip); +if (xfs_inode_clean (ip)) {xfs_ifunlock (ip); -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_no_inode; + goto out_iunlock; } -return ip; -out_rcu_unlock: -rcu_read_unlock (); -out_no_inode:
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2608
Quoted 10 lines
fs/xfs/xfs_inode.c +9
-1-return NULL ; +/* we have a dirty inode in memory that has not yet been flushed. */ +ASSERT (iip->ili_fields); +spin_lock (&iip->ili_lock); + iip->ili_last_fields = iip->ili_fields; + iip->ili_fields =0 ; + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); +ASSERT (iip->ili_last_fields);
We already asserted ->ili_fields and assigned ->ili_fields to ->ili_last_fields, so this assert seems spurious.
Brian
Quoted 117 lines
fs/xfs/xfs_inode.c +17
-61+ +out_iunlock: +if (ip != free_ip) +xfs_iunlock (ip,XFS_ILOCK_EXCL ); }/* @@ -2605,26 +2619,20 @@ xfs_ifree_get_one_inode(*/ STATIC intxfs_ifree_cluster ( -xfs_inode_t *free_ip, -xfs_trans_t *tp, +struct xfs_inode *free_ip, +struct xfs_trans *tp,struct xfs_icluster *xic) { -xfs_mount_t *mp = free_ip->i_mount; +struct xfs_mount *mp = free_ip->i_mount; +struct xfs_ino_geometry *igeo =M_IGEO (mp); +struct xfs_buf *bp; +xfs_daddr_t blkno; +xfs_ino_t inum = xic->first_ino;int nbufs;int i, j;int ioffset; -xfs_daddr_t blkno; -xfs_buf_t *bp; -xfs_inode_t *ip; -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip; -struct xfs_perag *pag; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -xfs_ino_t inum;int error; - inum = xic->first_ino; - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, inum)); nbufs = igeo->ialloc_blks / igeo->blocks_per_cluster;for (j =0 ; j < nbufs; j++, inum += igeo->inodes_per_cluster) {@@ -2668,59 +2676,16 @@ xfs_ifree_cluster( bp->b_ops = &xfs_inode_buf_ops;/* -* Walk the inodes already attached to the buffer and mark them -* stale. These will all have the flush locks held, so an -* in-memory inode walk can't lock them. By marking them all -* stale first, we will not attempt to lock them in the loop -* below as the XFS_ISTALE flag will be set. -*/ -list_for_each_entry (lip, &bp->b_li_list, li_bio_list) { -if (lip->li_type ==XFS_LI_INODE ) { - iip = (struct xfs_inode_log_item *)lip; -xfs_trans_ail_copy_lsn (mp->m_ail, - &iip->ili_flush_lsn, - &iip->ili_item.li_lsn); -xfs_iflags_set (iip->ili_inode,XFS_ISTALE ); - } - } - - -/* -* For each inode in memory attempt to add it to the inode -* buffer and set it up for being staled on buffer IO -* completion. This is safe as we've locked out tail pushing -* and flushing by locking the buffer. -* -* We have already marked every inode that was part of a -* transaction stale above, which means there is no point in -* even trying to lock them. +* Now we need to set all the cached clean inodes as XFS_ISTALE, +* too. This requires lookups, and will skip inodes that we've +* already marked XFS_ISTALE. */ -for (i =0 ; i < igeo->inodes_per_cluster; i++) { - ip =xfs_ifree_get_one_inode (pag, free_ip, inum + i); -if (!ip) -continue ; - - iip = ip->i_itemp; -spin_lock (&iip->ili_lock); - iip->ili_last_fields = iip->ili_fields; - iip->ili_fields =0 ; - iip->ili_fsync_fields =0 ; -spin_unlock (&iip->ili_lock); -xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, - &iip->ili_item.li_lsn); - -list_add_tail (&iip->ili_item.li_bio_list, - &bp->b_li_list); - -if (ip != free_ip) -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - } +for (i =0 ; i < igeo->inodes_per_cluster; i++) +xfs_ifree_mark_inode_stale (bp, free_ip, inum + i);xfs_trans_stale_inode_buf (tp, bp);xfs_trans_binval (tp, bp); } - -xfs_perag_put (pag);return 0 ; }@@ -3845,6 +3810,7 @@ xfs_iflush_int( iip->ili_fields =0 ; iip->ili_fsync_fields =0 ;spin_unlock (&iip->ili_lock); +ASSERT (iip->ili_last_fields);/* * Store the current LSN of the inode so that we can tell whether the Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 24/30] xfs: rework stale inodes in xfs_ifree_cluster
Quoted 47 lines · Brian Foster
Quoted 29 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Once we have inodes pinning the cluster buffer and attached whenever they are dirty, we no longer have a guarantee that the items are flush locked when we lock the cluster buffer. Hence we cannot just walk the buffer log item list and modify the attached inodes.
If the inode is not flush locked, we have to ILOCK it first and then flush lock it to do all the prerequisite checks needed to avoid races with other code. This is already handled by
xfs_ifree_get_one_inode(), so rework the inode iteration loop and function to update all inodes in cache whether they are attached to the buffer or not.Note: we also remove the copying of the log item lsn to the
ili_flush_lsnasxfs_iflush_done()now uses theXFS_ISTALEflag to trigger aborts and so flush lsn matching is not needed in IO completion for processing freed inodes.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 158 ++++++++++++++++++--------------------------- 1 file changed, 62 insertions(+), 96 deletions(-)fs/xfs/xfs_inode.c +0
-0...
Quoted 16 lines
fs/xfs/xfs_inode.c +1
-7@@ -2559,43 +2563,53 @@ xfs_ifree_get_one_inode(*/ if (ip != free_ip) {if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) { +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();delay (1 ); goto retry; } - -/* -* Check the inode number again in case we're racing with -* freeing in xfs_reclaim_inode(). See the comments in that -* function for more information as to why the initial check is -* not sufficient. -*/
Quoted 8 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2576
Quoted 4 lines
fs/xfs/xfs_inode.c +0
-4-if (ip->i_ino != inum) { -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_rcu_unlock; - }Why is the recheck under
ILOCK_EXCLno longer necessary? It looks like reclaim decides whether to proceed or not under the ilock and doesn't acquire the spinlock until it decides to reclaim. Hm?
Because we now take the ILOCK while still holding the i_flags_lock instead of dropping the spin lock then trying to get the ILOCK. Hence with this change, if we get the ILOCK we are guaranteed that the inode number has not changed and don't need to recheck it.
This is guaranteed by xfs_reclaim_inode() because it locks in the order of ILOCK -> i_flags_lock and it zeroes the ip->i_ino while holding both these locks. Hence if we've got the i_flags_lock and we try to get the ILOCK, the inode is either going to be valid and reclaim will skip the inode (because we hold locks) or the inode will already be in reclaim and the ip->i_ino will be zero....
Quoted 5 lines
Quoted 5 lines
fs/xfs/xfs_inode.c +2
-0} + ip->i_flags |=XFS_ISTALE ; +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();
Quoted 11 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2580
Quoted 7 lines
fs/xfs/xfs_inode.c +5
-2-xfs_iflock (ip); -xfs_iflags_set (ip,XFS_ISTALE ); +/* +* If we can't get the flush lock, the inode is already attached. All +* we needed to do here is mark the inode stale so buffer IO completion +* will remove it from the AIL. +*/ To make sure I'm following this correctly, we can assume the inode is attached based on an
iflock_nowait()failure because we hold the ilock, right?
Actually, because we hold the buffer lock. We only flush the inode to the buffer when we are holding the buffer lock, so all flush locking shold nest inside the buffer lock. So for taking the flock, the lock order is bp->b_sema -> ILOCK_EXCL -> iflock. We drop the flush lock before we drop the buffer lock in IO completion, and hence if we hold the buffer lock, nothing else can actually unlock the inode flush lock.
Quoted 3 lines
IOW, any other task doing a similar iflock check would have to do so under ilock and release the flush lock first if the inode didn't end up flushed, for whatever reason.
Yes, anything taking the flush lock needs to first hold the ILOCK - that's always been the case and we've always done that because the ILOCK is needed to provides serialisation against a) other modifications while we are accessing/flushing the inode, and b) inode reclaim.
/me checks.
After this patchset nothing calls xfs_iflock() at all - everything uses xfs_iflock_nowait(), so it might be time to turn this back into a plain status flag and get rid of the iflock stuff altogether as it's just a state flag now...
Quoted 11 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2608
Quoted 8 lines
fs/xfs/xfs_inode.c +8
-0+ASSERT (iip->ili_fields); +spin_lock (&iip->ili_lock); + iip->ili_last_fields = iip->ili_fields; + iip->ili_fields =0 ; + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); +ASSERT (iip->ili_last_fields);We already asserted ->ili_fields and assigned ->ili_fields to ->ili_last_fields, so this assert seems spurious.
Ah, the first ASSERT goes away in the next patch, I think. It was debug, and I may have removed it from the wrong patch...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Brian Foster
Reviewed-byRe: [PATCH 24/30] xfs: rework stale inodes in xfs_ifree_cluster
Quoted 62 lines · Dave Chinner
Quoted 47 lines · Brian Foster
Quoted 29 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Once we have inodes pinning the cluster buffer and attached whenever they are dirty, we no longer have a guarantee that the items are flush locked when we lock the cluster buffer. Hence we cannot just walk the buffer log item list and modify the attached inodes.
If the inode is not flush locked, we have to ILOCK it first and then flush lock it to do all the prerequisite checks needed to avoid races with other code. This is already handled by
xfs_ifree_get_one_inode(), so rework the inode iteration loop and function to update all inodes in cache whether they are attached to the buffer or not.Note: we also remove the copying of the log item lsn to the
ili_flush_lsnasxfs_iflush_done()now uses theXFS_ISTALEflag to trigger aborts and so flush lsn matching is not needed in IO completion for processing freed inodes.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 158 ++++++++++++++++++--------------------------- 1 file changed, 62 insertions(+), 96 deletions(-)fs/xfs/xfs_inode.c +0
-0...
Quoted 16 lines
fs/xfs/xfs_inode.c +1
-7@@ -2559,43 +2563,53 @@ xfs_ifree_get_one_inode(*/ if (ip != free_ip) {if (!xfs_ilock_nowait (ip,XFS_ILOCK_EXCL )) { +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();delay (1 ); goto retry; } - -/* -* Check the inode number again in case we're racing with -* freeing in xfs_reclaim_inode(). See the comments in that -* function for more information as to why the initial check is -* not sufficient. -*/ Quoted 8 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2576
Quoted 4 lines
fs/xfs/xfs_inode.c +0
-4-if (ip->i_ino != inum) { -xfs_iunlock (ip,XFS_ILOCK_EXCL ); - goto out_rcu_unlock; - }Why is the recheck under
ILOCK_EXCLno longer necessary? It looks like reclaim decides whether to proceed or not under the ilock and doesn't acquire the spinlock until it decides to reclaim. Hm?Because we now take the ILOCK while still holding the
i_flags_lockinstead of dropping the spin lock then trying to get the ILOCK. Hence with this change, if we get the ILOCK we are guaranteed that the inode number has not changed and don't need to recheck it.
Quoted 7 lines
This is guaranteed by
xfs_reclaim_inode()because it locks in the order of ILOCK ->i_flags_lockand it zeroes theip->i_inowhile holding both these locks. Hence if we've got thei_flags_lockand we try to get the ILOCK, the inode is either going to be valid and reclaim will skip the inode (because we hold locks) or the inode will already be in reclaim and theip->i_inowill be zero....
Got it, thanks.
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
fs/xfs/xfs_inode.c +2
-0} + ip->i_flags |=XFS_ISTALE ; +spin_unlock (&ip->i_flags_lock);rcu_read_unlock ();Quoted 11 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2580
Quoted 7 lines
fs/xfs/xfs_inode.c +5
-2-xfs_iflock (ip); -xfs_iflags_set (ip,XFS_ISTALE ); +/* +* If we can't get the flush lock, the inode is already attached. All +* we needed to do here is mark the inode stale so buffer IO completion +* will remove it from the AIL. +*/ To make sure I'm following this correctly, we can assume the inode is attached based on an
iflock_nowait()failure because we hold the ilock, right?
Quoted 8 lines
Actually, because we hold the buffer lock. We only flush the inode to the buffer when we are holding the buffer lock, so all flush locking shold nest inside the buffer lock. So for taking the flock, the lock order is
bp->b_sema->ILOCK_EXCL-> iflock. We drop the flush lock before we drop the buffer lock in IO completion, and hence if we hold the buffer lock, nothing else can actually unlock the inode flush lock.
I was thinking about xfs_reclaim_inode() again where we trylock the flush lock before checking clean state. There's no buffer involved in that path (as of this patch, at least), so afaict the ILOCK is what protects us in that general scenario.
Quoted 10 lines
Quoted 3 lines
IOW, any other task doing a similar iflock check would have to do so under ilock and release the flush lock first if the inode didn't end up flushed, for whatever reason.
Yes, anything taking the flush lock needs to first hold the ILOCK - that's always been the case and we've always done that because the ILOCK is needed to provides serialisation against a) other modifications while we are accessing/flushing the inode, and b) inode reclaim.
Right, Ok.
Quoted 7 lines
/me checks.
After this patchset nothing calls
xfs_iflock()at all - everything usesxfs_iflock_nowait(), so it might be time to turn this back into a plain status flag and get rid of the iflock stuff altogether as it's just a state flag now...
Yeah, I think that would be a nice follow on cleanup. The whole "if trylock fails, then assume we own the lock" type logic is really obscure and confusing to read IMO. Treating it as the "inode is flushed" state the lock fundamentally represents would be far more readable.
That addresses both my concerns, so with the nit below fixed:
Brian
Quoted 22 lines
Quoted 11 lines
fs/xfs/xfs_inode.c xfs_ifree_get_one_inode( · line 2608
Quoted 8 lines
fs/xfs/xfs_inode.c +8
-0+ASSERT (iip->ili_fields); +spin_lock (&iip->ili_lock); + iip->ili_last_fields = iip->ili_fields; + iip->ili_fields =0 ; + iip->ili_fsync_fields =0 ; +spin_unlock (&iip->ili_lock); +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); +ASSERT (iip->ili_last_fields);We already asserted ->ili_fields and assigned ->ili_fields to ->ili_last_fields, so this assert seems spurious.
Ah, the first ASSERT goes away in the next patch, I think. It was debug, and I may have removed it from the wrong patch...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 25/30 xfs: attach inodes to the cluster buffer when dirtied
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Rather than attach inodes to the cluster buffer just when we are doing IO, attach the inodes to the cluster buffer when they are dirtied. The means the buffer always carries a list of dirty inodes that reference it, and we can use that list to make more fundamental changes to inode writeback that aren't otherwise possible.
--- fs/xfs/libxfs/xfs_trans_inode.c | 9 ++++++--- fs/xfs/xfs_buf_item.c | 1 + fs/xfs/xfs_icache.c | 1 + fs/xfs/xfs_inode.c | 24 +++++------------------- fs/xfs/xfs_inode_item.c | 16 ++++++++++++++-- 5 files changed, 27 insertions(+), 24 deletions(-)
fs/xfs/libxfs/xfs_trans_inode.c +6 -3
@@ -163,13 +163,16 @@ xfs_trans_log_inode(/* * We need an explicit buffer reference for the log item but * don't want the buffer to remain attached to the transaction. -* Hold the buffer but release the transaction reference. +* Hold the buffer but release the transaction reference once +* we've attached the inode log item to the buffer log item +* list. */ xfs_buf_hold (bp); -xfs_trans_brelse (tp, bp); -spin_lock (&iip->ili_lock); iip->ili_item.li_buf = bp; + bp->b_flags |= _XBF_INODES; +list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); +xfs_trans_brelse (tp, bp); }/*
fs/xfs/xfs_buf_item.c +1 -0
@@ -465,6 +465,7 @@ xfs_buf_item_unpin(if (bip->bli_flags &XFS_BLI_STALE_INODE ) {xfs_buf_item_done (bp);xfs_iflush_done (bp); +ASSERT (list_empty (&bp->b_li_list)); }else {xfs_trans_ail_delete (lip,SHUTDOWN_LOG_IO_ERROR );xfs_buf_item_relse (bp);
fs/xfs/xfs_icache.c +1 -0
@@ -115,6 +115,7 @@ __xfs_inode_free( {/* asserts to verify all state is correct here */ ASSERT (atomic_read (&ip->i_pincount) ==0 ); +ASSERT (!ip->i_itemp ||list_empty (&ip->i_itemp->ili_item.li_bio_list));XFS_STATS_DEC (ip->i_mount, vn_active);call_rcu (&VFS_I (ip)->i_rcu, xfs_inode_free_callback);
fs/xfs/xfs_inode.c +5 -19
@@ -2584,27 +2584,24 @@ xfs_ifree_mark_inode_stale(ASSERT (iip->ili_last_fields); goto out_iunlock; } -ASSERT (!iip ||list_empty (&iip->ili_item.li_bio_list));/* -* Clean inodes can be released immediately. Everything else has to go -* through xfs_iflush_abort() on journal commit as the flock -* synchronises removal of the inode from the cluster buffer against -* inode reclaim. +* Inodes not attached to the buffer can be released immediately. +* Everything else has to go through xfs_iflush_abort() on journal +* commit as the flock synchronises removal of the inode from the +* cluster buffer against inode reclaim. */ -if (xfs_inode_clean (ip)) { +if (!iip ||list_empty (&iip->ili_item.li_bio_list)) {xfs_ifunlock (ip); goto out_iunlock; }/* we have a dirty inode in memory that has not yet been flushed. */ -ASSERT (iip->ili_fields);spin_lock (&iip->ili_lock); iip->ili_last_fields = iip->ili_fields; iip->ili_fields =0 ; iip->ili_fsync_fields =0 ;spin_unlock (&iip->ili_lock); -list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list);ASSERT (iip->ili_last_fields); out_iunlock:
@@ -3819,19 +3816,8 @@ xfs_iflush_int(xfs_trans_ail_copy_lsn (mp->m_ail, &iip->ili_flush_lsn, &iip->ili_item.li_lsn); -/* -* Attach the inode item callback to the buffer whether the flush -* succeeded or not. If not, the caller will shut down and fail I/O -* completion on the buffer to remove the inode from the AIL and release -* the flush lock. -*/ - bp->b_flags |= _XBF_INODES; -list_add_tail (&iip->ili_item.li_bio_list, &bp->b_li_list); -/* generate the checksum. */ xfs_dinode_calc_crc (mp, dip); - -ASSERT (!list_empty (&bp->b_li_list));return error; }
fs/xfs/xfs_inode_item.c +14 -2
@@ -660,6 +660,10 @@ xfs_inode_item_destroy(* list for other inodes that will run this function. We remove them from the * buffer list so we can process all the inode IO completions in one AIL lock * traversal. +* +* Note: Now that we attach the log item to the buffer when we first log the +* inode in memory, we can have unflushed inodes on the buffer list here. These +* inodes will have a zero ili_last_fields, so skip over them here. */ void xfs_iflush_done(
@@ -677,12 +681,15 @@ xfs_iflush_done(*/ list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { iip =INODE_ITEM (lip); +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -list_del_init (&lip->li_bio_list);xfs_iflush_abort (iip->ili_inode);continue ; } +if (!iip->ili_last_fields) +continue ; +list_move_tail (&lip->li_bio_list, &tmp);/* Do an unlocked check for needing the AIL lock. */
@@ -728,12 +735,16 @@ xfs_iflush_done(/* * Remove the reference to the cluster buffer if the inode is * clean in memory. Drop the buffer reference once we've dropped -* the locks we hold. +* the locks we hold. If the inode is dirty in memory, we need +* to put the inode item back on the buffer list for another +* pass through the flush machinery. */ ASSERT (iip->ili_item.li_buf == bp);if (!iip->ili_fields) { iip->ili_item.li_buf =NULL ; drop_buffer = true; + }else { +list_add (&lip->li_bio_list, &bp->b_li_list); } iip->ili_last_fields =0 ; iip->ili_flush_lsn =0 ;
@@ -777,6 +788,7 @@ xfs_iflush_abort( iip->ili_flush_lsn =0 ; bp = iip->ili_item.li_buf; iip->ili_item.li_buf =NULL ; +list_del_init (&iip->ili_item.li_bio_list);spin_unlock (&iip->ili_lock); }xfs_ifunlock (ip);
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 25/30] xfs: attach inodes to the cluster buffer when dirtied
Quoted 18 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Rather than attach inodes to the cluster buffer just when we are doing IO, attach the inodes to the cluster buffer when they are dirtied. The means the buffer always carries a list of dirty inodes that reference it, and we can use that list to make more fundamental changes to inode writeback that aren't otherwise possible.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/libxfs/xfs_trans_inode.c | 9 ++++++--- fs/xfs/xfs_buf_item.c | 1 + fs/xfs/xfs_icache.c | 1 + fs/xfs/xfs_inode.c | 24 +++++------------------- fs/xfs/xfs_inode_item.c | 16 ++++++++++++++-- 5 files changed, 27 insertions(+), 24 deletions(-)
...
Quoted 19 lines
fs/xfs/xfs_inode_item.c +4
-0@@ -660,6 +660,10 @@ xfs_inode_item_destroy(* list for other inodes that will run this function. We remove them from the * buffer list so we can process all the inode IO completions in one AIL lock * traversal. +* +* Note: Now that we attach the log item to the buffer when we first log the +* inode in memory, we can have unflushed inodes on the buffer list here. These +* inodes will have a zero ili_last_fields, so skip over them here. */ void xfs_iflush_done(@@ -677,12 +681,15 @@ xfs_iflush_done(*/ list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { iip =INODE_ITEM (lip);
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 692
Quoted 10 lines
fs/xfs/xfs_inode_item.c +4
-1+if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -list_del_init (&lip->li_bio_list);xfs_iflush_abort (iip->ili_inode);continue ; } +if (!iip->ili_last_fields) +continue ; +
If I follow the comment above this is essentially a proxy for checking whether the inode is flushed. IOW, could this eventually be replaced with the state flag check based on the cleanup discussed in the previous patch, right?
Otherwise LGTM:
Quoted 32 lines
fs/xfs/xfs_inode_item.c +6
-1list_move_tail (&lip->li_bio_list, &tmp);/* Do an unlocked check for needing the AIL lock. */ @@ -728,12 +735,16 @@ xfs_iflush_done(/* * Remove the reference to the cluster buffer if the inode is * clean in memory. Drop the buffer reference once we've dropped -* the locks we hold. +* the locks we hold. If the inode is dirty in memory, we need +* to put the inode item back on the buffer list for another +* pass through the flush machinery. */ ASSERT (iip->ili_item.li_buf == bp);if (!iip->ili_fields) { iip->ili_item.li_buf =NULL ; drop_buffer = true; + }else { +list_add (&lip->li_bio_list, &bp->b_li_list); } iip->ili_last_fields =0 ; iip->ili_flush_lsn =0 ;@@ -777,6 +788,7 @@ xfs_iflush_abort( iip->ili_flush_lsn =0 ; bp = iip->ili_item.li_buf; iip->ili_item.li_buf =NULL ; +list_del_init (&iip->ili_item.li_bio_list);spin_unlock (&iip->ili_lock); }xfs_ifunlock (ip);Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 25/30] xfs: attach inodes to the cluster buffer when dirtied
Quoted 46 lines · Brian Foster
Quoted 18 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Rather than attach inodes to the cluster buffer just when we are doing IO, attach the inodes to the cluster buffer when they are dirtied. The means the buffer always carries a list of dirty inodes that reference it, and we can use that list to make more fundamental changes to inode writeback that aren't otherwise possible.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/libxfs/xfs_trans_inode.c | 9 ++++++--- fs/xfs/xfs_buf_item.c | 1 + fs/xfs/xfs_icache.c | 1 + fs/xfs/xfs_inode.c | 24 +++++------------------- fs/xfs/xfs_inode_item.c | 16 ++++++++++++++-- 5 files changed, 27 insertions(+), 24 deletions(-)...
Quoted 26 lines
fs/xfs/xfs_inode_item.c +5
-1@@ -660,6 +660,10 @@ xfs_inode_item_destroy(* list for other inodes that will run this function. We remove them from the * buffer list so we can process all the inode IO completions in one AIL lock * traversal. +* +* Note: Now that we attach the log item to the buffer when we first log the +* inode in memory, we can have unflushed inodes on the buffer list here. These +* inodes will have a zero ili_last_fields, so skip over them here. */ void xfs_iflush_done(@@ -677,12 +681,15 @@ xfs_iflush_done(*/ list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { iip =INODE_ITEM (lip); +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -list_del_init (&lip->li_bio_list);xfs_iflush_abort (iip->ili_inode);continue ; }
Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 692
Quoted 3 lines
fs/xfs/xfs_inode_item.c +3
-0+if (!iip->ili_last_fields) +continue ; +If I follow the comment above this is essentially a proxy for checking whether the inode is flushed. IOW, could this eventually be replaced with the state flag check based on the cleanup discussed in the previous patch, right?
Yes, likely it can.
Quoted 3 lines
Otherwise LGTM:
Reviewed-by Brian Foster <[email protected]>
Thanks!
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 26/30 xfs: xfs_iflush() is no longer necessary
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now we have a cached buffer on inode log items, we don't need to do buffer lookups when flushing inodes anymore - all we need to do is lock the buffer and we are ready to go.
This largely gets rid of the need for xfs_iflush(), which is essentially just a mechanism to look up the buffer and flush the inode to it. Instead, we can just call xfs_iflush_cluster() with a few modifications to ensure it also flushes the inode we already hold locked.
This allows the AIL inode item pushing to be almost entirely non-blocking in XFS - we won't block unless memory allocation for the cluster inode lookup blocks or the block device queues are full.
Writeback during inode reclaim becomes a little more complex because we now have to lock the buffer ourselves, but otherwise this change is largely a functional no-op that removes a whole lot of code.
--- fs/xfs/xfs_inode.c | 106 ++++++---------------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 54 +++++++++----------- 3 files changed, 37 insertions(+), 125 deletions(-)
fs/xfs/xfs_inode.c +14 -92
@@ -3450,7 +3450,18 @@ xfs_rename(return error; } -STATIC int +/* +* Non-blocking flush of dirty inode metadata into the backing buffer. +* +* The caller must have a reference to the inode and hold the cluster buffer +* locked. The function will walk across all the inodes on the cluster buffer it +* can find and lock without blocking, and flush them to the cluster buffer. +* +* On success, the caller must write out the buffer returned in *bp and +* release it. On failure, the filesystem will be shut down, the buffer will +* have been unlocked and released, and EFSCORRUPTED will be returned. +*/ +int xfs_iflush_cluster (struct xfs_inode *ip,struct xfs_buf *bp)
@@ -3485,8 +3496,6 @@ xfs_iflush_cluster(for (i =0 ; i < nr_found; i++) { cip = cilist[i]; -if (cip == ip) -continue ;/* * because this is an RCU protected lookup, we could find a
@@ -3577,99 +3586,11 @@ xfs_iflush_cluster(kmem_free (cilist); out_put:xfs_perag_put (pag); -return error; -} - -/* -* Flush dirty inode metadata into the backing buffer. -* -* The caller must have the inode lock and the inode flush lock held. The -* inode lock will still be held upon return to the caller, and the inode -* flush lock will be released after the inode has reached the disk. -* -* The caller must write out the buffer returned in *bpp and release it. -*/ -int -xfs_iflush ( -struct xfs_inode *ip, -struct xfs_buf **bpp) -{ -struct xfs_mount *mp = ip->i_mount; -struct xfs_buf *bp =NULL ; -struct xfs_dinode *dip; -int error; - -XFS_STATS_INC (mp, xs_iflush_count); - -ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL |XFS_ILOCK_SHARED )); -ASSERT (xfs_isiflocked (ip)); -ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || - ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK )); - - *bpp =NULL ; - -xfs_iunpin_wait (ip); - -/* -* For stale inodes we cannot rely on the backing buffer remaining -* stale in cache for the remaining life of the stale inode and so -* xfs_imap_to_bp() below may give us a buffer that no longer contains -* inodes below. We have to check this after ensuring the inode is -* unpinned so that it is safe to reclaim the stale inode after the -* flush call. -*/ -if (xfs_iflags_test (ip,XFS_ISTALE )) { -xfs_ifunlock (ip); -return 0 ; - } - -/* -* Get the buffer containing the on-disk inode. We are doing a try-lock -* operation here, so we may get an EAGAIN error. In that case, return -* leaving the inode dirty. -* -* If we get any other error, we effectively have a corruption situation -* and we cannot flush the inode. Abort the flush and shut down. -*/ - error =xfs_imap_to_bp (mp,NULL , &ip->i_imap, &dip, &bp,XBF_TRYLOCK ); -if (error == -EAGAIN ) { -xfs_ifunlock (ip); -return error; - } -if (error) - goto abort; - -/* -* If the buffer is pinned then push on the log now so we won't -* get stuck waiting in the write for too long. -*/ -if (xfs_buf_ispinned (bp)) -xfs_log_force (mp,0 ); - -/* -* Flush the provided inode then attempt to gather others from the -* cluster into the write. -* -* Note: Once we attempt to flush an inode, we must run buffer -* completion callbacks on any failure. If this fails, simulate an I/O -* failure on the buffer and shut down. -*/ - error =xfs_iflush_int (ip, bp); -if (!error) - error =xfs_iflush_cluster (ip, bp);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp); - goto shutdown; +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); } - - *bpp = bp; -return 0 ; - -abort: -xfs_iflush_abort (ip); -shutdown: -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );return error; }
@@ -3688,6 +3609,7 @@ xfs_iflush_int(ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK ));ASSERT (iip !=NULL && iip->ili_fields !=0 ); +ASSERT (iip->ili_item.li_buf == bp); dip =xfs_buf_offset (bp, ip->i_imap.im_boffset);
fs/xfs/xfs_inode.h +1 -1
@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush (struct xfs_inode *,struct xfs_buf **); +int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);
fs/xfs/xfs_inode_item.c +22 -32
81 lines not on this page · raw message
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 26/30] xfs: xfs_iflush() is no longer necessary
Quoted 17 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now we have a cached buffer on inode log items, we don't need to do buffer lookups when flushing inodes anymore - all we need to do is lock the buffer and we are ready to go.
This largely gets rid of the need for
xfs_iflush(), which is essentially just a mechanism to look up the buffer and flush the inode to it. Instead, we can just callxfs_iflush_cluster()with a few modifications to ensure it also flushes the inode we already hold locked.This allows the AIL inode item pushing to be almost entirely non-blocking in XFS - we won't block unless memory allocation for the cluster inode lookup blocks or the block device queues are full.
Quoted 7 lines
Writeback during inode reclaim becomes a little more complex because we now have to lock the buffer ourselves, but otherwise this change is largely a functional no-op that removes a whole lot of code.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Looks mostly reasonable..
Quoted 9 lines
fs/xfs/xfs_inode.c | 106 ++++++---------------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 54 +++++++++----------- 3 files changed, 37 insertions(+), 125 deletions(-)fs/xfs/xfs_inode.c +0
-0
...
fs/xfs/xfs_inode.c xfs_iflush_int( · line 3612
Quoted 5 lines
fs/xfs/xfs_inode.c +1
-0@@ -3688,6 +3609,7 @@ xfs_iflush_int(ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK ));ASSERT (iip !=NULL && iip->ili_fields !=0 ); +ASSERT (iip->ili_item.li_buf == bp);
FWIW, the previous assert includes an iip NULL check.
Quoted 3 lines
fs/xfs/xfs_inode.c +0
-0dip =xfs_buf_offset (bp, ip->i_imap.im_boffset);
...
Quoted 8 lines
fs/xfs/xfs_inode_item.c +0
-0@@ -485,53 +485,42 @@ xfs_inode_item_push(uint rval =XFS_ITEM_SUCCESS ;int error;
fs/xfs/xfs_inode_item.c xfs_inode_item_push( · line 494
Quoted 10 lines
fs/xfs/xfs_inode_item.c +5
-3-if (xfs_ipincount (ip) >0 ) +ASSERT (iip->ili_item.li_buf); + +if (xfs_ipincount (ip) >0 ||xfs_buf_ispinned (bp) || + (ip->i_flags &XFS_ISTALE ))return XFS_ITEM_PINNED ; -if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) -return XFS_ITEM_LOCKED ; +/* If the inode is already flush locked, we're already flushing. */
Or we're racing with reclaim (since we don't have the ilock here any longer)?
Quoted 13 lines
fs/xfs/xfs_inode_item.c +4
-8+if (xfs_isiflocked (ip)) +return XFS_ITEM_FLUSHING ; -/* -* Re-check the pincount now that we stabilized the value by -* taking the ilock. -*/ -if (xfs_ipincount (ip) >0 ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; - } +if (!xfs_buf_trylock (bp)) +return XFS_ITEM_LOCKED ;
fs/xfs/xfs_inode_item.c xfs_inode_item_push( · line 503
Quoted 10 lines
fs/xfs/xfs_inode_item.c +3
-6-/* -* Stale inode items should force out the iclog. -*/ -if (ip->i_flags &XFS_ISTALE ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; +if (bp->b_flags & _XBF_DELWRI_Q) { +xfs_buf_unlock (bp); +return XFS_ITEM_FLUSHING ;
Hmm, what's the purpose of this check? I would expect that we'd still be able to flush to a buffer even though it's delwri queued. We drop the buffer lock after queueing it (and then it's reacquired on delwri submit).
Quoted 39 lines
fs/xfs/xfs_inode_item.c +9
-15} +spin_unlock (&lip->li_ailp->ail_lock);/* -* Someone else is already flushing the inode. Nothing we can do -* here but wait for the flush to finish and remove the item from -* the AIL. +* We need to hold a reference for flushing the cluster buffer as it may +* fail the buffer without IO submission. In which case, we better get a +* reference for that completion because otherwise we don't get a +* reference for IO until we queue the buffer for delwri submission. */ -if (!xfs_iflock_nowait (ip)) { - rval =XFS_ITEM_FLUSHING ; - goto out_unlock; - } - -ASSERT (iip->ili_fields !=0 ||XFS_FORCED_SHUTDOWN (ip->i_mount)); -spin_unlock (&lip->li_ailp->ail_lock); - - error =xfs_iflush (ip, &bp); +xfs_buf_hold (bp); + error =xfs_iflush_cluster (ip, bp);if (!error) {if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - }else if (error == -EAGAIN ) + }else { rval =XFS_ITEM_LOCKED ; + }spin_lock (&lip->li_ailp->ail_lock); -out_unlock: -xfs_iunlock (ip,XFS_ILOCK_SHARED );return rval; }
fs/xfs/xfs_inode_item.c xfs_inode_item_release( · line 540
Quoted 4 lines
fs/xfs/xfs_inode_item.c +1
-0@@ -548,6 +537,7 @@ xfs_inode_item_release(ASSERT (ip->i_itemp !=NULL );ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL )); +ASSERT (lip->li_buf || !test_bit (XFS_LI_DIRTY , &lip->li_flags));
This is the transaction cancel/abort path, so seems like this should be part of the patch that attaches the ili on logging the inode?
Brian
Quoted 6 lines
fs/xfs/xfs_inode_item.c +0
-0lock_flags = iip->ili_lock_flags; iip->ili_lock_flags =0 ;Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 26/30] xfs: xfs_iflush() is no longer necessary
Quoted 38 lines · Brian Foster
Quoted 17 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now we have a cached buffer on inode log items, we don't need to do buffer lookups when flushing inodes anymore - all we need to do is lock the buffer and we are ready to go.
This largely gets rid of the need for
xfs_iflush(), which is essentially just a mechanism to look up the buffer and flush the inode to it. Instead, we can just callxfs_iflush_cluster()with a few modifications to ensure it also flushes the inode we already hold locked.This allows the AIL inode item pushing to be almost entirely non-blocking in XFS - we won't block unless memory allocation for the cluster inode lookup blocks or the block device queues are full.
Quoted 7 lines
Writeback during inode reclaim becomes a little more complex because we now have to lock the buffer ourselves, but otherwise this change is largely a functional no-op that removes a whole lot of code.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Looks mostly reasonable..
Quoted 9 lines
fs/xfs/xfs_inode.c | 106 ++++++---------------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 54 +++++++++----------- 3 files changed, 37 insertions(+), 125 deletions(-)fs/xfs/xfs_inode.c +0
-0...
Quoted 7 lines
fs/xfs/xfs_inode.c xfs_iflush_int( · line 3612
Quoted 5 lines
fs/xfs/xfs_inode.c +1
-0@@ -3688,6 +3609,7 @@ xfs_iflush_int(ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK ));ASSERT (iip !=NULL && iip->ili_fields !=0 ); +ASSERT (iip->ili_item.li_buf == bp);FWIW, the previous assert includes an iip NULL check.
If iip is NULL in this function, we are going to oops anyway. We've slowly been weeding these types of "<x> must not be null" asserts out of the code because it doesn't add any value. Especially as the assert doesn't tell you exactly what failed and that's kinda important here.
FWIW, this assert is largely redundant, anyway, because it's just an open coded xfs_inode_clean() check, and we check that in all cases before calling xfs_iflush_int(). So I'd be tempted to remove it if it really mattered...
Quoted 20 lines
Quoted 4 lines
fs/xfs/xfs_inode.c +0
-0dip =xfs_buf_offset (bp, ip->i_imap.im_boffset);...
Quoted 15 lines
fs/xfs/xfs_inode_item.c +4
-1@@ -485,53 +485,42 @@ xfs_inode_item_push(uint rval =XFS_ITEM_SUCCESS ;int error; -if (xfs_ipincount (ip) >0 ) +ASSERT (iip->ili_item.li_buf); + +if (xfs_ipincount (ip) >0 ||xfs_buf_ispinned (bp) || + (ip->i_flags &XFS_ISTALE ))return XFS_ITEM_PINNED ;
Quoted 6 lines
fs/xfs/xfs_inode_item.c xfs_inode_item_push( · line 494
Quoted 3 lines
fs/xfs/xfs_inode_item.c +1
-2-if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) -return XFS_ITEM_LOCKED ; +/* If the inode is already flush locked, we're already flushing. */ Or we're racing with reclaim (since we don't have the ilock here any longer)?
Possible, but unlikely. The 99.999% case here is the inode was flush locked by a push of another item on the same cluster buffer, and that happens so often it's worth adding a racy check ahead of the flushing to avoid trying to lock the cluster buffer...
Quoted 12 lines
Quoted 12 lines
fs/xfs/xfs_inode_item.c +2
-8-/* -* Re-check the pincount now that we stabilized the value by -* taking the ilock. -*/ -if (xfs_ipincount (ip) >0 ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; - } +if (!xfs_buf_trylock (bp)) +return XFS_ITEM_LOCKED ;
Quoted 14 lines
fs/xfs/xfs_inode_item.c xfs_inode_item_push( · line 503
Quoted 9 lines
fs/xfs/xfs_inode_item.c +3
-6-/* -* Stale inode items should force out the iclog. -*/ -if (ip->i_flags &XFS_ISTALE ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; +if (bp->b_flags & _XBF_DELWRI_Q) { +xfs_buf_unlock (bp); +return XFS_ITEM_FLUSHING ;Hmm, what's the purpose of this check? I would expect that we'd still be able to flush to a buffer even though it's delwri queued. We drop the buffer lock after queueing it (and then it's reacquired on delwri submit).
Yes, we can. I did this when tracking down whacky inode flushing issues and AIL deadlocks this patch exposed to simplify the behaviour, and I'd completely forgotten about it once I'd found the underlying problem a week and a half later. It seems harmless at this point, I'll see what happens if I remove it...
Quoted 5 lines
Quoted 5 lines
fs/xfs/xfs_inode_item.c +0
-2-out_unlock: -xfs_iunlock (ip,XFS_ILOCK_SHARED );return rval; }
Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_inode_item_release( · line 540
Quoted 5 lines
fs/xfs/xfs_inode_item.c +1
-0@@ -548,6 +537,7 @@ xfs_inode_item_release(ASSERT (ip->i_itemp !=NULL );ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL )); +ASSERT (lip->li_buf || !test_bit (XFS_LI_DIRTY , &lip->li_flags));This is the transaction cancel/abort path, so seems like this should be part of the patch that attaches the ili on logging the inode?
Again, likely left over from debugging some whacky inode flushing/writeback/completion issue this patch exposed. I'll remove it...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 26/30 xfs: xfs_iflush() is no longer necessary
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now we have a cached buffer on inode log items, we don't need to do buffer lookups when flushing inodes anymore - all we need to do is lock the buffer and we are ready to go.
This largely gets rid of the need for xfs_iflush(), which is essentially just a mechanism to look up the buffer and flush the inode to it. Instead, we can just call xfs_iflush_cluster() with a few modifications to ensure it also flushes the inode we already hold locked.
This allows the AIL inode item pushing to be almost entirely non-blocking in XFS - we won't block unless memory allocation for the cluster inode lookup blocks or the block device queues are full.
Writeback during inode reclaim becomes a little more complex because we now have to lock the buffer ourselves, but otherwise this change is largely a functional no-op that removes a whole lot of code.
---
Version 2
- removed unnecessary asserts
- removed buffer delwri queue check from xfs_inode_item_push()
fs/xfs/xfs_inode.c | 107 +++++++----------------------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 51 ++++++++--------------- 3 files changed, 33 insertions(+), 127 deletions(-)
fs/xfs/xfs_inode.c +14 -93
@@ -3450,7 +3450,18 @@ xfs_rename(return error; } -STATIC int +/* +* Non-blocking flush of dirty inode metadata into the backing buffer. +* +* The caller must have a reference to the inode and hold the cluster buffer +* locked. The function will walk across all the inodes on the cluster buffer it +* can find and lock without blocking, and flush them to the cluster buffer. +* +* On success, the caller must write out the buffer returned in *bp and +* release it. On failure, the filesystem will be shut down, the buffer will +* have been unlocked and released, and EFSCORRUPTED will be returned. +*/ +int xfs_iflush_cluster (struct xfs_inode *ip,struct xfs_buf *bp)
@@ -3485,8 +3496,6 @@ xfs_iflush_cluster(for (i =0 ; i < nr_found; i++) { cip = cilist[i]; -if (cip == ip) -continue ;/* * because this is an RCU protected lookup, we could find a
@@ -3577,99 +3586,11 @@ xfs_iflush_cluster(kmem_free (cilist); out_put:xfs_perag_put (pag); -return error; -} - -/* -* Flush dirty inode metadata into the backing buffer. -* -* The caller must have the inode lock and the inode flush lock held. The -* inode lock will still be held upon return to the caller, and the inode -* flush lock will be released after the inode has reached the disk. -* -* The caller must write out the buffer returned in *bpp and release it. -*/ -int -xfs_iflush ( -struct xfs_inode *ip, -struct xfs_buf **bpp) -{ -struct xfs_mount *mp = ip->i_mount; -struct xfs_buf *bp =NULL ; -struct xfs_dinode *dip; -int error; - -XFS_STATS_INC (mp, xs_iflush_count); - -ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL |XFS_ILOCK_SHARED )); -ASSERT (xfs_isiflocked (ip)); -ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || - ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK )); - - *bpp =NULL ; - -xfs_iunpin_wait (ip); - -/* -* For stale inodes we cannot rely on the backing buffer remaining -* stale in cache for the remaining life of the stale inode and so -* xfs_imap_to_bp() below may give us a buffer that no longer contains -* inodes below. We have to check this after ensuring the inode is -* unpinned so that it is safe to reclaim the stale inode after the -* flush call. -*/ -if (xfs_iflags_test (ip,XFS_ISTALE )) { -xfs_ifunlock (ip); -return 0 ; - } - -/* -* Get the buffer containing the on-disk inode. We are doing a try-lock -* operation here, so we may get an EAGAIN error. In that case, return -* leaving the inode dirty. -* -* If we get any other error, we effectively have a corruption situation -* and we cannot flush the inode. Abort the flush and shut down. -*/ - error =xfs_imap_to_bp (mp,NULL , &ip->i_imap, &dip, &bp,XBF_TRYLOCK ); -if (error == -EAGAIN ) { -xfs_ifunlock (ip); -return error; - } -if (error) - goto abort; - -/* -* If the buffer is pinned then push on the log now so we won't -* get stuck waiting in the write for too long. -*/ -if (xfs_buf_ispinned (bp)) -xfs_log_force (mp,0 ); - -/* -* Flush the provided inode then attempt to gather others from the -* cluster into the write. -* -* Note: Once we attempt to flush an inode, we must run buffer -* completion callbacks on any failure. If this fails, simulate an I/O -* failure on the buffer and shut down. -*/ - error =xfs_iflush_int (ip, bp); -if (!error) - error =xfs_iflush_cluster (ip, bp);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp); - goto shutdown; +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); } - - *bpp = bp; -return 0 ; - -abort: -xfs_iflush_abort (ip); -shutdown: -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );return error; }
@@ -3687,7 +3608,7 @@ xfs_iflush_int(ASSERT (xfs_isiflocked (ip));ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK )); -ASSERT (iip !=NULL && iip->ili_fields !=0 ); +ASSERT (iip->ili_item.li_buf == bp); dip =xfs_buf_offset (bp, ip->i_imap.im_boffset);
fs/xfs/xfs_inode.h +1 -1
@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush (struct xfs_inode *,struct xfs_buf **); +int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);
fs/xfs/xfs_inode_item.c +18 -33
71 lines not on this page · raw message
Re: [PATCH 26/30 V2] xfs: xfs_iflush() is no longer necessary
Quoted 22 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now we have a cached buffer on inode log items, we don't need to do buffer lookups when flushing inodes anymore - all we need to do is lock the buffer and we are ready to go.
This largely gets rid of the need for
xfs_iflush(), which is essentially just a mechanism to look up the buffer and flush the inode to it. Instead, we can just callxfs_iflush_cluster()with a few modifications to ensure it also flushes the inode we already hold locked.This allows the AIL inode item pushing to be almost entirely non-blocking in XFS - we won't block unless memory allocation for the cluster inode lookup blocks or the block device queues are full.
Writeback during inode reclaim becomes a little more complex because we now have to lock the buffer ourselves, but otherwise this change is largely a functional no-op that removes a whole lot of code.
Quoted 7 lines
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Version 2
- removed unnecessary asserts
- removed buffer delwri queue check fromxfs_inode_item_push()
Quoted 238 lines
fs/xfs/xfs_inode.c | 107 +++++++----------------------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 51 ++++++++--------------- 3 files changed, 33 insertions(+), 127 deletions(-)fs/xfs/xfs_inode.c +14
-93@@ -3450,7 +3450,18 @@ xfs_rename(return error; } -STATIC int +/* +* Non-blocking flush of dirty inode metadata into the backing buffer. +* +* The caller must have a reference to the inode and hold the cluster buffer +* locked. The function will walk across all the inodes on the cluster buffer it +* can find and lock without blocking, and flush them to the cluster buffer. +* +* On success, the caller must write out the buffer returned in *bp and +* release it. On failure, the filesystem will be shut down, the buffer will +* have been unlocked and released, and EFSCORRUPTED will be returned. +*/ +int xfs_iflush_cluster (struct xfs_inode *ip,struct xfs_buf *bp)@@ -3485,8 +3496,6 @@ xfs_iflush_cluster(for (i =0 ; i < nr_found; i++) { cip = cilist[i]; -if (cip == ip) -continue ;/* * because this is an RCU protected lookup, we could find a @@ -3577,99 +3586,11 @@ xfs_iflush_cluster(kmem_free (cilist); out_put:xfs_perag_put (pag); -return error; -} - -/* -* Flush dirty inode metadata into the backing buffer. -* -* The caller must have the inode lock and the inode flush lock held. The -* inode lock will still be held upon return to the caller, and the inode -* flush lock will be released after the inode has reached the disk. -* -* The caller must write out the buffer returned in *bpp and release it. -*/ -int -xfs_iflush ( -struct xfs_inode *ip, -struct xfs_buf **bpp) -{ -struct xfs_mount *mp = ip->i_mount; -struct xfs_buf *bp =NULL ; -struct xfs_dinode *dip; -int error; - -XFS_STATS_INC (mp, xs_iflush_count); - -ASSERT (xfs_isilocked (ip,XFS_ILOCK_EXCL |XFS_ILOCK_SHARED )); -ASSERT (xfs_isiflocked (ip)); -ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || - ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK )); - - *bpp =NULL ; - -xfs_iunpin_wait (ip); - -/* -* For stale inodes we cannot rely on the backing buffer remaining -* stale in cache for the remaining life of the stale inode and so -* xfs_imap_to_bp() below may give us a buffer that no longer contains -* inodes below. We have to check this after ensuring the inode is -* unpinned so that it is safe to reclaim the stale inode after the -* flush call. -*/ -if (xfs_iflags_test (ip,XFS_ISTALE )) { -xfs_ifunlock (ip); -return 0 ; - } - -/* -* Get the buffer containing the on-disk inode. We are doing a try-lock -* operation here, so we may get an EAGAIN error. In that case, return -* leaving the inode dirty. -* -* If we get any other error, we effectively have a corruption situation -* and we cannot flush the inode. Abort the flush and shut down. -*/ - error =xfs_imap_to_bp (mp,NULL , &ip->i_imap, &dip, &bp,XBF_TRYLOCK ); -if (error == -EAGAIN ) { -xfs_ifunlock (ip); -return error; - } -if (error) - goto abort; - -/* -* If the buffer is pinned then push on the log now so we won't -* get stuck waiting in the write for too long. -*/ -if (xfs_buf_ispinned (bp)) -xfs_log_force (mp,0 ); - -/* -* Flush the provided inode then attempt to gather others from the -* cluster into the write. -* -* Note: Once we attempt to flush an inode, we must run buffer -* completion callbacks on any failure. If this fails, simulate an I/O -* failure on the buffer and shut down. -*/ - error =xfs_iflush_int (ip, bp); -if (!error) - error =xfs_iflush_cluster (ip, bp);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp); - goto shutdown; +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); } - - *bpp = bp; -return 0 ; - -abort: -xfs_iflush_abort (ip); -shutdown: -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE );return error; }@@ -3687,7 +3608,7 @@ xfs_iflush_int(ASSERT (xfs_isiflocked (ip));ASSERT (ip->i_df.if_format !=XFS_DINODE_FMT_BTREE || ip->i_df.if_nextents >XFS_IFORK_MAXEXT (ip,XFS_DATA_FORK )); -ASSERT (iip !=NULL && iip->ili_fields !=0 ); +ASSERT (iip->ili_item.li_buf == bp); dip =xfs_buf_offset (bp, ip->i_imap.im_boffset);fs/xfs/xfs_inode.h +1
-1@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush (struct xfs_inode *,struct xfs_buf **); +int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);fs/xfs/xfs_inode_item.c +18
-33@@ -485,53 +485,38 @@ xfs_inode_item_push(uint rval =XFS_ITEM_SUCCESS ;int error; -if (xfs_ipincount (ip) >0 ) +ASSERT (iip->ili_item.li_buf); + +if (xfs_ipincount (ip) >0 ||xfs_buf_ispinned (bp) || + (ip->i_flags &XFS_ISTALE ))return XFS_ITEM_PINNED ; -if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) -return XFS_ITEM_LOCKED ; +/* If the inode is already flush locked, we're already flushing. */ +if (xfs_isiflocked (ip)) +return XFS_ITEM_FLUSHING ; -/* -* Re-check the pincount now that we stabilized the value by -* taking the ilock. -*/ -if (xfs_ipincount (ip) >0 ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; - } +if (!xfs_buf_trylock (bp)) +return XFS_ITEM_LOCKED ; -/* -* Stale inode items should force out the iclog. -*/ -if (ip->i_flags &XFS_ISTALE ) { - rval =XFS_ITEM_PINNED ; - goto out_unlock; - } +spin_unlock (&lip->li_ailp->ail_lock);/* -* Someone else is already flushing the inode. Nothing we can do -* here but wait for the flush to finish and remove the item from -* the AIL. +* We need to hold a reference for flushing the cluster buffer as it may +* fail the buffer without IO submission. In which case, we better get a +* reference for that completion because otherwise we don't get a +* reference for IO until we queue the buffer for delwri submission. */ -if (!xfs_iflock_nowait (ip)) { - rval =XFS_ITEM_FLUSHING ; - goto out_unlock; - } - -ASSERT (iip->ili_fields !=0 ||XFS_FORCED_SHUTDOWN (ip->i_mount)); -spin_unlock (&lip->li_ailp->ail_lock); - - error =xfs_iflush (ip, &bp); +xfs_buf_hold (bp); + error =xfs_iflush_cluster (ip, bp);if (!error) {if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); - }else if (error == -EAGAIN ) + }else { rval =XFS_ITEM_LOCKED ; + }spin_lock (&lip->li_ailp->ail_lock); -out_unlock: -xfs_iunlock (ip,XFS_ILOCK_SHARED );return rval; }
PATCH 27/30 xfs: rename xfs_iflush_int()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
with xfs_iflush() gone, we can rename xfs_iflush_int() back to xfs_iflush(). Also move it up above xfs_iflush_cluster() so we don't need the forward definition any more.
--- fs/xfs/xfs_inode.c | 293 ++++++++++++++++++++++----------------------- 1 file changed, 146 insertions(+), 147 deletions(-)
fs/xfs/xfs_inode.c +146 -147
@@ -44,7 +44,6 @@ kmem_zone_t *xfs_inode_zone;*/ #define XFS_ITRUNC_MAX_EXTENTS 2 -STATIC int xfs_iflush_int(struct xfs_inode *, struct xfs_buf *); STATIC int xfs_iunlink(struct xfs_trans *, struct xfs_inode *); STATIC int xfs_iunlink_remove(struct xfs_trans *, struct xfs_inode *);
@@ -3450,152 +3449,8 @@ xfs_rename(return error; } -/* -* Non-blocking flush of dirty inode metadata into the backing buffer. -* -* The caller must have a reference to the inode and hold the cluster buffer -* locked. The function will walk across all the inodes on the cluster buffer it -* can find and lock without blocking, and flush them to the cluster buffer. -* -* On success, the caller must write out the buffer returned in *bp and -* release it. On failure, the filesystem will be shut down, the buffer will -* have been unlocked and released, and EFSCORRUPTED will be returned. -*/ -int -xfs_iflush_cluster ( -struct xfs_inode *ip, -struct xfs_buf *bp) -{ -struct xfs_mount *mp = ip->i_mount; -struct xfs_perag *pag; -unsigned long first_index, mask; -int cilist_size; -struct xfs_inode **cilist; -struct xfs_inode *cip; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -int error =0 ; -int nr_found; -int clcount =0 ; -int i; - - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); - - cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); - cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); -if (!cilist) - goto out_put; - - mask = ~(igeo->inodes_per_cluster -1 ); - first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; -rcu_read_lock (); -/* really need a gang lookup range call here */ - nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, - first_index, igeo->inodes_per_cluster); -if (nr_found ==0 ) - goto out_free; - -for (i =0 ; i < nr_found; i++) { - cip = cilist[i]; - -/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. -*/ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); -continue ; - } - -/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. -*/ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; - } -spin_unlock (&cip->i_flags_lock); - -/* -* Do an un-protected check to see if the inode is dirty and -* is a candidate for flushing. These checks will be repeated -* later after the appropriate locks are acquired. -*/ -if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) -continue ; - -/* -* Try to get locks. If any are unavailable or it is pinned, -* then this inode cannot be flushed and is skipped. -*/ - -if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) -continue ; -if (!xfs_iflock_nowait (cip)) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } -if (xfs_ipincount (cip)) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } - - -/* -* Check the inode number again, just to be certain we are not -* racing with freeing in xfs_reclaim_inode(). See the comments -* in that function for more information as to why the initial -* check is not sufficient. -*/ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } - -/* -* arriving here means that this inode can be flushed. First -* re-check that it's dirty before flushing. -*/ -if (!xfs_inode_clean (cip)) { - error =xfs_iflush_int (cip, bp); -if (error) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto out_free; - } - clcount++; - }else { -xfs_ifunlock (cip); - } -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - } - -if (clcount) { -XFS_STATS_INC (mp, xs_icluster_flushcnt); -XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); - } - -out_free: -rcu_read_unlock (); -kmem_free (cilist); -out_put: -xfs_perag_put (pag); -if (error) { - bp->b_flags |=XBF_ASYNC ; -xfs_buf_ioend_fail (bp); -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); - } -return error; -} - -STATIC int -xfs_iflush_int ( +static int +xfs_iflush (struct xfs_inode *ip,struct xfs_buf *bp) {
@@ -3743,6 +3598,150 @@ xfs_iflush_int(return error; } +/* +* Non-blocking flush of dirty inode metadata into the backing buffer. +* +* The caller must have a reference to the inode and hold the cluster buffer +* locked. The function will walk across all the inodes on the cluster buffer it +* can find and lock without blocking, and flush them to the cluster buffer. +* +* On success, the caller must write out the buffer returned in *bp and +* release it. On failure, the filesystem will be shut down, the buffer will +* have been unlocked and released, and EFSCORRUPTED will be returned. +*/ +int +xfs_iflush_cluster ( +struct xfs_inode *ip, +struct xfs_buf *bp) +{ +struct xfs_mount *mp = ip->i_mount; +struct xfs_perag *pag; +unsigned long first_index, mask; +int cilist_size; +struct xfs_inode **cilist; +struct xfs_inode *cip; +struct xfs_ino_geometry *igeo =M_IGEO (mp); +int error =0 ; +int nr_found; +int clcount =0 ; +int i; + + pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); + + cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); + cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); +if (!cilist) + goto out_put; + + mask = ~(igeo->inodes_per_cluster -1 ); + first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; +rcu_read_lock (); +/* really need a gang lookup range call here */ + nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, + first_index, igeo->inodes_per_cluster); +if (nr_found ==0 ) + goto out_free; + +for (i =0 ; i < nr_found; i++) { + cip = cilist[i]; + +/* +* because this is an RCU protected lookup, we could find a +* recently freed or even reallocated inode during the lookup. +* We need to check under the i_flags_lock for a valid inode +* here. Skip it if it is not valid or the wrong inode. +*/ +spin_lock (&cip->i_flags_lock); +if (!cip->i_ino || +__xfs_iflags_test (cip,XFS_ISTALE )) { +spin_unlock (&cip->i_flags_lock); +continue ; + } + +/* +* Once we fall off the end of the cluster, no point checking +* any more inodes in the list because they will also all be +* outside the cluster. +*/ +if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { +spin_unlock (&cip->i_flags_lock); +break ; + } +spin_unlock (&cip->i_flags_lock); + +/* +* Do an un-protected check to see if the inode is dirty and +* is a candidate for flushing. These checks will be repeated +* later after the appropriate locks are acquired. +*/ +if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) +continue ; + +/* +* Try to get locks. If any are unavailable or it is pinned, +* then this inode cannot be flushed and is skipped. +*/ + +if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) +continue ; +if (!xfs_iflock_nowait (cip)) { +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } +if (xfs_ipincount (cip)) { +xfs_ifunlock (cip); +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } + + +/* +* Check the inode number again, just to be certain we are not +* racing with freeing in xfs_reclaim_inode(). See the comments +* in that function for more information as to why the initial +* check is not sufficient. +*/ +if (!cip->i_ino) { +xfs_ifunlock (cip); +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } + +/* +* arriving here means that this inode can be flushed. First +* re-check that it's dirty before flushing. +*/ +if (!xfs_inode_clean (cip)) { + error =xfs_iflush (cip, bp); +if (error) { +xfs_iunlock (cip,XFS_ILOCK_SHARED ); + goto out_free; + } + clcount++; + }else { +xfs_ifunlock (cip); + } +xfs_iunlock (cip,XFS_ILOCK_SHARED ); + } + +if (clcount) { +XFS_STATS_INC (mp, xs_icluster_flushcnt); +XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); + } + +out_free: +rcu_read_unlock (); +kmem_free (cilist); +out_put: +xfs_perag_put (pag); +if (error) { + bp->b_flags |=XBF_ASYNC ; +xfs_buf_ioend_fail (bp); +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); + } +return error; +} +/* Release an inode. */ void xfs_irele(
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 27/30] xfs: rename xfs_iflush_int()
Quoted 10 lines · Dave Chinner
From: Dave Chinner <[email protected]>
with
xfs_iflush()gone, we can renamexfs_iflush_int()back toxfs_iflush(). Also move it up abovexfs_iflush_cluster()so we don't need the forward definition any more.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Amir Goldstein <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Quoted 324 lines
fs/xfs/xfs_inode.c | 293 ++++++++++++++++++++++----------------------- 1 file changed, 146 insertions(+), 147 deletions(-)fs/xfs/xfs_inode.c +146
-147@@ -44,7 +44,6 @@ kmem_zone_t *xfs_inode_zone;*/ #define XFS_ITRUNC_MAX_EXTENTS 2 -STATIC int xfs_iflush_int(struct xfs_inode *, struct xfs_buf *); STATIC int xfs_iunlink(struct xfs_trans *, struct xfs_inode *); STATIC int xfs_iunlink_remove(struct xfs_trans *, struct xfs_inode *);@@ -3450,152 +3449,8 @@ xfs_rename(return error; } -/* -* Non-blocking flush of dirty inode metadata into the backing buffer. -* -* The caller must have a reference to the inode and hold the cluster buffer -* locked. The function will walk across all the inodes on the cluster buffer it -* can find and lock without blocking, and flush them to the cluster buffer. -* -* On success, the caller must write out the buffer returned in *bp and -* release it. On failure, the filesystem will be shut down, the buffer will -* have been unlocked and released, and EFSCORRUPTED will be returned. -*/ -int -xfs_iflush_cluster ( -struct xfs_inode *ip, -struct xfs_buf *bp) -{ -struct xfs_mount *mp = ip->i_mount; -struct xfs_perag *pag; -unsigned long first_index, mask; -int cilist_size; -struct xfs_inode **cilist; -struct xfs_inode *cip; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -int error =0 ; -int nr_found; -int clcount =0 ; -int i; - - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); - - cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); - cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); -if (!cilist) - goto out_put; - - mask = ~(igeo->inodes_per_cluster -1 ); - first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; -rcu_read_lock (); -/* really need a gang lookup range call here */ - nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, - first_index, igeo->inodes_per_cluster); -if (nr_found ==0 ) - goto out_free; - -for (i =0 ; i < nr_found; i++) { - cip = cilist[i]; - -/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. -*/ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); -continue ; - } - -/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. -*/ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; - } -spin_unlock (&cip->i_flags_lock); - -/* -* Do an un-protected check to see if the inode is dirty and -* is a candidate for flushing. These checks will be repeated -* later after the appropriate locks are acquired. -*/ -if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) -continue ; - -/* -* Try to get locks. If any are unavailable or it is pinned, -* then this inode cannot be flushed and is skipped. -*/ - -if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) -continue ; -if (!xfs_iflock_nowait (cip)) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } -if (xfs_ipincount (cip)) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } - - -/* -* Check the inode number again, just to be certain we are not -* racing with freeing in xfs_reclaim_inode(). See the comments -* in that function for more information as to why the initial -* check is not sufficient. -*/ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } - -/* -* arriving here means that this inode can be flushed. First -* re-check that it's dirty before flushing. -*/ -if (!xfs_inode_clean (cip)) { - error =xfs_iflush_int (cip, bp); -if (error) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto out_free; - } - clcount++; - }else { -xfs_ifunlock (cip); - } -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - } - -if (clcount) { -XFS_STATS_INC (mp, xs_icluster_flushcnt); -XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); - } - -out_free: -rcu_read_unlock (); -kmem_free (cilist); -out_put: -xfs_perag_put (pag); -if (error) { - bp->b_flags |=XBF_ASYNC ; -xfs_buf_ioend_fail (bp); -xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); - } -return error; -} - -STATIC int -xfs_iflush_int ( +static int +xfs_iflush (struct xfs_inode *ip,struct xfs_buf *bp) {@@ -3743,6 +3598,150 @@ xfs_iflush_int(return error; } +/* +* Non-blocking flush of dirty inode metadata into the backing buffer. +* +* The caller must have a reference to the inode and hold the cluster buffer +* locked. The function will walk across all the inodes on the cluster buffer it +* can find and lock without blocking, and flush them to the cluster buffer. +* +* On success, the caller must write out the buffer returned in *bp and +* release it. On failure, the filesystem will be shut down, the buffer will +* have been unlocked and released, and EFSCORRUPTED will be returned. +*/ +int +xfs_iflush_cluster ( +struct xfs_inode *ip, +struct xfs_buf *bp) +{ +struct xfs_mount *mp = ip->i_mount; +struct xfs_perag *pag; +unsigned long first_index, mask; +int cilist_size; +struct xfs_inode **cilist; +struct xfs_inode *cip; +struct xfs_ino_geometry *igeo =M_IGEO (mp); +int error =0 ; +int nr_found; +int clcount =0 ; +int i; + + pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); + + cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); + cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); +if (!cilist) + goto out_put; + + mask = ~(igeo->inodes_per_cluster -1 ); + first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; +rcu_read_lock (); +/* really need a gang lookup range call here */ + nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, + first_index, igeo->inodes_per_cluster); +if (nr_found ==0 ) + goto out_free; + +for (i =0 ; i < nr_found; i++) { + cip = cilist[i]; + +/* +* because this is an RCU protected lookup, we could find a +* recently freed or even reallocated inode during the lookup. +* We need to check under the i_flags_lock for a valid inode +* here. Skip it if it is not valid or the wrong inode. +*/ +spin_lock (&cip->i_flags_lock); +if (!cip->i_ino || +__xfs_iflags_test (cip,XFS_ISTALE )) { +spin_unlock (&cip->i_flags_lock); +continue ; + } + +/* +* Once we fall off the end of the cluster, no point checking +* any more inodes in the list because they will also all be +* outside the cluster. +*/ +if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { +spin_unlock (&cip->i_flags_lock); +break ; + } +spin_unlock (&cip->i_flags_lock); + +/* +* Do an un-protected check to see if the inode is dirty and +* is a candidate for flushing. These checks will be repeated +* later after the appropriate locks are acquired. +*/ +if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) +continue ; + +/* +* Try to get locks. If any are unavailable or it is pinned, +* then this inode cannot be flushed and is skipped. +*/ + +if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) +continue ; +if (!xfs_iflock_nowait (cip)) { +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } +if (xfs_ipincount (cip)) { +xfs_ifunlock (cip); +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } + + +/* +* Check the inode number again, just to be certain we are not +* racing with freeing in xfs_reclaim_inode(). See the comments +* in that function for more information as to why the initial +* check is not sufficient. +*/ +if (!cip->i_ino) { +xfs_ifunlock (cip); +xfs_iunlock (cip,XFS_ILOCK_SHARED ); +continue ; + } + +/* +* arriving here means that this inode can be flushed. First +* re-check that it's dirty before flushing. +*/ +if (!xfs_inode_clean (cip)) { + error =xfs_iflush (cip, bp); +if (error) { +xfs_iunlock (cip,XFS_ILOCK_SHARED ); + goto out_free; + } + clcount++; + }else { +xfs_ifunlock (cip); + } +xfs_iunlock (cip,XFS_ILOCK_SHARED ); + } + +if (clcount) { +XFS_STATS_INC (mp, xs_icluster_flushcnt); +XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); + } + +out_free: +rcu_read_unlock (); +kmem_free (cilist); +out_put: +xfs_perag_put (pag); +if (error) { + bp->b_flags |=XBF_ASYNC ; +xfs_buf_ioend_fail (bp); +xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); + } +return error; +} +/* Release an inode. */ void xfs_irele(Signature · 1 line
2.26.2.761.g0e0b3e54be
PATCH 28/30 xfs: rework xfs_iflush_cluster() dirty inode iteration
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we have all the dirty inodes attached to the cluster buffer, we don't actually have to do radix tree lookups to find them. Sure, the radix tree is efficient, but walking a linked list of just the dirty inodes attached to the buffer is much better.
We are also no longer dependent on having a locked inode passed into the function to determine where to start the lookup. This means we can drop it from the function call and treat all inodes the same.
We also make xfs_iflush_cluster skip inodes marked with XFS_IRECLAIM. This we avoid races with inodes that reclaim is actively referencing or are being re-initialised by inode lookup. If they are actually dirty, they'll get written by a future cluster flush....
We also add a shutdown check after obtaining the flush lock so that we catch inodes that are dirty in memory and may have inconsistent state due to the shutdown in progress. We abort these inodes directly and so they remove themselves directly from the buffer list and the AIL rather than having to wait for the buffer to be failed and callbacks run to be processed correctly.
--- fs/xfs/xfs_inode.c | 148 ++++++++++++++++------------------------ fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 2 +- 3 files changed, 62 insertions(+), 90 deletions(-)
fs/xfs/xfs_inode.c +60 -88
@@ -3611,117 +3611,94 @@ xfs_iflush(*/ int xfs_iflush_cluster ( -struct xfs_inode *ip,struct xfs_buf *bp) { -struct xfs_mount *mp = ip->i_mount; -struct xfs_perag *pag; -unsigned long first_index, mask; -int cilist_size; -struct xfs_inode **cilist; -struct xfs_inode *cip; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -int error =0 ; -int nr_found; +struct xfs_mount *mp = bp->b_mount; +struct xfs_log_item *lip, *n; +struct xfs_inode *ip; +struct xfs_inode_log_item *iip;int clcount =0 ; -int i; - - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); - - cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); - cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); -if (!cilist) - goto out_put; - - mask = ~(igeo->inodes_per_cluster -1 ); - first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; -rcu_read_lock (); -/* really need a gang lookup range call here */ - nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, - first_index, igeo->inodes_per_cluster); -if (nr_found ==0 ) - goto out_free; +int error =0 ; -for (i =0 ; i < nr_found; i++) { - cip = cilist[i]; +/* +* We must use the safe variant here as on shutdown xfs_iflush_abort() +* can remove itself from the list. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { + iip = (struct xfs_inode_log_item *)lip; + ip = iip->ili_inode;/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. +* Quick and dirty check to avoid locks if possible. */ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) +continue ; +if (xfs_ipincount (ip))continue ; - }/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. +* The inode is still attached to the buffer, which means it is +* dirty but reclaim might try to grab it. Check carefully for +* that, and grab the ilock while still holding the i_flags_lock +* to guarantee reclaim will not be able to reclaim this inode +* once we drop the i_flags_lock. */ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; +spin_lock (&ip->i_flags_lock); +ASSERT (!__xfs_iflags_test (ip,XFS_ISTALE )); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) { +spin_unlock (&ip->i_flags_lock); +continue ; } -spin_unlock (&cip->i_flags_lock);/* -* Do an un-protected check to see if the inode is dirty and -* is a candidate for flushing. These checks will be repeated -* later after the appropriate locks are acquired. +* ILOCK will pin the inode against reclaim and prevent +* concurrent transactions modifying the inode while we are +* flushing the inode. */ -if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) +if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) { +spin_unlock (&ip->i_flags_lock);continue ; + } +spin_unlock (&ip->i_flags_lock);/* -* Try to get locks. If any are unavailable or it is pinned, -* then this inode cannot be flushed and is skipped. +* Skip inodes that are already flush locked as they have +* already been written to the buffer. */ - -if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) -continue ; -if (!xfs_iflock_nowait (cip)) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } -if (xfs_ipincount (cip)) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (!xfs_iflock_nowait (ip)) { +xfs_iunlock (ip,XFS_ILOCK_SHARED );continue ; } -/* -* Check the inode number again, just to be certain we are not -* racing with freeing in xfs_reclaim_inode(). See the comments -* in that function for more information as to why the initial -* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip); +/* xfs_iflush_abort() drops the flush lock */ +xfs_iflush_abort (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); + error = -EIO ;continue ; } -/* -* arriving here means that this inode can be flushed. First -* re-check that it's dirty before flushing. -*/ -if (!xfs_inode_clean (cip)) { - error =xfs_iflush (cip, bp); -if (error) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto out_free; - } - clcount++; - }else { -xfs_ifunlock (cip); +/* don't block waiting on a log force to unpin dirty inodes */ +if (xfs_ipincount (ip)) { +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +continue ; } -xfs_iunlock (cip,XFS_ILOCK_SHARED ); + +if (!xfs_inode_clean (ip)) + error =xfs_iflush (ip, bp); +else +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +if (error) +break ; + clcount++; }if (clcount) {
@@ -3729,11 +3706,6 @@ xfs_iflush_cluster(XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); } -out_free: -rcu_read_unlock (); -kmem_free (cilist); -out_put: -xfs_perag_put (pag);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp);
fs/xfs/xfs_inode.h +1 -1
@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *); +int xfs_iflush_cluster (struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);
fs/xfs/xfs_inode_item.c +1 -1
@@ -511,7 +511,7 @@ xfs_inode_item_push(* reference for IO until we queue the buffer for delwri submission. */ xfs_buf_hold (bp); - error =xfs_iflush_cluster (ip, bp); + error =xfs_iflush_cluster (bp);if (!error) {if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 43 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have all the dirty inodes attached to the cluster buffer, we don't actually have to do radix tree lookups to find them. Sure, the radix tree is efficient, but walking a linked list of just the dirty inodes attached to the buffer is much better.
We are also no longer dependent on having a locked inode passed into the function to determine where to start the lookup. This means we can drop it from the function call and treat all inodes the same.
We also make
xfs_iflush_clusterskip inodes marked withXFS_IRECLAIM. This we avoid races with inodes that reclaim is actively referencing or are being re-initialised by inode lookup. If they are actually dirty, they'll get written by a future cluster flush....We also add a shutdown check after obtaining the flush lock so that we catch inodes that are dirty in memory and may have inconsistent state due to the shutdown in progress. We abort these inodes directly and so they remove themselves directly from the buffer list and the AIL rather than having to wait for the buffer to be failed and callbacks run to be processed correctly.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 148 ++++++++++++++++------------------------ fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 2 +- 3 files changed, 62 insertions(+), 90 deletions(-)fs/xfs/xfs_inode.c +0
-1@@ -3611,117 +3611,94 @@ xfs_iflush(*/ int xfs_iflush_cluster ( -struct xfs_inode *ip,struct xfs_buf *bp) {
...
Quoted 30 lines
fs/xfs/xfs_inode.c +12
-12+/* +* We must use the safe variant here as on shutdown xfs_iflush_abort() +* can remove itself from the list. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { + iip = (struct xfs_inode_log_item *)lip; + ip = iip->ili_inode;/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. +* Quick and dirty check to avoid locks if possible. */ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) +continue ; +if (xfs_ipincount (ip))continue ; - }/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. +* The inode is still attached to the buffer, which means it is
fs/xfs/xfs_inode.c xfs_iflush( · line 3647
Quoted 10 lines
fs/xfs/xfs_inode.c +6
-3+* dirty but reclaim might try to grab it. Check carefully for +* that, and grab the ilock while still holding the i_flags_lock +* to guarantee reclaim will not be able to reclaim this inode +* once we drop the i_flags_lock. */ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; +spin_lock (&ip->i_flags_lock); +ASSERT (!__xfs_iflags_test (ip,XFS_ISTALE ));
What prevents a race with staling an inode here? The push open codes an unlocked (i.e. no memory barrier) check before it acquires the buffer lock, so afaict it's technically possible to race and grab the buffer immediately after the cluster was freed. If that could happen, it looks like we'd also queue the buffer for write.
That also raises the question.. between this patch and the earlier push rework, why do we queue the buffer for write when nothing might have been flushed by this call?
Quoted 47 lines
fs/xfs/xfs_inode.c +14
-21+if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) { +spin_unlock (&ip->i_flags_lock); +continue ; } -spin_unlock (&cip->i_flags_lock);/* -* Do an un-protected check to see if the inode is dirty and -* is a candidate for flushing. These checks will be repeated -* later after the appropriate locks are acquired. +* ILOCK will pin the inode against reclaim and prevent +* concurrent transactions modifying the inode while we are +* flushing the inode. */ -if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) +if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) { +spin_unlock (&ip->i_flags_lock);continue ; + } +spin_unlock (&ip->i_flags_lock);/* -* Try to get locks. If any are unavailable or it is pinned, -* then this inode cannot be flushed and is skipped. +* Skip inodes that are already flush locked as they have +* already been written to the buffer. */ - -if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) -continue ; -if (!xfs_iflock_nowait (cip)) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } -if (xfs_ipincount (cip)) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (!xfs_iflock_nowait (ip)) { +xfs_iunlock (ip,XFS_ILOCK_SHARED );continue ; } -/* -* Check the inode number again, just to be certain we are not -* racing with freeing in xfs_reclaim_inode(). See the comments -* in that function for more information as to why the initial
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 10 lines
fs/xfs/xfs_inode.c +5
-4-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);
Note that we have an unlocked check above that skips pinned inodes.
Brian
Quoted 80 lines
fs/xfs/xfs_inode.c +18
-19+/* xfs_iflush_abort() drops the flush lock */ +xfs_iflush_abort (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); + error = -EIO ;continue ; } -/* -* arriving here means that this inode can be flushed. First -* re-check that it's dirty before flushing. -*/ -if (!xfs_inode_clean (cip)) { - error =xfs_iflush (cip, bp); -if (error) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto out_free; - } - clcount++; - }else { -xfs_ifunlock (cip); +/* don't block waiting on a log force to unpin dirty inodes */ +if (xfs_ipincount (ip)) { +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +continue ; } -xfs_iunlock (cip,XFS_ILOCK_SHARED ); + +if (!xfs_inode_clean (ip)) + error =xfs_iflush (ip, bp); +else +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +if (error) +break ; + clcount++; }if (clcount) {@@ -3729,11 +3706,6 @@ xfs_iflush_cluster(XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); } -out_free: -rcu_read_unlock (); -kmem_free (cilist); -out_put: -xfs_perag_put (pag);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp);fs/xfs/xfs_inode.h +1
-1@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *); +int xfs_iflush_cluster (struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);fs/xfs/xfs_inode_item.c +1
-1@@ -511,7 +511,7 @@ xfs_inode_item_push(* reference for IO until we queue the buffer for delwri submission. */ xfs_buf_hold (bp); - error =xfs_iflush_cluster (ip, bp); + error =xfs_iflush_cluster (bp);if (!error) {if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 79 lines · Brian Foster
Quoted 43 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have all the dirty inodes attached to the cluster buffer, we don't actually have to do radix tree lookups to find them. Sure, the radix tree is efficient, but walking a linked list of just the dirty inodes attached to the buffer is much better.
We are also no longer dependent on having a locked inode passed into the function to determine where to start the lookup. This means we can drop it from the function call and treat all inodes the same.
We also make
xfs_iflush_clusterskip inodes marked withXFS_IRECLAIM. This we avoid races with inodes that reclaim is actively referencing or are being re-initialised by inode lookup. If they are actually dirty, they'll get written by a future cluster flush....We also add a shutdown check after obtaining the flush lock so that we catch inodes that are dirty in memory and may have inconsistent state due to the shutdown in progress. We abort these inodes directly and so they remove themselves directly from the buffer list and the AIL rather than having to wait for the buffer to be failed and callbacks run to be processed correctly.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 148 ++++++++++++++++------------------------ fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 2 +- 3 files changed, 62 insertions(+), 90 deletions(-)fs/xfs/xfs_inode.c +0
-1@@ -3611,117 +3611,94 @@ xfs_iflush(*/ int xfs_iflush_cluster ( -struct xfs_inode *ip,struct xfs_buf *bp) {...
Quoted 34 lines
fs/xfs/xfs_inode.c +16
-12+/* +* We must use the safe variant here as on shutdown xfs_iflush_abort() +* can remove itself from the list. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { + iip = (struct xfs_inode_log_item *)lip; + ip = iip->ili_inode;/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. +* Quick and dirty check to avoid locks if possible. */ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) +continue ; +if (xfs_ipincount (ip))continue ; - }/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. +* The inode is still attached to the buffer, which means it is +* dirty but reclaim might try to grab it. Check carefully for +* that, and grab the ilock while still holding the i_flags_lock +* to guarantee reclaim will not be able to reclaim this inode +* once we drop the i_flags_lock.
Quoted 8 lines
fs/xfs/xfs_inode.c xfs_iflush( · line 3647
Quoted 6 lines
fs/xfs/xfs_inode.c +2
-3*/ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; +spin_lock (&ip->i_flags_lock); +ASSERT (!__xfs_iflags_test (ip,XFS_ISTALE ));What prevents a race with staling an inode here?
xfs_buf_item_release() does not drop the buffer lock when stale buffers are committed. Hence the buffer it held locked until it is committed to the journal, and unpinning the buffer on journal IO completion runs xfs_iflush_done() -> xfs_iflush_abort() on all the stale attached inodes. At which point, they are removed from the buffer and the buffer is unlocked..
Hence we cannot run here with stale inodes attached to the buffer because the buffer is locked the entire time stale inodes are attached.
Quoted 5 lines
The push open codes an
unlocked (i.e. no memory barrier) check before it acquires the buffer lock, so afaict it's technically possible to race and grab the buffer immediately after the cluster was freed. If that could happen, it looks like we'd also queue the buffer for write.
Not that I can tell, because we'll fail to get the buffer lock under the AIL lock until the stale buffer is unpinned, but the unpin occurs under the buffer lock and that removes the buffer from the AIL. Hence there's no way the AIL pushing can actually push an inode cluster buffer that has stale inodes attached to it...
Quoted 3 lines
That also raises the question.. between this patch and the earlier push rework, why do we queue the buffer for write when nothing might have been flushed by this call?
Largely because I never observed a failure to flush at least one inode so I didn't really consider it worth the additional complexity in error handling in the push code. I've really been concerned about the other end of the scale where we try to push the same buffer 30+ times for each IO.
You are right in that it could happen - perhaps just returning EAGAIN if clcount == 0 at the end of the function is all that is necessary and translating that to XFS_ITEM_LOCKED in the push function would work.
Quoted 4 lines
Quoted 4 lines
fs/xfs/xfs_inode.c +3
-1-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL.
Quoted 8 lines
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 6 lines
fs/xfs/xfs_inode.c +2
-3*/ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);Note that we have an unlocked check above that skips pinned inodes.
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 96 lines · Dave Chinner
Quoted 79 lines · Brian Foster
Quoted 43 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Now that we have all the dirty inodes attached to the cluster buffer, we don't actually have to do radix tree lookups to find them. Sure, the radix tree is efficient, but walking a linked list of just the dirty inodes attached to the buffer is much better.
We are also no longer dependent on having a locked inode passed into the function to determine where to start the lookup. This means we can drop it from the function call and treat all inodes the same.
We also make
xfs_iflush_clusterskip inodes marked withXFS_IRECLAIM. This we avoid races with inodes that reclaim is actively referencing or are being re-initialised by inode lookup. If they are actually dirty, they'll get written by a future cluster flush....We also add a shutdown check after obtaining the flush lock so that we catch inodes that are dirty in memory and may have inconsistent state due to the shutdown in progress. We abort these inodes directly and so they remove themselves directly from the buffer list and the AIL rather than having to wait for the buffer to be failed and callbacks run to be processed correctly.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode.c | 148 ++++++++++++++++------------------------ fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 2 +- 3 files changed, 62 insertions(+), 90 deletions(-)fs/xfs/xfs_inode.c +0
-1@@ -3611,117 +3611,94 @@ xfs_iflush(*/ int xfs_iflush_cluster ( -struct xfs_inode *ip,struct xfs_buf *bp) {...
Quoted 34 lines
fs/xfs/xfs_inode.c +16
-12+/* +* We must use the safe variant here as on shutdown xfs_iflush_abort() +* can remove itself from the list. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { + iip = (struct xfs_inode_log_item *)lip; + ip = iip->ili_inode;/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. +* Quick and dirty check to avoid locks if possible. */ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) +continue ; +if (xfs_ipincount (ip))continue ; - }/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. +* The inode is still attached to the buffer, which means it is +* dirty but reclaim might try to grab it. Check carefully for +* that, and grab the ilock while still holding the i_flags_lock +* to guarantee reclaim will not be able to reclaim this inode +* once we drop the i_flags_lock. Quoted 8 lines
fs/xfs/xfs_inode.c xfs_iflush( · line 3647
Quoted 6 lines
fs/xfs/xfs_inode.c +2
-3*/ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; +spin_lock (&ip->i_flags_lock); +ASSERT (!__xfs_iflags_test (ip,XFS_ISTALE ));What prevents a race with staling an inode here?
xfs_buf_item_release()does not drop the buffer lock when stale buffers are committed. Hence the buffer it held locked until it is committed to the journal, and unpinning the buffer on journal IO completion runsxfs_iflush_done()->xfs_iflush_abort()on all the stale attached inodes. At which point, they are removed from the buffer and the buffer is unlocked..
Quoted 4 lines
Hence we cannot run here with stale inodes attached to the buffer because the buffer is locked the entire time stale inodes are attached.
Ok, so it's actually that the earlier ISTALE check on the inode is more of an optimization since if we saw the flag set, we'd never be able to lock the buffer anyways.
Quoted 6 lines
Quoted 5 lines
The push open codes an
unlocked (i.e. no memory barrier) check before it acquires the buffer lock, so afaict it's technically possible to race and grab the buffer immediately after the cluster was freed. If that could happen, it looks like we'd also queue the buffer for write.
Quoted 6 lines
Not that I can tell, because we'll fail to get the buffer lock under the AIL lock until the stale buffer is unpinned, but the unpin occurs under the buffer lock and that removes the buffer from the AIL. Hence there's no way the AIL pushing can actually push an inode cluster buffer that has stale inodes attached to it...
Makes sense.
Quoted 10 lines
Quoted 3 lines
That also raises the question.. between this patch and the earlier push rework, why do we queue the buffer for write when nothing might have been flushed by this call?
Largely because I never observed a failure to flush at least one inode so I didn't really consider it worth the additional complexity in error handling in the push code. I've really been concerned about the other end of the scale where we try to push the same buffer 30+ times for each IO.
Quoted 5 lines
You are right in that it could happen - perhaps just returning EAGAIN if clcount == 0 at the end of the function is all that is necessary and translating that to
XFS_ITEM_LOCKEDin the push function would work.
That sounds reasonable enough to cover that case.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
fs/xfs/xfs_inode.c +3
-1-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. Quoted 8 lines
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 6 lines
fs/xfs/xfs_inode.c +2
-3*/ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me. IIUC we rely on potentially coming back through the cluster flush path multiple times if inodes are pinned because of the racy pin/shutdown checks. If that's the case, then why not push the shutdown check to after the locked pin count check, skip the xfs_iunpin_wait() call entirely and then proceed to abort the flush at the same point we'd call xfs_iflush() if the fs weren't shutdown?
FWIW, I'm also kind of wondering if we need this shutdown check at all now that this path doesn't have to accommodate reclaim. We follow up with failing the buffer in the exit path, but there's no guarantee there aren't other inodes that are passed over due to the pin checks, trylocks, etc. Can we just flush the inodes as normal and let xfsaild fail the buffers on submission without additional shutdown checks?
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 23 lines · Brian Foster
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Brian Foster
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 10 lines · Dave Chinner
fs/xfs/xfs_inode.c +5
-4-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Quoted 6 lines
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me.
Yes, shutdown is racy, so it could be either. However, I'm not changing the shutdown logic or handling here. If the shutdown race could happen before this patchset (and it can), it can still happen after the patchset, and this patchset does not change the way we handle the shutdown race at all.
IOWs, while this shutdown logic may appear "random", that's not a result of this patchset - it a result of design decisions made in the last major shutdown rework/cleanup that required checks to be added to places that could hang waiting for an event that would never occur because shutdown state prevented it from occurring.
There's already enough complexity in this patchset that adding shutdown logic changes is just too much to ask for. If we want to change how various shutdown logics work, lets do it as a separate set of changes so all the subtle bugs that result from the changes bisect to the isolated shutdown logic changes...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 37 lines · Dave Chinner
Quoted 23 lines · Brian Foster
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Brian Foster
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 10 lines · Dave Chinner
fs/xfs/xfs_inode.c +5
-4-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Quoted 6 lines
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me.
Yes, shutdown is racy, so it could be either. However, I'm not changing the shutdown logic or handling here. If the shutdown race could happen before this patchset (and it can), it can still happen after the patchset, and this patchset does not change the way we handle the shutdown race at all.
Quoted 6 lines
IOWs, while this shutdown logic may appear "random", that's not a result of this patchset - it a result of design decisions made in the last major shutdown rework/cleanup that required checks to be added to places that could hang waiting for an event that would never occur because shutdown state prevented it from occurring.
It's not so much the shutdown check that I find random as much as how it intends to handle pinned inodes.
Quoted 6 lines
There's already enough complexity in this patchset that adding shutdown logic changes is just too much to ask for. If we want to change how various shutdown logics work, lets do it as a separate set of changes so all the subtle bugs that result from the changes bisect to the isolated shutdown logic changes...
The fact that shutdown is racy is just background context. My point is that this patch appears to introduce special shutdown handling for a condition where it 1.) didn't previously exist and 2.) doesn't appear to be necessary.
The current push/flush code only incorporates a shutdown check indirectly via mapping the buffer, which simulates an I/O failure and causes us to abort the flush (and shutdown if the I/O failure happened for some other reason). If the shutdown happened sometime after we acquired the buffer, then there's no real impact on this code path. We flush the inode(s) and return success. The shutdown will be handled appropriately when xfsaild attempts to submit the buffer.
The new code no longer maps the buffer because that is done much earlier, but for some reason incorporates a new check to abort the flush if the fs is already shutdown. The problem I have with this is that these checks tend to be brittle, untested and a maintenance burden. As such, I don't think we should ever add new shutdown checks for cases that aren't required for functional correctness. That way we hopefully move to a state where we have the minimum number of shutdown checks with broadest coverage to ensure everything unwinds correctly, but don't have to constantly battle with insufficiently tested logic in obscure contexts that silently break as surrounding code changes over time and leads to random fstests hangs and shutdown logic cleanouts every couple of years.
So my question for any newly added shutdown check is: what problem does this check solve? If there isn't an explicit functional problem and it's intended more as convenience/efficiency logic (which is what the comment implies), then I don't think it's justified. If there is one, then perhaps it is justified, but should be more clearly documented (and I do still think the pin check logic should be cleaned up, but that's a very minor tweak).
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 38 lines · Brian Foster
Quoted 37 lines · Dave Chinner
Quoted 23 lines · Brian Foster
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Brian Foster
fs/xfs/xfs_inode.c xfs_iflush( · line 3679
Quoted 10 lines · Dave Chinner
fs/xfs/xfs_inode.c +5
-4-* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Quoted 6 lines
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me.
Yes, shutdown is racy, so it could be either. However, I'm not changing the shutdown logic or handling here. If the shutdown race could happen before this patchset (and it can), it can still happen after the patchset, and this patchset does not change the way we handle the shutdown race at all.
Quoted 8 lines
Quoted 5 lines
IOWs, while this shutdown logic may appear "random", that's not a result of this patchset - it a result of design decisions made in the last major shutdown rework/cleanup that required checks to be added to places that could hang waiting for an event that would never occur because shutdown state prevented it from occurring.
It's not so much the shutdown check that I find random as much as how it intends to handle pinned inodes.
I can change that, but that's the shutdown was handled by similar code in the past, so it seemed appropriate here because this code was hanging on shutdowns during development.
Quoted 11 lines
Quoted 6 lines
There's already enough complexity in this patchset that adding shutdown logic changes is just too much to ask for. If we want to change how various shutdown logics work, lets do it as a separate set of changes so all the subtle bugs that result from the changes bisect to the isolated shutdown logic changes...
The fact that shutdown is racy is just background context. My point is that this patch appears to introduce special shutdown handling for a condition where it 1.) didn't previously exist and 2.) doesn't appear to be necessary.
The random shutdown failures I kept seeing in shutdown tests says otherwise.
Quoted 7 lines
The current push/flush code only incorporates a shutdown check indirectly via mapping the buffer, which simulates an I/O failure and causes us to abort the flush (and shutdown if the I/O failure happened for some other reason). If the shutdown happened sometime after we acquired the buffer, then there's no real impact on this code path. We flush the inode(s) and return success. The shutdown will be handled appropriately when xfsaild attempts to submit the buffer.
The long-standing rule of thumb is "shutdown == abort in-progress IO immediately", which is what I followed here when it became apparent there was some kind of subtle shutdown issue occurring. That made the shutdown problem go away.
It may be that changes I've been making to other parts of this writeback code make the shutdown check here unnecessary. My testing said otherwise, but maybe that's all been cleared up. Hence if the shutdown check is truly unnecessary, let's clean it up in a future patchset where that assertion can be bisected down cleanly. I needed this to get fstests to pass, and for this patchset which is entirely unrelated to shutdown architecture, that's all the justification that should be necessary.
Quoted 4 lines
The new code no longer maps the buffer because that is done much earlier, but for some reason incorporates a new check to abort the flush if the fs is already shutdown. The problem I have with this is that these checks tend to be brittle, untested and a maintenance burden.
Shutdown has *always* been brittle. You've seen it go from being completely fucking awful to actually being somewhat reliable because your experience with XFS matches to roughly when we first added substantial shutdown validation to fstests. We had a huge mountain of technical debt around shutdown, but the importance of addressing it has historically been pretty low because -shutdown is extremely rare- in production systems.
Quoted 3 lines
As
such, I don't think we should ever add new shutdown checks for cases that aren't required for functional correctness.
I think the burden of proof is the wrong way around for the current shutdown architecture. If you want to make a rule like this, you need to fix define how the shutdown architecture is going to allow this sort of rule to be applied without placing unreasonable demands on the patch author.
Quoted 7 lines
That way we hopefully
move to a state where we have the minimum number of shutdown checks with broadest coverage to ensure everything unwinds correctly, but don't have to constantly battle with insufficiently tested logic in obscure contexts that silently break as surrounding code changes over time and leads to random fstests hangs and shutdown logic cleanouts every couple of years.
Yes, I think this is an admirable goal, but I think you've got how to get there completely backward. First work out the architecture and logic that allows us to remove/avoid "randomly" placed checks, then work towards cleaning up the code. We don't get there by saying "no new checks!" and then ignoring the underlying problems those checks are trying to avoid.
If you can come up with a shutdown mechanism that:
a) prevents all IO the moment a shutdown is triggered, and
b) avoids shutdown detection ordering hangs between different objects and subsystems,
then you have grounds for saying "nobody should need to add new shutdown checks". But right now, that's not even close to being the reality. There needs to be lots more cleanup and rework to the shutdown code to be done before we get anywhere near this...
Quoted 2 lines
So my question for any newly added shutdown check is: what problem does this check solve?
Repeated fstests hangs on various shutdown tests while developing this code.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 45 lines · Dave Chinner
Quoted 44 lines · Brian Foster
Quoted 42 lines · Dave Chinner
Quoted 23 lines · Brian Foster
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Brian Foster
Review
Quoted 10 lines · Dave Chinner
(quoted diff) +5
-4- * check is not sufficient. + * If we are shut down, unpin and abort the inode now as there + * is no point in flushing it to the buffer just to get an IO + * completion to abort the buffer and remove it from the AIL. */ - if (!cip->i_ino) { - xfs_ifunlock(cip); - xfs_iunlock(cip, XFS_ILOCK_SHARED); + if (XFS_FORCED_SHUTDOWN(mp)) { + xfs_iunpin_wait(ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Quoted 6 lines
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me.
Yes, shutdown is racy, so it could be either. However, I'm not changing the shutdown logic or handling here. If the shutdown race could happen before this patchset (and it can), it can still happen after the patchset, and this patchset does not change the way we handle the shutdown race at all.
IOWs, while this shutdown logic may appear "random", that's not a result of this patchset - it a result of design decisions made in the last major shutdown rework/cleanup that required checks to be added to places that could hang waiting for an event that would never occur because shutdown state prevented it from occurring.
Quoted 7 lines
Quoted 2 lines
It's not so much the shutdown check that I find random as much as how it intends to handle pinned inodes.
I can change that, but that's the shutdown was handled by similar code in the past, so it seemed appropriate here because this code was hanging on shutdowns during development.
Ok.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
There's already enough complexity in this patchset that adding shutdown logic changes is just too much to ask for. If we want to change how various shutdown logics work, lets do it as a separate set of changes so all the subtle bugs that result from the changes bisect to the isolated shutdown logic changes...
Quoted 8 lines
Quoted 4 lines
The fact that shutdown is racy is just background context. My point is that this patch appears to introduce special shutdown handling for a condition where it 1.) didn't previously exist and 2.) doesn't appear to be necessary.
The random shutdown failures I kept seeing in shutdown tests says otherwise.
Then can we please focus on the issue? My initial feedback was around the pin state logic and that made me question whether the broader shutdown logic was spurious. The response to that was that there's too much complexity in this series to change shutdown logic, etc., and that suggested I should justify the feedback with this being shutdown code and all. Now we're debating about whether I want to make some kind of architectural shutdown rule (I don't, and the reasoning for my question doesn't invalidate the question itself), while I still have no idea what issue this code actually fixes.
Can you elaborate on the problem, please? I realize that specifics may be lost, but what in general about the failure suggested placing this shutdown check in the cluster flush path? Does it concern you at all that since shutdown is racy as such, this is possibly suppressing a new (or old) bug and making it harder to reproduce and diagnose? It's also quite possible that this was a latent issue exposed by removing the buffer mapping from this path. In that case it probably does make more sense to keep the check in this patch, but if that's the case I also might be willing to spend some time digging into it to try and address it sooner rather than have it fall to the wayside if you'd be willing to provide some useful information...
Brian
Quoted 84 lines
Quoted 7 lines
The current push/flush code only incorporates a shutdown check indirectly via mapping the buffer, which simulates an I/O failure and causes us to abort the flush (and shutdown if the I/O failure happened for some other reason). If the shutdown happened sometime after we acquired the buffer, then there's no real impact on this code path. We flush the inode(s) and return success. The shutdown will be handled appropriately when xfsaild attempts to submit the buffer.
The long-standing rule of thumb is "shutdown == abort in-progress IO immediately", which is what I followed here when it became apparent there was some kind of subtle shutdown issue occurring. That made the shutdown problem go away.
It may be that changes I've been making to other parts of this writeback code make the shutdown check here unnecessary. My testing said otherwise, but maybe that's all been cleared up. Hence if the shutdown check is truly unnecessary, let's clean it up in a future patchset where that assertion can be bisected down cleanly. I needed this to get fstests to pass, and for this patchset which is entirely unrelated to shutdown architecture, that's all the justification that should be necessary.
Quoted 4 lines
The new code no longer maps the buffer because that is done much earlier, but for some reason incorporates a new check to abort the flush if the fs is already shutdown. The problem I have with this is that these checks tend to be brittle, untested and a maintenance burden.
Shutdown has *always* been brittle. You've seen it go from being completely fucking awful to actually being somewhat reliable because your experience with XFS matches to roughly when we first added substantial shutdown validation to fstests. We had a huge mountain of technical debt around shutdown, but the importance of addressing it has historically been pretty low because -shutdown is extremely rare- in production systems.
Quoted 3 lines
As
such, I don't think we should ever add new shutdown checks for cases that aren't required for functional correctness.I think the burden of proof is the wrong way around for the current shutdown architecture. If you want to make a rule like this, you need to fix define how the shutdown architecture is going to allow this sort of rule to be applied without placing unreasonable demands on the patch author.
Quoted 7 lines
That way we hopefully
move to a state where we have the minimum number of shutdown checks with broadest coverage to ensure everything unwinds correctly, but don't have to constantly battle with insufficiently tested logic in obscure contexts that silently break as surrounding code changes over time and leads to random fstests hangs and shutdown logic cleanouts every couple of years.Yes, I think this is an admirable goal, but I think you've got how to get there completely backward. First work out the architecture and logic that allows us to remove/avoid "randomly" placed checks, then work towards cleaning up the code. We don't get there by saying "no new checks!" and then ignoring the underlying problems those checks are trying to avoid.
If you can come up with a shutdown mechanism that:
a) prevents all IO the moment a shutdown is triggered, and
b) avoids shutdown detection ordering hangs between different objects and subsystems,then you have grounds for saying "nobody should need to add new shutdown checks". But right now, that's not even close to being the reality. There needs to be lots more cleanup and rework to the shutdown code to be done before we get anywhere near this...
Quoted 2 lines
So my question for any newly added shutdown check is: what problem does this check solve?
Repeated fstests hangs on various shutdown tests while developing this code.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 28/30] xfs: rework xfs_iflush_cluster() dirty inode iteration
Quoted 82 lines · Brian Foster
Quoted 45 lines · Dave Chinner
Quoted 44 lines · Brian Foster
Quoted 42 lines · Dave Chinner
Quoted 23 lines · Brian Foster
Quoted 15 lines · Dave Chinner
Quoted 13 lines · Brian Foster
Review
Quoted 10 lines · Dave Chinner
(quoted diff) +5
-4- * check is not sufficient. + * If we are shut down, unpin and abort the inode now as there + * is no point in flushing it to the buffer just to get an IO + * completion to abort the buffer and remove it from the AIL. */ - if (!cip->i_ino) { - xfs_ifunlock(cip); - xfs_iunlock(cip, XFS_ILOCK_SHARED); + if (XFS_FORCED_SHUTDOWN(mp)) { + xfs_iunpin_wait(ip);Note that we have an unlocked check above that skips pinned inodes.
Quoted 6 lines
Right, but we could be racing with a transaction commit that pinned the inode and a shutdown. As the comment says: it's a quick and dirty check to avoid trying to get locks when we know that it is unlikely we can flush the inode without blocking. We still have to recheck that state once we have the ILOCK....
Quoted 6 lines
Right, but that means we can just as easily skip the shutdown processing (which waits for unpin) if a particular inode is pinned. So which is supposed to happen in the shutdown case?
ISTM that either could happen. As a result this kind of looks like random logic to me.
Yes, shutdown is racy, so it could be either. However, I'm not changing the shutdown logic or handling here. If the shutdown race could happen before this patchset (and it can), it can still happen after the patchset, and this patchset does not change the way we handle the shutdown race at all.
IOWs, while this shutdown logic may appear "random", that's not a result of this patchset - it a result of design decisions made in the last major shutdown rework/cleanup that required checks to be added to places that could hang waiting for an event that would never occur because shutdown state prevented it from occurring.
Quoted 7 lines
Quoted 2 lines
It's not so much the shutdown check that I find random as much as how it intends to handle pinned inodes.
I can change that, but that's the shutdown was handled by similar code in the past, so it seemed appropriate here because this code was hanging on shutdowns during development.
Ok.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
There's already enough complexity in this patchset that adding shutdown logic changes is just too much to ask for. If we want to change how various shutdown logics work, lets do it as a separate set of changes so all the subtle bugs that result from the changes bisect to the isolated shutdown logic changes...
Quoted 8 lines
Quoted 4 lines
The fact that shutdown is racy is just background context. My point is that this patch appears to introduce special shutdown handling for a condition where it 1.) didn't previously exist and 2.) doesn't appear to be necessary.
The random shutdown failures I kept seeing in shutdown tests says otherwise.
Then can we please focus on the issue? My initial feedback was around the pin state logic and that made me question whether the broader shutdown logic was spurious. The response to that was that there's too much complexity in this series to change shutdown logic, etc., and that suggested I should justify the feedback with this being shutdown code and all. Now we're debating about whether I want to make some kind of architectural shutdown rule (I don't, and the reasoning for my question doesn't invalidate the question itself), while I still have no idea what issue this code actually fixes.
Quoted 12 lines
Can you elaborate on the problem, please? I realize that specifics may be lost, but what in general about the failure suggested placing this shutdown check in the cluster flush path? Does it concern you at all that since shutdown is racy as such, this is possibly suppressing a new (or old) bug and making it harder to reproduce and diagnose? It's also quite possible that this was a latent issue exposed by removing the buffer mapping from this path. In that case it probably does make more sense to keep the check in this patch, but if that's the case I also might be willing to spend some time digging into it to try and address it sooner rather than have it fall to the wayside if you'd be willing to provide some useful information...
For reference, I dropped the shutdown check and reproduced a generic/051 explosion (kernel BUG) fairly quickly. Digging into that a bit, it looks like the problem is we essentially can flush an inode that isn't in the AIL because in the log I/O error (shutdown) scenario, we unpin the inode and don't insert it. If the buffer has other inodes in the AIL, then the cluster flush sees the aforementioned inode is dirty and unpinned and presumably safe to flush, even though it wasn't inserted because the log I/O failed. The problem thus manifests as a double remove from the AIL when the buffer is submitted and failed by xfsaild with a non-AIL resident flushed inode.
I think that fundamentally this is not a problem outside of shutdown because we're in a shutdown state before the item is unpinned, and not necessarily a race introduced by this series when you consider the current cluster flush implementation. Therefore it's reasonable to include the shutdown check in the flush path as this patch does. I think there are several options to clean this up fairly easily (reset dirty state on abort, abort the flush on buffer completion when shutdown, etc.), but that can be evaluated separately from this series. Given all of that, I'm Ok with this patch if you fixed up the slightly erratic pin checking logic (including the shutdown check) and replace the misleading comment above the shutdown check with something like:
/*
* Abort the flush if we are shut down because the inode may not
* currently be in the AIL. Log I/O failure unpins the inode
* without inserting it, leaving a dirty/unpinned inode that
* otherwise looks like it should be flushed.
*/
Brian
Quoted 87 lines
Brian
Quoted 84 lines
Quoted 7 lines
The current push/flush code only incorporates a shutdown check indirectly via mapping the buffer, which simulates an I/O failure and causes us to abort the flush (and shutdown if the I/O failure happened for some other reason). If the shutdown happened sometime after we acquired the buffer, then there's no real impact on this code path. We flush the inode(s) and return success. The shutdown will be handled appropriately when xfsaild attempts to submit the buffer.
The long-standing rule of thumb is "shutdown == abort in-progress IO immediately", which is what I followed here when it became apparent there was some kind of subtle shutdown issue occurring. That made the shutdown problem go away.
It may be that changes I've been making to other parts of this writeback code make the shutdown check here unnecessary. My testing said otherwise, but maybe that's all been cleared up. Hence if the shutdown check is truly unnecessary, let's clean it up in a future patchset where that assertion can be bisected down cleanly. I needed this to get fstests to pass, and for this patchset which is entirely unrelated to shutdown architecture, that's all the justification that should be necessary.
Quoted 4 lines
The new code no longer maps the buffer because that is done much earlier, but for some reason incorporates a new check to abort the flush if the fs is already shutdown. The problem I have with this is that these checks tend to be brittle, untested and a maintenance burden.
Shutdown has *always* been brittle. You've seen it go from being completely fucking awful to actually being somewhat reliable because your experience with XFS matches to roughly when we first added substantial shutdown validation to fstests. We had a huge mountain of technical debt around shutdown, but the importance of addressing it has historically been pretty low because -shutdown is extremely rare- in production systems.
Quoted 3 lines
As
such, I don't think we should ever add new shutdown checks for cases that aren't required for functional correctness.I think the burden of proof is the wrong way around for the current shutdown architecture. If you want to make a rule like this, you need to fix define how the shutdown architecture is going to allow this sort of rule to be applied without placing unreasonable demands on the patch author.
Quoted 7 lines
That way we hopefully
move to a state where we have the minimum number of shutdown checks with broadest coverage to ensure everything unwinds correctly, but don't have to constantly battle with insufficiently tested logic in obscure contexts that silently break as surrounding code changes over time and leads to random fstests hangs and shutdown logic cleanouts every couple of years.Yes, I think this is an admirable goal, but I think you've got how to get there completely backward. First work out the architecture and logic that allows us to remove/avoid "randomly" placed checks, then work towards cleaning up the code. We don't get there by saying "no new checks!" and then ignoring the underlying problems those checks are trying to avoid.
If you can come up with a shutdown mechanism that:
a) prevents all IO the moment a shutdown is triggered, and
b) avoids shutdown detection ordering hangs between different objects and subsystems,then you have grounds for saying "nobody should need to add new shutdown checks". But right now, that's not even close to being the reality. There needs to be lots more cleanup and rework to the shutdown code to be done before we get anywhere near this...
Quoted 2 lines
So my question for any newly added shutdown check is: what problem does this check solve?
Repeated fstests hangs on various shutdown tests while developing this code.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 28/30 xfs: rework xfs_iflush_cluster() dirty inode iteration
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Now that we have all the dirty inodes attached to the cluster buffer, we don't actually have to do radix tree lookups to find them. Sure, the radix tree is efficient, but walking a linked list of just the dirty inodes attached to the buffer is much better.
We are also no longer dependent on having a locked inode passed into the function to determine where to start the lookup. This means we can drop it from the function call and treat all inodes the same.
We also make xfs_iflush_cluster skip inodes marked with XFS_IRECLAIM. This we avoid races with inodes that reclaim is actively referencing or are being re-initialised by inode lookup. If they are actually dirty, they'll get written by a future cluster flush....
We also add a shutdown check after obtaining the flush lock so that we catch inodes that are dirty in memory and may have inconsistent state due to the shutdown in progress. We abort these inodes directly and so they remove themselves directly from the buffer list and the AIL rather than having to wait for the buffer to be failed and callbacks run to be processed correctly.
---
V2:
- rework return value from xfs_iflush_cluster to indicate -EAGAIN if no inodes were flushed and handle that case in the caller.
fs/xfs/xfs_inode.c | 169 +++++++++++++++++++++--------------------------- fs/xfs/xfs_inode.h | 2 +- fs/xfs/xfs_inode_item.c | 8 ++- 3 files changed, 81 insertions(+), 98 deletions(-)
fs/xfs/xfs_inode.c +73 -96
@@ -3604,141 +3604,118 @@ xfs_iflush(* locked. The function will walk across all the inodes on the cluster buffer it * can find and lock without blocking, and flush them to the cluster buffer. * -* On success, the caller must write out the buffer returned in *bp and -* release it. On failure, the filesystem will be shut down, the buffer will -* have been unlocked and released, and EFSCORRUPTED will be returned. +* On successful flushing of at least one inode, the caller must write out the +* buffer and release it. If no inodes are flushed, -EAGAIN will be returned and +* the caller needs to release the buffer. On failure, the filesystem will be +* shut down, the buffer will have been unlocked and released, and EFSCORRUPTED +* will be returned. */ int xfs_iflush_cluster ( -struct xfs_inode *ip,struct xfs_buf *bp) { -struct xfs_mount *mp = ip->i_mount; -struct xfs_perag *pag; -unsigned long first_index, mask; -int cilist_size; -struct xfs_inode **cilist; -struct xfs_inode *cip; -struct xfs_ino_geometry *igeo =M_IGEO (mp); -int error =0 ; -int nr_found; +struct xfs_mount *mp = bp->b_mount; +struct xfs_log_item *lip, *n; +struct xfs_inode *ip; +struct xfs_inode_log_item *iip;int clcount =0 ; -int i; - - pag =xfs_perag_get (mp,XFS_INO_TO_AGNO (mp, ip->i_ino)); - - cilist_size = igeo->inodes_per_cluster *sizeof (struct xfs_inode *); - cilist =kmem_alloc (cilist_size,KM_MAYFAIL |KM_NOFS ); -if (!cilist) - goto out_put; - - mask = ~(igeo->inodes_per_cluster -1 ); - first_index =XFS_INO_TO_AGINO (mp, ip->i_ino) & mask; -rcu_read_lock (); -/* really need a gang lookup range call here */ - nr_found =radix_tree_gang_lookup (&pag->pag_ici_root, (void **)cilist, - first_index, igeo->inodes_per_cluster); -if (nr_found ==0 ) - goto out_free; +int error =0 ; -for (i =0 ; i < nr_found; i++) { - cip = cilist[i]; +/* +* We must use the safe variant here as on shutdown xfs_iflush_abort() +* can remove itself from the list. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { + iip = (struct xfs_inode_log_item *)lip; + ip = iip->ili_inode;/* -* because this is an RCU protected lookup, we could find a -* recently freed or even reallocated inode during the lookup. -* We need to check under the i_flags_lock for a valid inode -* here. Skip it if it is not valid or the wrong inode. +* Quick and dirty check to avoid locks if possible. */ -spin_lock (&cip->i_flags_lock); -if (!cip->i_ino || -__xfs_iflags_test (cip,XFS_ISTALE )) { -spin_unlock (&cip->i_flags_lock); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) +continue ; +if (xfs_ipincount (ip))continue ; - }/* -* Once we fall off the end of the cluster, no point checking -* any more inodes in the list because they will also all be -* outside the cluster. +* The inode is still attached to the buffer, which means it is +* dirty but reclaim might try to grab it. Check carefully for +* that, and grab the ilock while still holding the i_flags_lock +* to guarantee reclaim will not be able to reclaim this inode +* once we drop the i_flags_lock. */ -if ((XFS_INO_TO_AGINO (mp, cip->i_ino) & mask) != first_index) { -spin_unlock (&cip->i_flags_lock); -break ; +spin_lock (&ip->i_flags_lock); +ASSERT (!__xfs_iflags_test (ip,XFS_ISTALE )); +if (__xfs_iflags_test (ip,XFS_IRECLAIM |XFS_IFLOCK )) { +spin_unlock (&ip->i_flags_lock); +continue ; } -spin_unlock (&cip->i_flags_lock);/* -* Do an un-protected check to see if the inode is dirty and -* is a candidate for flushing. These checks will be repeated -* later after the appropriate locks are acquired. +* ILOCK will pin the inode against reclaim and prevent +* concurrent transactions modifying the inode while we are +* flushing the inode. */ -if (xfs_inode_clean (cip) &&xfs_ipincount (cip) ==0 ) +if (!xfs_ilock_nowait (ip,XFS_ILOCK_SHARED )) { +spin_unlock (&ip->i_flags_lock);continue ; + } +spin_unlock (&ip->i_flags_lock);/* -* Try to get locks. If any are unavailable or it is pinned, -* then this inode cannot be flushed and is skipped. +* Skip inodes that are already flush locked as they have +* already been written to the buffer. */ - -if (!xfs_ilock_nowait (cip,XFS_ILOCK_SHARED )) -continue ; -if (!xfs_iflock_nowait (cip)) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (!xfs_iflock_nowait (ip)) { +xfs_iunlock (ip,XFS_ILOCK_SHARED );continue ; } -if (xfs_ipincount (cip)) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); -continue ; - } -/* -* Check the inode number again, just to be certain we are not -* racing with freeing in xfs_reclaim_inode(). See the comments -* in that function for more information as to why the initial -* check is not sufficient. +* If we are shut down, unpin and abort the inode now as there +* is no point in flushing it to the buffer just to get an IO +* completion to abort the buffer and remove it from the AIL. */ -if (!cip->i_ino) { -xfs_ifunlock (cip); -xfs_iunlock (cip,XFS_ILOCK_SHARED ); +if (XFS_FORCED_SHUTDOWN (mp)) { +xfs_iunpin_wait (ip); +/* xfs_iflush_abort() drops the flush lock */ +xfs_iflush_abort (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); + error = -EIO ;continue ; } -/* -* arriving here means that this inode can be flushed. First -* re-check that it's dirty before flushing. -*/ -if (!xfs_inode_clean (cip)) { - error =xfs_iflush (cip, bp); -if (error) { -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - goto out_free; - } - clcount++; - }else { -xfs_ifunlock (cip); +/* don't block waiting on a log force to unpin dirty inodes */ +if (xfs_ipincount (ip)) { +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +continue ; } -xfs_iunlock (cip,XFS_ILOCK_SHARED ); - } -if (clcount) { -XFS_STATS_INC (mp, xs_icluster_flushcnt); -XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); +if (!xfs_inode_clean (ip)) + error =xfs_iflush (ip, bp); +else +xfs_ifunlock (ip); +xfs_iunlock (ip,XFS_ILOCK_SHARED ); +if (error) +break ; + clcount++; } -out_free: -rcu_read_unlock (); -kmem_free (cilist); -out_put: -xfs_perag_put (pag);if (error) { bp->b_flags |=XBF_ASYNC ;xfs_buf_ioend_fail (bp);xfs_force_shutdown (mp,SHUTDOWN_CORRUPT_INCORE ); +return error; } -return error; + +if (!clcount) +return -EAGAIN ; + +XFS_STATS_INC (mp, xs_icluster_flushcnt); +XFS_STATS_ADD (mp, xs_icluster_flushinode, clcount); +return 0 ; + }/* Release an inode. */
fs/xfs/xfs_inode.h +1 -1
@@ -427,7 +427,7 @@ int xfs_log_force_inode(struct xfs_inode *ip);void xfs_iunpin_wait (xfs_inode_t *);#define xfs_ipincount (ip) ((unsigned int) atomic_read(&ip->i_pincount)) -int xfs_iflush_cluster (struct xfs_inode *,struct xfs_buf *); +int xfs_iflush_cluster (struct xfs_buf *);void xfs_lock_two_inodes (struct xfs_inode *ip0,uint ip0_mode,struct xfs_inode *ip1,uint ip1_mode);
fs/xfs/xfs_inode_item.c +7 -1
@@ -507,12 +507,18 @@ xfs_inode_item_push(* reference for IO until we queue the buffer for delwri submission. */ xfs_buf_hold (bp); - error =xfs_iflush_cluster (ip, bp); + error =xfs_iflush_cluster (bp);if (!error) {if (!xfs_buf_delwri_queue (bp, buffer_list)) rval =XFS_ITEM_FLUSHING ;xfs_buf_relse (bp); }else { +/* +* Release the buffer if we were unable to flush anything. On +* any other error, the buffer has already been released. +*/ +if (error == -EAGAIN ) +xfs_buf_relse (bp); rval =XFS_ITEM_LOCKED ; }
PATCH 29/30 xfs: factor xfs_iflush_done
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
xfs_iflush_done() does 3 distinct operations to the inodes attached to the buffer. Separate these operations out into functions so that it is easier to modify these operations independently in future.
--- fs/xfs/xfs_inode_item.c | 154 +++++++++++++++++++++------------------- 1 file changed, 81 insertions(+), 73 deletions(-)
fs/xfs/xfs_inode_item.c +81 -73
@@ -640,101 +640,64 @@ xfs_inode_item_destroy(/* -* This is the inode flushing I/O completion routine. It is called -* from interrupt level when the buffer containing the inode is -* flushed to disk. It is responsible for removing the inode item -* from the AIL if it has not been re-logged, and unlocking the inode's -* flush lock. -* -* To reduce AIL lock traffic as much as possible, we scan the buffer log item -* list for other inodes that will run this function. We remove them from the -* buffer list so we can process all the inode IO completions in one AIL lock -* traversal. -* -* Note: Now that we attach the log item to the buffer when we first log the -* inode in memory, we can have unflushed inodes on the buffer list here. These -* inodes will have a zero ili_last_fields, so skip over them here. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ void -xfs_iflush_done ( -struct xfs_buf *bp) +xfs_iflush_ail_updates ( +struct xfs_ail *ailp, +struct list_head *list) { -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip, *n; -struct xfs_ail *ailp = bp->b_mount->m_ail; -int need_ail =0 ; -LIST_HEAD (tmp); +struct xfs_log_item *lip; +xfs_lsn_t tail_lsn =0 ; -/* -* Pull the attached inodes from the buffer one at a time and take the -* appropriate action on them. -*/ -list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { - iip =INODE_ITEM (lip); +/* this is an opencoded batch version of xfs_trans_ail_delete */ +spin_lock (&ailp->ail_lock); +list_for_each_entry (lip, list, li_bio_list) { +xfs_lsn_t lsn; -if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn) { +clear_bit (XFS_LI_FAILED , &lip->li_flags);continue ; } -if (!iip->ili_last_fields) -continue ; - -list_move_tail (&lip->li_bio_list, &tmp); - -/* Do an unlocked check for needing the AIL lock. */ -if (iip->ili_flush_lsn == lip->li_lsn || -test_bit (XFS_LI_FAILED , &lip->li_flags)) - need_ail++; + lsn =xfs_ail_delete_one (ailp, lip); +if (!tail_lsn && lsn) + tail_lsn = lsn; } +xfs_ail_update_finish (ailp, tail_lsn); +} -/* -* We only want to pull the item from the AIL if it is actually there -* and its location in the log has not changed since we started the -* flush. Thus, we only bother if the inode's lsn has not changed. -*/ -if (need_ail) { -xfs_lsn_t tail_lsn =0 ; - -/* this is an opencoded batch version of xfs_trans_ail_delete */ -spin_lock (&ailp->ail_lock); -list_for_each_entry (lip, &tmp, li_bio_list) { -clear_bit (XFS_LI_FAILED , &lip->li_flags); -if (lip->li_lsn ==INODE_ITEM (lip)->ili_flush_lsn) { -xfs_lsn_t lsn =xfs_ail_delete_one (ailp, lip); -if (!tail_lsn && lsn) - tail_lsn = lsn; - } - } -xfs_ail_update_finish (ailp, tail_lsn); - } +/* +* Walk the list of inodes that have completed their IOs. If they are clean +* remove them from the list and dissociate them from the buffer. Buffers that +* are still dirty remain linked to the buffer and on the list. Caller must +* handle them appropriately. +*/ +void +xfs_iflush_finish ( +struct xfs_buf *bp, +struct list_head *list) +{ +struct xfs_log_item *lip, *n; -/* -* Clean up and unlock the flush lock now we are done. We can clear the -* ili_last_fields bits now that we know that the data corresponding to -* them is safely on disk. -*/ -list_for_each_entry_safe (lip, n, &tmp, li_bio_list) { +list_for_each_entry_safe (lip, n, list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip);bool drop_buffer = false; -list_del_init (&lip->li_bio_list); - iip =INODE_ITEM (lip); -spin_lock (&iip->ili_lock);/* * Remove the reference to the cluster buffer if the inode is -* clean in memory. Drop the buffer reference once we've dropped -* the locks we hold. If the inode is dirty in memory, we need -* to put the inode item back on the buffer list for another -* pass through the flush machinery. +* clean in memory and drop the buffer reference once we've +* dropped the locks we hold. */ ASSERT (iip->ili_item.li_buf == bp);if (!iip->ili_fields) { iip->ili_item.li_buf =NULL ; +list_del_init (&lip->li_bio_list); drop_buffer = true; - }else { -list_add (&lip->li_bio_list, &bp->b_li_list); } iip->ili_last_fields =0 ; iip->ili_flush_lsn =0 ;
@@ -745,6 +708,51 @@ xfs_iflush_done( } } +/* +* Inode buffer IO completion routine. It is responsible for removing inodes +* attached to the buffer from the AIL if they have not been re-logged, as well +* as completing the flush and unlocking the inode. +*/ +void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ; + } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */ +if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes); + } + +if (!list_empty (&ail_updates)) { +xfs_iflush_ail_updates (bp->b_mount->m_ail, &ail_updates); +list_splice_tail (&ail_updates, &flushed_inodes); + } + +xfs_iflush_finish (bp, &flushed_inodes); +if (!list_empty (&flushed_inodes)) +list_splice_tail (&flushed_inodes, &bp->b_li_list); +} +/* * This is the inode flushing abort routine. It is called from xfs_iflush when * the filesystem is shutting down to clean up the inode state. It is
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 60 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfs_iflush_done()does 3 distinct operations to the inodes attached to the buffer. Separate these operations out into functions so that it is easier to modify these operations independently in future.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode_item.c | 154 +++++++++++++++++++++------------------- 1 file changed, 81 insertions(+), 73 deletions(-)fs/xfs/xfs_inode_item.c +9
-27@@ -640,101 +640,64 @@ xfs_inode_item_destroy(/* -* This is the inode flushing I/O completion routine. It is called -* from interrupt level when the buffer containing the inode is -* flushed to disk. It is responsible for removing the inode item -* from the AIL if it has not been re-logged, and unlocking the inode's -* flush lock. -* -* To reduce AIL lock traffic as much as possible, we scan the buffer log item -* list for other inodes that will run this function. We remove them from the -* buffer list so we can process all the inode IO completions in one AIL lock -* traversal. -* -* Note: Now that we attach the log item to the buffer when we first log the -* inode in memory, we can have unflushed inodes on the buffer list here. These -* inodes will have a zero ili_last_fields, so skip over them here. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ void -xfs_iflush_done ( -struct xfs_buf *bp) +xfs_iflush_ail_updates ( +struct xfs_ail *ailp, +struct list_head *list) { -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip, *n; -struct xfs_ail *ailp = bp->b_mount->m_ail; -int need_ail =0 ; -LIST_HEAD (tmp); +struct xfs_log_item *lip; +xfs_lsn_t tail_lsn =0 ; -/* -* Pull the attached inodes from the buffer one at a time and take the -* appropriate action on them. -*/ -list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { - iip =INODE_ITEM (lip); +/* this is an opencoded batch version of xfs_trans_ail_delete */
fs/xfs/xfs_inode_item.c xfs_inode_item_destroy( · line 663
Quoted 10 lines
fs/xfs/xfs_inode_item.c +5
-2+spin_lock (&ailp->ail_lock); +list_for_each_entry (lip, list, li_bio_list) { +xfs_lsn_t lsn; -if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn) { +clear_bit (XFS_LI_FAILED , &lip->li_flags);continue ; }
That seems like strange logic. Shouldn't we clear LI_FAILED regardless?
Quoted 17 lines
fs/xfs/xfs_inode_item.c +5
-9-if (!iip->ili_last_fields) -continue ; - -list_move_tail (&lip->li_bio_list, &tmp); - -/* Do an unlocked check for needing the AIL lock. */ -if (iip->ili_flush_lsn == lip->li_lsn || -test_bit (XFS_LI_FAILED , &lip->li_flags)) - need_ail++; + lsn =xfs_ail_delete_one (ailp, lip); +if (!tail_lsn && lsn) + tail_lsn = lsn; } +xfs_ail_update_finish (ailp, tail_lsn); +}
...
Quoted 27 lines
fs/xfs/xfs_inode_item.c +23
-0@@ -745,6 +708,51 @@ xfs_iflush_done( } } +/* +* Inode buffer IO completion routine. It is responsible for removing inodes +* attached to the buffer from the AIL if they have not been re-logged, as well +* as completing the flush and unlocking the inode. +*/ +void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ;
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 743
Quoted 10 lines
fs/xfs/xfs_inode_item.c +10
-0+ } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */ +if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes);
Not sure I see the point of having two lists here, particularly since this is all based on lockless logic. At the very least it's a subtle change in AIL processing logic and I don't think that should be buried in a refactoring patch.
Brian
Quoted 18 lines
fs/xfs/xfs_inode_item.c +12
-0+ } + +if (!list_empty (&ail_updates)) { +xfs_iflush_ail_updates (bp->b_mount->m_ail, &ail_updates); +list_splice_tail (&ail_updates, &flushed_inodes); + } + +xfs_iflush_finish (bp, &flushed_inodes); +if (!list_empty (&flushed_inodes)) +list_splice_tail (&flushed_inodes, &bp->b_li_list); +} +/* * This is the inode flushing abort routine. It is called from xfs_iflush when * the filesystem is shutting down to clean up the inode state. It is Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 65 lines · Brian Foster
Quoted 64 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfs_iflush_done()does 3 distinct operations to the inodes attached to the buffer. Separate these operations out into functions so that it is easier to modify these operations independently in future.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode_item.c | 154 +++++++++++++++++++++------------------- 1 file changed, 81 insertions(+), 73 deletions(-)fs/xfs/xfs_inode_item.c +12
-27@@ -640,101 +640,64 @@ xfs_inode_item_destroy(/* -* This is the inode flushing I/O completion routine. It is called -* from interrupt level when the buffer containing the inode is -* flushed to disk. It is responsible for removing the inode item -* from the AIL if it has not been re-logged, and unlocking the inode's -* flush lock. -* -* To reduce AIL lock traffic as much as possible, we scan the buffer log item -* list for other inodes that will run this function. We remove them from the -* buffer list so we can process all the inode IO completions in one AIL lock -* traversal. -* -* Note: Now that we attach the log item to the buffer when we first log the -* inode in memory, we can have unflushed inodes on the buffer list here. These -* inodes will have a zero ili_last_fields, so skip over them here. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ void -xfs_iflush_done ( -struct xfs_buf *bp) +xfs_iflush_ail_updates ( +struct xfs_ail *ailp, +struct list_head *list) { -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip, *n; -struct xfs_ail *ailp = bp->b_mount->m_ail; -int need_ail =0 ; -LIST_HEAD (tmp); +struct xfs_log_item *lip; +xfs_lsn_t tail_lsn =0 ; -/* -* Pull the attached inodes from the buffer one at a time and take the -* appropriate action on them. -*/ -list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { - iip =INODE_ITEM (lip); +/* this is an opencoded batch version of xfs_trans_ail_delete */ +spin_lock (&ailp->ail_lock); +list_for_each_entry (lip, list, li_bio_list) { +xfs_lsn_t lsn;
Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_inode_item_destroy( · line 663
Quoted 6 lines
fs/xfs/xfs_inode_item.c +2
-2-if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn) { +clear_bit (XFS_LI_FAILED , &lip->li_flags);continue ; }That seems like strange logic. Shouldn't we clear
LI_FAILEDregardless?
It's the same logic as before this patch series:
if (INODE_ITEM(blip)->ili_logged &&
blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) {
/*
* xfs_ail_update_finish() only cares about the
* lsn of the first tail item removed, any
* others will be at the same or higher lsn so
* we just ignore them.
*/
xfs_lsn_t lsn = xfs_ail_delete_one(ailp, blip);
if (!tail_lsn && lsn)
tail_lsn = lsn;
} else {
xfs_clear_li_failed(blip);
}I've just re-ordered it to check for relogged inodes first instead of handling that in the else branch.
i.e. we do clear XFS_LI_FAILED always: xfs_ail_delete_one() does it for the log items that are being removed from the AIL....
Quoted 28 lines
Quoted 28 lines
fs/xfs/xfs_inode_item.c +28
-0+/* +* Inode buffer IO completion routine. It is responsible for removing inodes +* attached to the buffer from the AIL if they have not been re-logged, as well +* as completing the flush and unlocking the inode. +*/ +void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ; + } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */
Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 743
Quoted 5 lines
fs/xfs/xfs_inode_item.c +5
-0+if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes);Not sure I see the point of having two lists here, particularly since this is all based on lockless logic.
It's not lockless - it's all done under the buffer lock which protects the buffer log item list...
Quoted 3 lines
At the very least it's a subtle
change in AIL processing logic and I don't think that should be buried in a refactoring patch.
I don't think it changes logic at all - what am I missing?
FWIW, I untangled the function this way because the "track dirty inodes by ordered buffers" patchset completely removes the AIL stuff
- the ail_updates list and the xfs_iflush_ail_updates() function go away completely and the rest of the refactoring remains unchanged. i.e. as the commit messages says, this change makes follow-on patches much easier to understand...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 92 lines · Dave Chinner
Quoted 65 lines · Brian Foster
Quoted 64 lines · Dave Chinner
From: Dave Chinner <[email protected]>
xfs_iflush_done()does 3 distinct operations to the inodes attached to the buffer. Separate these operations out into functions so that it is easier to modify these operations independently in future.Signed-off-by Dave Chinner <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>--- fs/xfs/xfs_inode_item.c | 154 +++++++++++++++++++++------------------- 1 file changed, 81 insertions(+), 73 deletions(-)fs/xfs/xfs_inode_item.c +12
-27@@ -640,101 +640,64 @@ xfs_inode_item_destroy(/* -* This is the inode flushing I/O completion routine. It is called -* from interrupt level when the buffer containing the inode is -* flushed to disk. It is responsible for removing the inode item -* from the AIL if it has not been re-logged, and unlocking the inode's -* flush lock. -* -* To reduce AIL lock traffic as much as possible, we scan the buffer log item -* list for other inodes that will run this function. We remove them from the -* buffer list so we can process all the inode IO completions in one AIL lock -* traversal. -* -* Note: Now that we attach the log item to the buffer when we first log the -* inode in memory, we can have unflushed inodes on the buffer list here. These -* inodes will have a zero ili_last_fields, so skip over them here. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ void -xfs_iflush_done ( -struct xfs_buf *bp) +xfs_iflush_ail_updates ( +struct xfs_ail *ailp, +struct list_head *list) { -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip, *n; -struct xfs_ail *ailp = bp->b_mount->m_ail; -int need_ail =0 ; -LIST_HEAD (tmp); +struct xfs_log_item *lip; +xfs_lsn_t tail_lsn =0 ; -/* -* Pull the attached inodes from the buffer one at a time and take the -* appropriate action on them. -*/ -list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { - iip =INODE_ITEM (lip); +/* this is an opencoded batch version of xfs_trans_ail_delete */ +spin_lock (&ailp->ail_lock); +list_for_each_entry (lip, list, li_bio_list) { +xfs_lsn_t lsn;Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_inode_item_destroy( · line 663
Quoted 6 lines
fs/xfs/xfs_inode_item.c +2
-2-if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn) { +clear_bit (XFS_LI_FAILED , &lip->li_flags);continue ; }That seems like strange logic. Shouldn't we clear
LI_FAILEDregardless?It's the same logic as before this patch series:
if (INODE_ITEM(blip)->ili_logged && blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) { /* * xfs_ail_update_finish() only cares about the * lsn of the first tail item removed, any * others will be at the same or higher lsn so * we just ignore them. */ xfs_lsn_t lsn = xfs_ail_delete_one(ailp, blip); if (!tail_lsn && lsn) tail_lsn = lsn; } else { xfs_clear_li_failed(blip); }
Quoted 3 lines
I've just re-ordered it to check for relogged inodes first instead of handling that in the else branch.
Hmm.. I guess I'm confused why the logic seems to be swizzled around. An earlier patch lifted the bit clear outside of this check, then we seem to put it back in place in a different order for some reason..?
Quoted 40 lines
i.e. we do clear
XFS_LI_FAILEDalways:xfs_ail_delete_one()does it for the log items that are being removed from the AIL....Quoted 28 lines
Quoted 28 lines
fs/xfs/xfs_inode_item.c +28
-0+/* +* Inode buffer IO completion routine. It is responsible for removing inodes +* attached to the buffer from the AIL if they have not been re-logged, as well +* as completing the flush and unlocking the inode. +*/ +void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ; + } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */ Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 743
Quoted 5 lines
fs/xfs/xfs_inode_item.c +5
-0+if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes);Not sure I see the point of having two lists here, particularly since this is all based on lockless logic.
Quoted 9 lines
It's not lockless - it's all done under the buffer lock which protects the buffer log item list...
Quoted 3 lines
At the very least it's a subtle
change in AIL processing logic and I don't think that should be buried in a refactoring patch.I don't think it changes logic at all - what am I missing?
I'm referring to the fact that we no longer check the lsn of each (flushed) log item attached to the buffer under the ail lock. Note that I am not saying it's necessarily wrong, but rather that IMO it's too subtle a change to silently squash into a refactoring patch.
Quoted 7 lines
FWIW, I untangled the function this way because the "track dirty inodes by ordered buffers" patchset completely removes the AIL stuff
- theail_updateslist and thexfs_iflush_ail_updates()function go away completely and the rest of the refactoring remains unchanged. i.e. as the commit messages says, this change makes follow-on patches much easier to understand...
The general function breakdown seems fine to me. I find the multiple list processing to be a bit overdone, particularly if it doesn't serve a current functional purpose. If the purpose is to support a future patch series, I'd suggest to continue using the existing logic of moving all flushed inodes to a single list and leave the separate list bits to the start of the series where it's useful so it's possible to review with the associated context (or alternatively just defer the entire patch).
Brian
Quoted 7 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 29 lines · Brian Foster
Quoted 28 lines · Dave Chinner
Quoted 9 lines · Brian Foster
fs/xfs/xfs_inode_item.c xfs_inode_item_destroy( · line 663
Quoted 6 lines · Dave Chinner
fs/xfs/xfs_inode_item.c +2
-2-if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn) { +clear_bit (XFS_LI_FAILED , &lip->li_flags);continue ; }That seems like strange logic. Shouldn't we clear
LI_FAILEDregardless?It's the same logic as before this patch series:
if (INODE_ITEM(blip)->ili_logged && blip->li_lsn == INODE_ITEM(blip)->ili_flush_lsn) { /* * xfs_ail_update_finish() only cares about the * lsn of the first tail item removed, any * others will be at the same or higher lsn so * we just ignore them. */ xfs_lsn_t lsn = xfs_ail_delete_one(ailp, blip); if (!tail_lsn && lsn) tail_lsn = lsn; } else { xfs_clear_li_failed(blip); }
Quoted 6 lines
Quoted 2 lines
I've just re-ordered it to check for relogged inodes first instead of handling that in the else branch.
Hmm.. I guess I'm confused why the logic seems to be swizzled around. An earlier patch lifted the bit clear outside of this check, then we seem to put it back in place in a different order for some reason..?
Oh, you're right - xfs_ail_delete_one() doesn't do that anymore - it got pulled up into xfs_trans_ail_delete() instead. So much stuff has changed in this patchset and I've largely moved on to all the followup stuff now, I'm starting to lose track of what this patchset does and the reasons why I did stuff a couple of months ago...
I'll fix that up.
Quoted 43 lines
Quoted 43 lines
Quoted 27 lines
Quoted 27 lines
+ * Inode buffer IO completion routine. It is responsible for removing inodes
+ * attached to the buffer from the AIL if they have not been re-logged, as well
+ * as completing the flush and unlocking the inode.
+ */fs/xfs/xfs_inode_item.c +23
-0+void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ; + } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */ Quoted 8 lines
fs/xfs/xfs_inode_item.c xfs_iflush_done( · line 743
Quoted 5 lines
fs/xfs/xfs_inode_item.c +5
-0+if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes);Not sure I see the point of having two lists here, particularly since this is all based on lockless logic.
It's not lockless - it's all done under the buffer lock which protects the buffer log item list...
Quoted 3 lines
At the very least it's a subtle
change in AIL processing logic and I don't think that should be buried in a refactoring patch.
Quoted 5 lines
Quoted 2 lines
I don't think it changes logic at all - what am I missing?
I'm referring to the fact that we no longer check the lsn of each (flushed) log item attached to the buffer under the ail lock.
That whole loop in xfs_iflush_ail_updates() runs under the AIL lock, so it does the right thing for anything that is moved to the "ail_updates" list.
If we win the unlocked race (li_lsn does not change) then we move the inode to the ail update list and it gets rechecked under the AIL lock and does the right thing. If we lose the race (li_lsn changes) then the inode has been redirtied and we *don't need to check it under the AIL* - all we need to do is leave it attached to the buffer.
This is the same as the old code: win the race, need_ail is incremented and we recheck under the AIL lock. Lose the race and we don't recheck under the AIL because we don't need to. This happened less under the old code, because it typically only happened with single dirty inodes on a cluster buffer (think directory inode under long running large directory modification operations), but that race most definitely existed and the code most definitely handled it correctly.
Keep in mind that this inode redirtying/AIL repositioning race can even occur /after/ we've locked and removed items from the AIL but before we've run xfs_iflush_finish(). i.e. we can remove it from the AIL but by the time xfs_iflush_finish() runs it's back in the AIL.
Quoted 3 lines
Note that
I am not saying it's necessarily wrong, but rather that IMO it's too subtle a change to silently squash into a refactoring patch.
Except it isn't a change at all. The same subtle issue exists in the code before this patch. It's just that this refactoring makes subtle race conditions that were previously unknown to reviewers so much more obvious they can now see them clearly. That tells me the code is much improved by this refactoring, not that there's a problem that needs reworking....
Quoted 8 lines
Quoted 7 lines
FWIW, I untangled the function this way because the "track dirty inodes by ordered buffers" patchset completely removes the AIL stuff
- theail_updateslist and thexfs_iflush_ail_updates()function go away completely and the rest of the refactoring remains unchanged. i.e. as the commit messages says, this change makes follow-on patches much easier to understand...
Quoted 7 lines
The general function breakdown seems fine to me. I find the multiple list processing to be a bit overdone, particularly if it doesn't serve a current functional purpose. If the purpose is to support a future patch series, I'd suggest to continue using the existing logic of moving all flushed inodes to a single list and leave the separate list bits to the start of the series where it's useful so it's possible to review with the associated context (or alternatively just defer the entire patch).
That's how I originally did it, and it was a mess. it didn't separate cleanly at all, and didn't make future patches much easier at all. Hence I don't think reworking the patch just to look different gains us anything at this point...
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 4 lines · Dave Chinner
Quoted 3 lines · Brian Foster
Quoted 2 lines · Dave Chinner
Quoted 1 line · Brian Foster
On Thu, Jun 04, 2020 at 05:46:05PM +1000, Dave Chinner wrote:
...
Quoted 24 lines
Quoted 3 lines
I'm referring to the fact that we no longer check the lsn of each (flushed) log item attached to the buffer under the ail lock.
That whole loop in
xfs_iflush_ail_updates()runs under the AIL lock, so it does the right thing for anything that is moved to the "ail_updates" list.If we win the unlocked race (
li_lsndoes not change) then we move the inode to the ail update list and it gets rechecked under the AIL lock and does the right thing. If we lose the race (li_lsnchanges) then the inode has been redirtied and we *don't need to check it under the AIL* - all we need to do is leave it attached to the buffer.This is the same as the old code: win the race,
need_ailis incremented and we recheck under the AIL lock. Lose the race and we don't recheck under the AIL because we don't need to. This happened less under the old code, because it typically only happened with single dirty inodes on a cluster buffer (think directory inode under long running large directory modification operations), but that race most definitely existed and the code most definitely handled it correctly.
Quoted 5 lines
Keep in mind that this inode redirtying/AIL repositioning race can even occur /after/ we've locked and removed items from the AIL but before we've run
xfs_iflush_finish(). i.e. we can remove it from the AIL but by the timexfs_iflush_finish()runs it's back in the AIL.
All of the above would make a nice commit log for an independent patch. ;) Note again that I wasn't suggesting the logic was incorrect...
Quoted 11 lines
Quoted 3 lines
Note that
I am not saying it's necessarily wrong, but rather that IMO it's too subtle a change to silently squash into a refactoring patch.Except it isn't a change at all. The same subtle issue exists in the code before this patch. It's just that this refactoring makes subtle race conditions that were previously unknown to reviewers so much more obvious they can now see them clearly. That tells me the code is much improved by this refactoring, not that there's a problem that needs reworking....
This patch elevates a bit of code from effectively being an (ail) lock avoidance optimization to essentially per-item filtering logic without any explanation beyond facilitating future modifications. Independent of whether it's correct, this is not purely a refactoring change IMO.
Quoted 16 lines
Quoted 8 lines
Quoted 7 lines
FWIW, I untangled the function this way because the "track dirty inodes by ordered buffers" patchset completely removes the AIL stuff
- theail_updateslist and thexfs_iflush_ail_updates()function go away completely and the rest of the refactoring remains unchanged. i.e. as the commit messages says, this change makes follow-on patches much easier to understand...Quoted 7 lines
The general function breakdown seems fine to me. I find the multiple list processing to be a bit overdone, particularly if it doesn't serve a current functional purpose. If the purpose is to support a future patch series, I'd suggest to continue using the existing logic of moving all flushed inodes to a single list and leave the separate list bits to the start of the series where it's useful so it's possible to review with the associated context (or alternatively just defer the entire patch).
Quoted 5 lines
That's how I originally did it, and it was a mess. it didn't separate cleanly at all, and didn't make future patches much easier at all. Hence I don't think reworking the patch just to look different gains us anything at this point...
I find that hard to believe. This patch splits the buffer list into two lists, processes the first one, immediately combines it with the second, then processes the second which is no different from the single list that was constructed by the original code. The only reasons I can see for this kind of churn is either to address some kind of performance or efficiency issue or if the lists are used for further changes. The former is not a documented reason and there's no context for the latter because it's apparently part of some future series.
TBH, I think this patch should probably be broken down into two or three independent patches anyways. What's the issue with something like the appended diff (on top of this patch) in the meantime? If the multiple list logic is truly necessary, reintroduce it when it's used so it's actually reviewable..
Brian
--- 8< ---
fs/xfs/xfs_inode_item.c +12 -13
@@ -718,8 +718,8 @@ xfs_iflush_done( struct xfs_buf *bp) {struct xfs_log_item *lip, *n; -LIST_HEAD (flushed_inodes); -LIST_HEAD (ail_updates); +int need_ail =0 ; +LIST_HEAD (tmp);/* * Pull the attached inodes from the buffer one at a time and take the
@@ -732,25 +732,24 @@ xfs_iflush_done(xfs_iflush_abort (iip->ili_inode);continue ; } +if (!iip->ili_last_fields)continue ; -/* Do an unlocked check for needing the AIL lock. */ +list_move_tail (&lip->li_bio_list, &tmp); + +/* Do an unlocked check for needing AIL processing */ if (iip->ili_flush_lsn == lip->li_lsn ||test_bit (XFS_LI_FAILED , &lip->li_flags)) -list_move_tail (&lip->li_bio_list, &ail_updates); -else -list_move_tail (&lip->li_bio_list, &flushed_inodes); + need_ail++; } -if (!list_empty (&ail_updates)) { -xfs_iflush_ail_updates (bp->b_mount->m_ail, &ail_updates); -list_splice_tail (&ail_updates, &flushed_inodes); - } +if (need_ail) +xfs_iflush_ail_updates (bp->b_mount->m_ail, &tmp); -xfs_iflush_finish (bp, &flushed_inodes); -if (!list_empty (&flushed_inodes)) -list_splice_tail (&flushed_inodes, &bp->b_li_list); +xfs_iflush_finish (bp, &tmp); +if (!list_empty (&tmp)) +list_splice_tail (&tmp, &bp->b_li_list); }/*
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 3 lines · Brian Foster
TBH, I think this patch should probably be broken down into two or three independent patches anyways.
To what end? The patch is already small, it's simple to understand and it's been tested. What does breaking it up into a bunch more smaller patches actually gain us?
It means hours more work on my side without any change in the end result. It's -completely wasted effort- if all I'm doing this for is to get you to issue a RVB on it. Fine grained patches do not come for free, and in a patch series that is already 30 patches long making it even longer just increases the time and resources it costs *me* to maintian it until it is merged.
Quoted 4 lines
What's the issue with something like the
appended diff (on top of this patch) in the meantime? If the multiple list logic is truly necessary, reintroduce it when it's used so it's actually reviewable..
Nothing. Except it causes conflicts further through my patch set which do the work of removing this AIL specific code. IOWs, it just *increases the amount of work I have to do* without actually providing any benefit to anyone...
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
Quoted 10 lines · Dave Chinner
Quoted 3 lines · Brian Foster
TBH, I think this patch should probably be broken down into two or three independent patches anyways.
To what end? The patch is already small, it's simple to understand and it's been tested. What does breaking it up into a bunch more smaller patches actually gain us?
Quoted 7 lines
It means hours more work on my side without any change in the end result. It's -completely wasted effort- if all I'm doing this for is to get you to issue a RVB on it. Fine grained patches do not come for free, and in a patch series that is already 30 patches long making it even longer just increases the time and resources it costs *me* to maintian it until it is merged.
This links back to a conversation we started about improving the review process.
One of the questions was regarding RVB being too "binary". If I am new to the subsystem, obviously my RVB weights less, but both Darrick and Dave expressed their desire to get review by more eyes to double check the "small details".
To that end, Darrick has suggested the RVB be accompanied with "verified correctness" "verified design" or whatnot.
So instead of a binary RVB, Brian could express his review outcome in a machine friendly way, because this:
"TBH, I think this patch should probably be broken down..." would be interpreted quite differently depending on culture.
My interpretation is: "ACK on correctness", "ACK on design", "not happy about patch breakdown", "not a NACK", but I could be wrong.
Then, instead of a rigid rule for maintainer to require two RVB per patch, we can employ more fine grained rules, for example:
- No NACKs
- Two RVB for correctness
- One RVB for design
- etc..
Also, it could help to write down review rules for the subsystem in a bureaucratic document that can be agreed on, so that not every patch needs to have the discussion about whether breaking this patch up is mandatory or not.
There is a big difference between saying: "this patch is too big for me to review, I will not be doing as good a job of review if it isn't split into patches" and "this patch has already been reviewed,
I already know everything there is to know about it, there are no bisect-ability issues, but for aesthetics, I think it should be broken down". I am not saying that latter opinion should not be voiced, but when said it should be said explicitly.
High coding standards have gotten xfs very far, but failing to maintain a healthy balance with pragmatism will inevitably come with a price. Need I remind you that the issue that Dave is fixing has been affecting real users and it has been reported over three years ago?
Thanks,
Amir.
Re: [PATCH 29/30] xfs: factor xfs_iflush_done
In reply to Dave ChinnerQuoted 9 lines · Dave Chinner
Quoted 3 lines · Brian Foster
TBH, I think this patch should probably be broken down into two or three independent patches anyways.
To what end? The patch is already small, it's simple to understand and it's been tested. What does breaking it up into a bunch more smaller patches actually gain us?
I think you overestimate the simplicity to somebody who doesn't have context on whatever upcoming changes you have. I spent more time staring at this wondering what the list filtering logic was for than I would have needed to review the entire patch were those changes not included.
Quoted 7 lines
It means hours more work on my side without any change in the end result. It's -completely wasted effort- if all I'm doing this for is to get you to issue a RVB on it. Fine grained patches do not come for free, and in a patch series that is already 30 patches long making it even longer just increases the time and resources it costs *me* to maintian it until it is merged.
Note that I said "two or three" and then sent you a diff that breaks it down into two. That addresses my concern.
Quoted 10 lines
Quoted 4 lines
What's the issue with something like the
appended diff (on top of this patch) in the meantime? If the multiple list logic is truly necessary, reintroduce it when it's used so it's actually reviewable..Nothing. Except it causes conflicts further through my patch set which do the work of removing this AIL specific code. IOWs, it just *increases the amount of work I have to do* without actually providing any benefit to anyone...
Reapply the list filtering logic (reverting the same diff I already sent) at the beginning of your upcoming series that uses it. I sent the diff as a courtesy because you seem to be rather frustrated wrt to any suggestion to change this series, but this seems like a standard case of misplaced code to me with a simple fix. The fact that this is used somehow or another in a series that is so far unposted and unreviewed is not a valid justification IMO. I really don't understand what the issue is here wrt to moving the changes to where they're used.
Brian
Quoted 5 lines
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v2 29/30 xfs: factor xfs_iflush_done
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
xfs_iflush_done() does 3 distinct operations to the inodes attached to the buffer. Separate these operations out into functions so that it is easier to modify these operations independently in future.
---
V2:
- always clear XFS_LI_FAILED for items requiring AIL processing
fs/xfs/xfs_inode_item.c | 155 +++++++++++++++++++++++++----------------------- 1 file changed, 81 insertions(+), 74 deletions(-)
fs/xfs/xfs_inode_item.c +81 -74
@@ -641,101 +641,63 @@ xfs_inode_item_destroy(/* -* This is the inode flushing I/O completion routine. It is called -* from interrupt level when the buffer containing the inode is -* flushed to disk. It is responsible for removing the inode item -* from the AIL if it has not been re-logged, and unlocking the inode's -* flush lock. -* -* To reduce AIL lock traffic as much as possible, we scan the buffer log item -* list for other inodes that will run this function. We remove them from the -* buffer list so we can process all the inode IO completions in one AIL lock -* traversal. -* -* Note: Now that we attach the log item to the buffer when we first log the -* inode in memory, we can have unflushed inodes on the buffer list here. These -* inodes will have a zero ili_last_fields, so skip over them here. +* We only want to pull the item from the AIL if it is actually there +* and its location in the log has not changed since we started the +* flush. Thus, we only bother if the inode's lsn has not changed. */ void -xfs_iflush_done ( -struct xfs_buf *bp) +xfs_iflush_ail_updates ( +struct xfs_ail *ailp, +struct list_head *list) { -struct xfs_inode_log_item *iip; -struct xfs_log_item *lip, *n; -struct xfs_ail *ailp = bp->b_mount->m_ail; -int need_ail =0 ; -LIST_HEAD (tmp); +struct xfs_log_item *lip; +xfs_lsn_t tail_lsn =0 ; -/* -* Pull the attached inodes from the buffer one at a time and take the -* appropriate action on them. -*/ -list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { - iip =INODE_ITEM (lip); - -if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { -xfs_iflush_abort (iip->ili_inode); -continue ; - } +/* this is an opencoded batch version of xfs_trans_ail_delete */ +spin_lock (&ailp->ail_lock); +list_for_each_entry (lip, list, li_bio_list) { +xfs_lsn_t lsn; -if (!iip->ili_last_fields) +clear_bit (XFS_LI_FAILED , &lip->li_flags); +if (INODE_ITEM (lip)->ili_flush_lsn != lip->li_lsn)continue ; -list_move_tail (&lip->li_bio_list, &tmp); - -/* Do an unlocked check for needing the AIL lock. */ -if (iip->ili_flush_lsn == lip->li_lsn || -test_bit (XFS_LI_FAILED , &lip->li_flags)) - need_ail++; + lsn =xfs_ail_delete_one (ailp, lip); +if (!tail_lsn && lsn) + tail_lsn = lsn; } +xfs_ail_update_finish (ailp, tail_lsn); +} -/* -* We only want to pull the item from the AIL if it is actually there -* and its location in the log has not changed since we started the -* flush. Thus, we only bother if the inode's lsn has not changed. -*/ -if (need_ail) { -xfs_lsn_t tail_lsn =0 ; - -/* this is an opencoded batch version of xfs_trans_ail_delete */ -spin_lock (&ailp->ail_lock); -list_for_each_entry (lip, &tmp, li_bio_list) { -clear_bit (XFS_LI_FAILED , &lip->li_flags); -if (lip->li_lsn ==INODE_ITEM (lip)->ili_flush_lsn) { -xfs_lsn_t lsn =xfs_ail_delete_one (ailp, lip); -if (!tail_lsn && lsn) - tail_lsn = lsn; - } - } -xfs_ail_update_finish (ailp, tail_lsn); - } +/* +* Walk the list of inodes that have completed their IOs. If they are clean +* remove them from the list and dissociate them from the buffer. Buffers that +* are still dirty remain linked to the buffer and on the list. Caller must +* handle them appropriately. +*/ +void +xfs_iflush_finish ( +struct xfs_buf *bp, +struct list_head *list) +{ +struct xfs_log_item *lip, *n; -/* -* Clean up and unlock the flush lock now we are done. We can clear the -* ili_last_fields bits now that we know that the data corresponding to -* them is safely on disk. -*/ -list_for_each_entry_safe (lip, n, &tmp, li_bio_list) { +list_for_each_entry_safe (lip, n, list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip);bool drop_buffer = false; -list_del_init (&lip->li_bio_list); - iip =INODE_ITEM (lip); -spin_lock (&iip->ili_lock);/* * Remove the reference to the cluster buffer if the inode is -* clean in memory. Drop the buffer reference once we've dropped -* the locks we hold. If the inode is dirty in memory, we need -* to put the inode item back on the buffer list for another -* pass through the flush machinery. +* clean in memory and drop the buffer reference once we've +* dropped the locks we hold. */ ASSERT (iip->ili_item.li_buf == bp);if (!iip->ili_fields) { iip->ili_item.li_buf =NULL ; +list_del_init (&lip->li_bio_list); drop_buffer = true; - }else { -list_add (&lip->li_bio_list, &bp->b_li_list); } iip->ili_last_fields =0 ; iip->ili_flush_lsn =0 ;
@@ -746,6 +708,51 @@ xfs_iflush_done( } } +/* +* Inode buffer IO completion routine. It is responsible for removing inodes +* attached to the buffer from the AIL if they have not been re-logged, as well +* as completing the flush and unlocking the inode. +*/ +void +xfs_iflush_done ( +struct xfs_buf *bp) +{ +struct xfs_log_item *lip, *n; +LIST_HEAD (flushed_inodes); +LIST_HEAD (ail_updates); + +/* +* Pull the attached inodes from the buffer one at a time and take the +* appropriate action on them. +*/ +list_for_each_entry_safe (lip, n, &bp->b_li_list, li_bio_list) { +struct xfs_inode_log_item *iip =INODE_ITEM (lip); + +if (xfs_iflags_test (iip->ili_inode,XFS_ISTALE )) { +xfs_iflush_abort (iip->ili_inode); +continue ; + } +if (!iip->ili_last_fields) +continue ; + +/* Do an unlocked check for needing the AIL lock. */ +if (iip->ili_flush_lsn == lip->li_lsn || +test_bit (XFS_LI_FAILED , &lip->li_flags)) +list_move_tail (&lip->li_bio_list, &ail_updates); +else +list_move_tail (&lip->li_bio_list, &flushed_inodes); + } + +if (!list_empty (&ail_updates)) { +xfs_iflush_ail_updates (bp->b_mount->m_ail, &ail_updates); +list_splice_tail (&ail_updates, &flushed_inodes); + } + +xfs_iflush_finish (bp, &flushed_inodes); +if (!list_empty (&flushed_inodes)) +list_splice_tail (&flushed_inodes, &bp->b_li_list); +} +/* * This is the inode flushing abort routine. It is called from xfs_iflush when * the filesystem is shutting down to clean up the inode state. It is
PATCH 30/30 xfs: remove xfs_inobp_check()
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
This debug code is called on every xfs_iflush() call, which then checks every inode in the buffer for non-zero unlinked list field. Hence it checks every inode in the cluster buffer every time a single inode on that cluster it flushed. This is resulting in:
- 38.91% 5.33% [kernel] [k] xfs_iflush
- 17.70% xfs_iflush
- 9.93% xfs_inobp_check
4.36% xfs_buf_offset
10% of the CPU time spent flushing inodes is repeatedly checking unlinked fields in the buffer. We don't need to do this.
The other place we call xfs_inobp_check() is xfs_iunlink_update_dinode(), and this is after we've done this assert for the agino we are about to write into that inode:
ASSERT(xfs_verify_agino_or_null(mp, agno, next_agino));
which means we've already checked that the agino we are about to write is not 0 on debug kernels. The inode buffer verifiers do everything else we need, so let's just remove this debug code.
--- fs/xfs/libxfs/xfs_inode_buf.c | 24 ------------------------ fs/xfs/libxfs/xfs_inode_buf.h | 6 ------ fs/xfs/xfs_inode.c | 2 -- 3 files changed, 32 deletions(-)
fs/xfs/libxfs/xfs_inode_buf.c +0 -24
@@ -20,30 +20,6 @@#include <linux/iversion.h> -/* -* Check that none of the inode's in the buffer have a next -* unlinked field of 0. -*/ -#if defined(DEBUG ) -void -xfs_inobp_check ( -xfs_mount_t *mp, -xfs_buf_t *bp) -{ -int i; -xfs_dinode_t *dip; - -for (i =0 ; i <M_IGEO (mp)->inodes_per_cluster; i++) { - dip =xfs_buf_offset (bp, i * mp->m_sb.sb_inodesize); -if (!dip->di_next_unlinked) { -xfs_alert (mp, -"Detected bogus zero next_unlinked field in inode %d buffer 0x%llx." , - i, (long long )bp->b_bn); - } - } -} -#endif -/* * If we are doing readahead on an inode buffer, we might be in log recovery * reading an inode allocation buffer that hasn't yet been replayed, and hence
fs/xfs/libxfs/xfs_inode_buf.h +0 -6
@@ -52,12 +52,6 @@ int xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from);void xfs_log_dinode_to_disk (struct xfs_log_dinode *from,struct xfs_dinode *to); -#if defined(DEBUG ) -void xfs_inobp_check (struct xfs_mount *,struct xfs_buf *); -#else -#define xfs_inobp_check (mp, bp) -#endif /* DEBUG */ -xfs_failaddr_t xfs_dinode_verify (struct xfs_mount *mp,xfs_ino_t ino,struct xfs_dinode *dip); xfs_failaddr_t xfs_inode_validate_extsize(struct xfs_mount *mp,
fs/xfs/xfs_inode.c +0 -2
@@ -2165,7 +2165,6 @@ xfs_iunlink_update_dinode(xfs_dinode_calc_crc (mp, dip);xfs_trans_inode_buf (tp, ibp);xfs_trans_log_buf (tp, ibp, offset, offset +sizeof (xfs_agino_t) -1 ); -xfs_inobp_check (mp, ibp); }/* Set an in-core inode's unlinked pointer and return the old value. */
@@ -3559,7 +3558,6 @@ xfs_iflush(xfs_iflush_fork (ip, dip, iip,XFS_DATA_FORK );if (XFS_IFORK_Q (ip))xfs_iflush_fork (ip, dip, iip,XFS_ATTR_FORK ); -xfs_inobp_check (mp, bp);/* * We've recorded everything logged in the inode, so we'd like to clear
Signature · 1 line
2.26.2.761.g0e0b3e54be
Re: [PATCH 30/30] xfs: remove xfs_inobp_check()
Quoted 21 lines · Dave Chinner
From: Dave Chinner <[email protected]>
This debug code is called on every
xfs_iflush()call, which then checks every inode in the buffer for non-zero unlinked list field. Hence it checks every inode in the cluster buffer every time a single inode on that cluster it flushed. This is resulting in:- 38.91% 5.33% [kernel] [k]
xfs_iflush
- 17.70%xfs_iflush
- 9.93%xfs_inobp_check
4.36%xfs_buf_offset10% of the CPU time spent flushing inodes is repeatedly checking unlinked fields in the buffer. We don't need to do this.
The other place we call
xfs_inobp_check()isxfs_iunlink_update_dinode(), and this is after we've done this assert for the agino we are about to write into that inode:ASSERT(xfs_verify_agino_or_null(mp, agno, next_agino));
Quoted 8 lines
which means we've already checked that the agino we are about to write is not 0 on debug kernels. The inode buffer verifiers do everything else we need, so let's just remove this debug code.
Signed-off-by Dave Chinner <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>---
Quoted 80 lines
fs/xfs/libxfs/xfs_inode_buf.c | 24 ------------------------ fs/xfs/libxfs/xfs_inode_buf.h | 6 ------ fs/xfs/xfs_inode.c | 2 -- 3 files changed, 32 deletions(-)fs/xfs/libxfs/xfs_inode_buf.c +0
-24@@ -20,30 +20,6 @@#include <linux/iversion.h> -/* -* Check that none of the inode's in the buffer have a next -* unlinked field of 0. -*/ -#if defined(DEBUG ) -void -xfs_inobp_check ( -xfs_mount_t *mp, -xfs_buf_t *bp) -{ -int i; -xfs_dinode_t *dip; - -for (i =0 ; i <M_IGEO (mp)->inodes_per_cluster; i++) { - dip =xfs_buf_offset (bp, i * mp->m_sb.sb_inodesize); -if (!dip->di_next_unlinked) { -xfs_alert (mp, -"Detected bogus zero next_unlinked field in inode %d buffer 0x%llx." , - i, (long long )bp->b_bn); - } - } -} -#endif -/* * If we are doing readahead on an inode buffer, we might be in log recovery * reading an inode allocation buffer that hasn't yet been replayed, and hence fs/xfs/libxfs/xfs_inode_buf.h +0
-6@@ -52,12 +52,6 @@ int xfs_inode_from_disk(struct xfs_inode *ip, struct xfs_dinode *from);void xfs_log_dinode_to_disk (struct xfs_log_dinode *from,struct xfs_dinode *to); -#if defined(DEBUG ) -void xfs_inobp_check (struct xfs_mount *,struct xfs_buf *); -#else -#define xfs_inobp_check (mp, bp) -#endif /* DEBUG */ -xfs_failaddr_t xfs_dinode_verify (struct xfs_mount *mp,xfs_ino_t ino,struct xfs_dinode *dip); xfs_failaddr_t xfs_inode_validate_extsize(struct xfs_mount *mp,fs/xfs/xfs_inode.c +0
-2@@ -2165,7 +2165,6 @@ xfs_iunlink_update_dinode(xfs_dinode_calc_crc (mp, dip);xfs_trans_inode_buf (tp, ibp);xfs_trans_log_buf (tp, ibp, offset, offset +sizeof (xfs_agino_t) -1 ); -xfs_inobp_check (mp, ibp); }/* Set an in-core inode's unlinked pointer and return the old value. */ @@ -3559,7 +3558,6 @@ xfs_iflush(xfs_iflush_fork (ip, dip, iip,XFS_DATA_FORK );if (XFS_IFORK_Q (ip))xfs_iflush_fork (ip, dip, iip,XFS_ATTR_FORK ); -xfs_inobp_check (mp, bp);/* * We've recorded everything logged in the inode, so we'd like to clear Signature · 1 line
2.26.2.761.g0e0b3e54be