fs: Deferred inode reclaim
Subsystems (5)
Jan Kara
maintainerv3 cover fs: Deferred inode reclaim
Hello,
here is the third revision of my patches implementing deferred inode reclaim to deal with MM warnings due to GFP_NOFAIL allocations from reclaim paths. This happens because to reclaim some inodes, filesystems have to do IO including complex operations using journalling and forward progress of these depends on successful memory allocations (which is impossible to guarantee from reclaim context).
Since the last revision the most notable changes include addition of automatic deferal of inode eviction from iput() when inode link-count is 0 and we are in reclaim (PF_MEMALLOC flag set). This path gets triggered e.g. by overlayfs whose inodes hold dentry references on upper filesystem and thus slab reclaim of overlayfs inodes can trigger the last dput() (and thus iput()) of already unlinked files. Since inode deletion is practically guaranteed to require IO (and thus possibly e.g. loading allocation information into memory) I've decided to unconditionally defer reclaim in this case instead of marking inodes in each filesystem.
Another bigger change in this series is that we now protect deferred inode reclaim from filesystem freezing. Otherwise inode eviction could block the reclaim worker on frozen filesystem which could stall deferred inode reclaim on the whole system. Deferred reclaim workers now skip reclaiming inodes on frozen filesystem and we block filesystem freezing while the worker is evicting current batch of its inodes.
The last bigger change is that mark_inode_reclaim_deferred() does not block when the queue of deferred inodes grows over limit anymore. Instead it queues task work to delay the task appropriately when returning to userspace. This solves possible lockups when tasks would get throttled when calling mark_inode_reclaim_deferred() with some locks or transaction started.
The patches have survived full fstests run on ext4 when marking all loaded ext4 inodes for deferred reclaim to stress the deferred reclaim paths. I have also tested (when marking all inodes for deferred reclaim) the patches by creating a memcg with 128m memory limit and then scanning from 10 processes 1m inodes in total which heavily exercises deferred inode reclaim paths. This also verified efficiency of throttling of marking of inodes for deferred reclaim. Basically we cannot guarantee any particular limit on the number of inodes queued for deferred reclaim as that is bound only by their number in memory and memory pressure. But as the number of queued inodes grows, the tasks creating them get slowed down so eventually some equilibrium is hit. In my VM this was at ~10k inodes (only slightly above the 8k limit when throttling kicks in) but this all very much depends on the reclaim pressure, speed of deferred reclaim etc. so I don't think this is some representative number.
The first patch is a pure fix for a problem I've hit when running fstests. The second patch deals with lazy timestamp updates which I've decided to handle better directly inside the writeback infrastructure instead of deferring reclaim. The remaining patches implement the inode reclaim deferal. Ext4 use of deferred inode reclaim is there mostly as a demonstration. Other filesystems need to determine which inodes need deferred reclaim and mark them as such which generally requires good internal knowledge of the filesystem. I'm hoping that once the infrastructure is there, fs developers prompted by MM warnings will start using it :). I know for a fact that besides ext4 e.g. gfs2 wants to use this infrastructure.
Honza
Changes since v2:
* Some cleanups and simplifications from Andreas Grunbacher
* Fixed couple of bugs pointed out by Sashiko
* Added support for auto-deferring eviction from iput() when inode has 0 link count and we are in reclaim path
* Added protection of deferred reclaim from filesystem freezing
* Throttling now doesn't happen directly in mark_inode_reclaim_deferred() but on exit to userspace
Changes since v1:
* Multiple workers to reclaim deferred inodes, related to that I've moved lists of deferred inodes from per-sb to global ones
* Added missed lazytime sync fix
Previous versions:
Jan Kara
maintainerPATCH v3 1/7 writeback: Fix missed lazytime flush
move_expired_inodes() moves to b_io list inodes which were dirtied at current_time - dirtytime_expire_interval or before. __writeback_single_inode() however calls sync_lazytime() only if the inode was dirtied strictly before current_time - dirtytime_expire_interval. It can thus happen that inode with dirty timestamps gets queued to b_io list, then fails the check in __writeback_single_inode() and as a result is just placed back to b_dirty_time list with a refreshed timestamp and so timestamps aren't written when they should have been.
This is a very difficult race to hit in practice but changes to how dirty time expiry is handled on last iput() will make this much easier to hit.
--- fs/fs-writeback.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/fs-writeback.c +2 -2
@@ -1822,8 +1822,8 @@ __writeback_single_inode(struct inode *inode, struct writeback_control *wbc)*/ if ((inode_state_read_once (inode) &I_DIRTY_TIME ) && (wbc->sync_mode ==WB_SYNC_ALL || -time_after (jiffies, inode->dirtied_time_when + - dirtytime_expire_interval *HZ ))) +time_after_eq (jiffies, inode->dirtied_time_when + + dirtytime_expire_interval *HZ )))sync_lazytime (inode);/*
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 2/7 fs: Avoid inode dirtying on last iput
In reply to Jan KaraWhen inode has dirtied timestamps, we currently call sync_lazytime() on last iput. This is done because inode with any dirty bit set is not inserted into LRU and dirty timestamps expire only after many (12 by default) hours so these inodes would be sitting outside of LRU aging for a really long time. However this can result in doing IO and consequently GFP_NOFAIL allocations from dentry reclaim making MM complain. Sample trace for ext4 is:
prune_dcache_sbshrink_dentry_list__dentry_kill
iputsync_lazytime__mark_inode_dirtyext4_dirty_inode__ext4_mark_inode_dirtyext4_reserve_inode_writeext4_get_inode_locbdev_getblk__filemap_get_folio_mpol
Avoid this dirtying on last iput by reshuffling unused inodes to the beginning of b_dirty_time list and clobbering dirtied_time_when instead so that they get written during next periodic writeback.
--- fs/fs-writeback.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ fs/inode.c | 15 +++++++-------- fs/internal.h | 1 + 3 files changed, 52 insertions(+), 8 deletions(-)
fs/fs-writeback.c +44 -0
@@ -2852,6 +2852,50 @@ void __mark_inode_dirty(struct inode *inode, int flags) }EXPORT_SYMBOL (__mark_inode_dirty); +/* +* If inode has dirty timestamps to write out, make sure flush worker writes +* them out during its next periodic writeback writeout. +*/ +void queue_dirtytime_writeback (struct inode *inode) +{ +struct bdi_writeback *wb; +unsigned long new_time; + +lockdep_assert_held (&inode->i_lock); + + wb =locked_inode_to_wb_and_lock_list (inode); +spin_lock (&inode->i_lock); +/* +* If inode writeback is already queued or inode got dirty, we have +* nothing to do and we mustn't touch writeback lists anyway. +*/ +if (inode_state_read (inode) & (I_SYNC_QUEUED |I_DIRTY )) + goto out_wb_lock; +/* Written back while we dropped i_lock? */ +if (!(inode_state_read (inode) &I_DIRTY_TIME )) + goto out_wb_lock; + +/* +* Move inode to the beginning of dirty queue and clobber dirtied time +* so that it gets written out during the next periodic writeback. +*/ + new_time = jiffies - dirtytime_expire_interval *HZ ; +if (!list_empty (&wb->b_dirty_time)) { +struct inode *first =wb_inode (wb->b_dirty_time.prev); +unsigned long first_time =READ_ONCE (first->dirtied_time_when); + +if (time_before (first_time, new_time)) + new_time = first_time; + } + inode->dirtied_when = new_time; + inode->dirtied_time_when = new_time; +list_move_tail (&inode->i_io_list, &wb->b_dirty_time); +/* Wake up flush worker to ensure timely flush of the inode */ +wb_wakeup_delayed (wb); +out_wb_lock: +spin_unlock (&wb->list_lock); +} +/* * The @s_sync_lock is used to serialise concurrent sync operations * to avoid lock contention problems with concurrent wait_sb_inodes() calls.
fs/inode.c +7 -8
@@ -2029,7 +2029,6 @@ void iput(struct inode *inode)if (unlikely (!inode))return ; -retry:lockdep_assert_not_held (&inode->i_lock);VFS_BUG_ON_INODE (inode_state_read_once (inode) & (I_FREEING |I_CLEAR ), inode);/*
@@ -2042,14 +2041,14 @@ void iput(struct inode *inode)if (atomic_add_unless (&inode->i_count,-1 ,1 ))return ; -if (inode->i_nlink &&sync_lazytime (inode)) - goto retry; -spin_lock (&inode->i_lock); -if (unlikely ((inode_state_read (inode) &I_DIRTY_TIME ) && inode->i_nlink)) { -spin_unlock (&inode->i_lock); - goto retry; - } +/* +* If inode has timestamp updates pending, queue flushing them now as +* otherwise the dirtiness could be preventing the inode from entering +* LRU for hours. +*/ +if (inode->i_nlink &&inode_state_read (inode) &I_DIRTY_TIME ) +queue_dirtytime_writeback (inode);if (!atomic_dec_and_test (&inode->i_count)) {spin_unlock (&inode->i_lock);
fs/internal.h +1 -0
@@ -219,6 +219,7 @@ bool in_group_or_capable(struct mnt_idmap *idmap,*/ long get_nr_dirty_inodes (void );bool sync_lazytime (struct inode *inode); +void queue_dirtytime_writeback (struct inode *inode);/* * dcache.c
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 3/7 fs: Basic infrastructure for offloading inode reclaim
In reply to Jan KaraReclaim of some inodes is rather complex requiring running transactions or doing other IO. Consequently filesystems end up doing GFP_NOFAIL allocations from kswapd or even direct reclaim which is problematic because forward progress of these allocations isn't guaranteed. Add infrastructure for marking inodes whose reclaim is difficult and offload reclaim of such inodes into a workqueue to not block kswapd with difficult inode reclaim. The processing of the queued inodes happens in parallel from several work items so that the reclaim has higher chances of keeping up with the inflow of inodes if reclaim of some inodes progresses slowly.
--- fs/inode.c | 212 ++++++++++++++++++++++++++++++- fs/super.c | 2 + include/linux/fs.h | 3 + include/linux/fs/super_types.h | 3 + include/trace/events/writeback.h | 3 +- 5 files changed, 219 insertions(+), 4 deletions(-)
fs/inode.c +209 -3
@@ -74,6 +74,25 @@ static DEFINE_PER_CPU(unsigned long, nr_unused);static struct kmem_cache *inode_cachep__ro_after_init ; +struct deferred_reclaim_queue { +struct list_head list; +spinlock_t lock; +}; + +/* Inodes for deferred reclaim */ +struct inode_deferred_reclaim { +int workers;/* Number of parallel workers */ +struct workqueue_struct *wq;/* Workqueue for workers */ +struct work_struct *work;/* Workers */ +/* +* Queue of deferred inodes. The number of queues is the same number +* as the number of workers +*/ +struct deferred_reclaim_queue *queues; +}; + +static struct inode_deferred_reclaim deferred_reclaim; +static long get_nr_inodes (void ) {int i;
@@ -949,7 +968,7 @@ static enum lru_status inode_lru_isolate(struct list_head *item,* sync, or the last page cache deletion will requeue them. */ if (icount_read (inode) || - (inode_state_read (inode) & ~I_REFERENCED ) || +inode_state_read (inode) & ~(I_REFERENCED |I_DEFER_RECLAIM ) || !mapping_shrinkable (&inode->i_data)) {list_lru_isolate (lru, &inode->i_lru);spin_unlock (&inode->i_lock);
@@ -994,6 +1013,38 @@ static enum lru_status inode_lru_isolate(struct list_head *item,return LRU_REMOVED ; } +/* Number of inodes to process in one reclaim batch */ +#define INODE_RECLAIM_BATCH_SIZE 16 + +static int get_deferred_sb_id (struct super_block *sb) +{ +return hash_ptr (sb,32 ) % deferred_reclaim.workers; +} + +static void queue_deferred_reclaim (struct super_block *sb,int count, +struct list_head *list) +{ +int id =get_deferred_sb_id (sb); +struct deferred_reclaim_queue *queue = + &deferred_reclaim.queues[id]; +int i, wake_count; + +spin_lock (&queue->lock); +list_splice_tail (list, &queue->list); +atomic_add (count, &sb->s_deferred_reclaim_count); +spin_unlock (&queue->lock); + +/* Queue works to process inodes we've added to the list */ + wake_count =DIV_ROUND_UP (count,INODE_RECLAIM_BATCH_SIZE ); +if (wake_count > deferred_reclaim.workers) + wake_count = deferred_reclaim.workers; +for (i =0 ; i < wake_count; i++) { +queue_work (deferred_reclaim.wq, + &deferred_reclaim.work[id]); + id = (id +1 ) % deferred_reclaim.workers; + } +} +/* * Walk the superblock inode LRU for freeable inodes and attempt to free them. * This is called from the superblock shrinker function with a number of inodes
@@ -1003,14 +1054,166 @@ static enum lru_status inode_lru_isolate(struct list_head *item,long prune_icache_sb (struct super_block *sb,struct shrink_control *sc) {LIST_HEAD (freeable); -long freed; +long freed, deferred =0 ; +struct inode *inode, *tmp; freed =list_lru_shrink_walk (&sb->s_inode_lru, sc, inode_lru_isolate, &freeable); -dispose_list (&freeable); + +list_for_each_entry_safe (inode, tmp, &freeable, i_lru) { +if (inode_state_read_once (inode) &I_DEFER_RECLAIM ) { + deferred++; +continue ; + } +list_del_init (&inode->i_lru); +evict (inode); +cond_resched (); + } + +if (!list_empty (&freeable)) +queue_deferred_reclaim (sb, deferred, &freeable);return freed; } +static void __get_inode_reclaim_batch (int *start,int end, +struct list_head *list,int *count) +{ +int i; + +for (i = *start; i < end; i++) { +struct deferred_reclaim_queue *queue = + &deferred_reclaim.queues[i]; +struct list_head *entry, tmp; + +if (list_empty_careful (&queue->list)) +continue ; +spin_lock (&queue->lock); +list_for_each (entry, &queue->list) { +if (*count >=INODE_RECLAIM_BATCH_SIZE ) +break ; + (*count)++; + } +list_cut_before (&tmp, &queue->list, entry); +list_splice_tail (&tmp, list); +spin_unlock (&queue->lock); +if (*count >=INODE_RECLAIM_BATCH_SIZE ) +break ; + } + *start = i; +} + +/* Move a batch of inodes from queued lists to our private list */ +static int get_inode_reclaim_batch (int *id,struct list_head *list) +{ +int count =0 ; +int orig_id = *id; + +__get_inode_reclaim_batch (id, deferred_reclaim.workers, list, &count); +if (count >=INODE_RECLAIM_BATCH_SIZE ) +return count; +/* Wrap around */ +if (orig_id >0 ) { + *id =0 ; +__get_inode_reclaim_batch (id, orig_id, list, &count); + } +return count; +} + + +static void inode_reclaim_deferred (struct work_struct *work) +{ +struct inode *inode; +LIST_HEAD (inode_batch); +/* +* We start with the list corresponding to the work but rolling a dice +* would work as well +*/ +int id = work - deferred_reclaim.work; + +while (1 ) { +if (!get_inode_reclaim_batch (&id, &inode_batch)) +break ; +while (!list_empty (&inode_batch)) { +struct super_block *sb; + +/* +* inode_batch list is private and I_FREEING flags +* protect us from anybody else trying to remove the +* inode from the LRU list. No locking needed. +*/ + inode = list_first_entry(&inode_batch, struct inode, + i_lru); +list_del_init (&inode->i_lru); + sb = inode->i_sb; +evict (inode); +/* +* The elevated s_deferred_reclaim_count keeps sb alive +* until we drop it +*/ +if (atomic_dec_and_test (&sb->s_deferred_reclaim_count)) +wake_up_var (&sb->s_deferred_reclaim_count); +cond_resched (); + } + } +} + +/* Maximum number of inode reclaim workers */ +#define MAX_RECLAIM_WORKERS 16 + +static void __init deferred_reclaim_init (void ) +{ +int workers =MAX_RECLAIM_WORKERS ; +int i; + +/* Scale down the number of workers for small systems */ +if (workers >num_possible_cpus ()) + workers =num_possible_cpus (); + + deferred_reclaim.workers = workers; + deferred_reclaim.work = kmalloc_objs(struct work_struct, workers); + deferred_reclaim.queues = + kmalloc_objs(struct deferred_reclaim_queue, workers); +if (!deferred_reclaim.work || !deferred_reclaim.queues) +panic ("Failed to allocate deferred inode reclaim queues" ); + + deferred_reclaim.wq =alloc_workqueue ("deferred-inode-reclaim" , +WQ_FREEZABLE |WQ_MEM_RECLAIM |WQ_UNBOUND , + workers); +if (!deferred_reclaim.wq) +panic ("Failed to allocate deferred inode reclaim workqueue" ); + +for (i =0 ; i < workers; i++) { +INIT_WORK (&deferred_reclaim.work[i], inode_reclaim_deferred); +spin_lock_init (&deferred_reclaim.queues[i].lock); +INIT_LIST_HEAD (&deferred_reclaim.queues[i].list); + } +} + +/* +* Mark inode for deferred reclaim. When slab reclaim path will be freeing the +* inode, it will not call evict() directly but rather offload it to a worker +* in a workqueue. This is useful when inode reclaim is expected to take a +* long time (e.g. because it needs to do IO) so that it doesn't block kswapd +* or because it needs allocating memory (which cannot be guaranteed from +* reclaim path). +* +* Note that we currently don't provide a way to unmark inode for deferred +* reclaim. It could be done but the benefit is unclear. The rationale is that +* to make inode hard to reclaim, relatively expensive things like writing data +* to the file have to be done and in such case the additional CPU overhead and +* latency of offloading the inode reclaim is negligible in the overall load. +*/ +void mark_inode_reclaim_deferred (struct inode *inode) +{ +if (inode_state_read_once (inode) &I_DEFER_RECLAIM ) +return ; + +spin_lock (&inode->i_lock); +inode_state_set (inode,I_DEFER_RECLAIM ); +spin_unlock (&inode->i_lock); +} +EXPORT_SYMBOL_GPL (mark_inode_reclaim_deferred); +static void __wait_on_freeing_inode (struct inode *inode,bool hash_locked,bool rcu_locked);static bool igrab_from_hash (struct inode *inode);
@@ -2658,6 +2861,9 @@ void __init inode_init(void) SLAB_ACCOUNT), init_once); +/* Deferred inode reclaim infrastructure */ +deferred_reclaim_init (); +/* Hash may have been set up in inode_init_early */ if (!hashdist)return ;
fs/super.c +2 -0
@@ -583,6 +583,8 @@ void deactivate_locked_super(struct super_block *s)struct file_system_type *fs = s->s_type;if (atomic_dec_and_test (&s->s_active)) {shrinker_free (s->s_shrink); +wait_var_event (&s->s_deferred_reclaim_count, + !atomic_read (&s->s_deferred_reclaim_count)); fs->kill_sb (s);kill_super_notify (s);
include/linux/fs.h +3 -0
@@ -741,6 +741,7 @@ enum inode_state_flags_enum {I_SYNC_QUEUED = (1U <<17 ),I_PINNING_NETFS_WB = (1U <<18 ),I_METADATA_WRITEBACK = (1U <<19 ), + I_DEFER_RECLAIM = (1U << 20), };#define I_DIRTY_INODE (I_DIRTY_SYNC | I_DIRTY_DATASYNC)
@@ -2218,6 +2219,8 @@ static inline void set_inode_metadata_writeback(struct inode *inode)spin_unlock (&inode->i_lock); } +void mark_inode_reclaim_deferred (struct inode *inode); +/* * returns the refcount on the inode. it can change arbitrarily. */
include/linux/fs/super_types.h +3 -0
@@ -232,6 +232,9 @@ struct super_block {struct shrinker *s_shrink;/* per-sb shrinker handle */ +/* Number of inodes queued for deferred reclaim */ +atomic_t s_deferred_reclaim_count; +/* Number of inodes with nlink == 0 but still referenced */ atomic_long_t s_remove_count;
include/trace/events/writeback.h +2 -1
@@ -28,7 +28,8 @@ {I_DONTCACHE, "I_DONTCACHE"}, \ {I_SYNC_QUEUED, "I_SYNC_QUEUED"}, \ {I_PINNING_NETFS_WB, "I_PINNING_NETFS_WB"}, \ - {I_LRU_ISOLATING, "I_LRU_ISOLATING"} \ + {I_LRU_ISOLATING, "I_LRU_ISOLATING"}, \ + {I_DEFER_RECLAIM, "I_DEFER_RECLAIM"} \ )/* enums need to be exported to user space */
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 4/7 fs: Defer reclaim of unlinked inodes
In reply to Jan KaraSometimes it can happen that reclaim of some slab object drops the last reference of an inode that is already unlinked. The frequently reported case of this is a reclaim of overlayfs inode dropping the last reference of an inode on upper filesystem. Because the inode is unlinked, evict() will delete it which generally requires writing to the filesystem and memory allocations - not good from kswapd or even direct reclaim. Defer inode reclaim of unlinked inodes to a workqueue when the last inode reference is dropped from a reclaim path.
--- fs/inode.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
fs/inode.c +17 -0
@@ -2214,6 +2214,23 @@ static void iput_final(struct inode *inode)inode_lru_list_del (inode);spin_unlock (&inode->i_lock); +/* +* Are we going to delete this inode while in reclaim path? Likely +* reclaim of some object (e.g. overlayfs inode) ended up dropping the +* last reference to an already deleted inode. Offload the eviction as +* it's likely to require memory allocations and block on IO. Since +* this eviction was triggered by a reclaim of unrelated object, the +* inode is already outliving the operation that originally deleted it +* (unlink, file close...). A slightly longer delay thus should not be +* observable by userspace. +*/ +if (current->flags &PF_MEMALLOC && !inode->i_nlink) { +LIST_HEAD (defer); + +list_add (&inode->i_lru, &defer); +queue_deferred_reclaim (sb,1 , &defer); +return ; + }evict (inode); }
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 5/7 fs: Add throttling to deferred inode reclaim
In reply to Jan KaraDeferring difficult inode reclaim from prune_icache_sb() to a workqueue removes the natural feedback loop of blocking tasks in direct reclaim until they make space for new allocations. This can result in the list of deferred inodes to grow beyond any bounds and possibly push the machine to a reclaim storm or OOM.
Add a throttling mechanism that makes tasks calling mark_inode_reclaim_deferred() wait on exit to userspace if the number of deferred inodes to reclaim for a superblock grows over limit. We measure average time it takes to reclaim inode on deferred list for a superblock and block tasks proportionally to that.
The mechanism to block tasks on exit to userspace is a bit simplistic but I don't expect more than one inode to be deferred per syscall and I expect throttling to be rare so it should be fine. If we wanted something more advanced, we could store the delay in task_struct and just wait for it when returning to userspace similarly as blkcg throttling does it but so far I consider the space in task_struct a waste for this purpose.
--- fs/inode.c | 128 ++++++++++++++++++++++++++++--- include/linux/fs/super_types.h | 2 + include/trace/events/writeback.h | 51 ++++++++++++ 3 files changed, 170 insertions(+), 11 deletions(-)
fs/inode.c +117 -11
@@ -23,6 +23,7 @@#include <linux/rw_hint.h> #include <linux/seq_file.h> #include <linux/debugfs.h> +#include <linux/task_work.h> #include <trace/events/writeback.h> #define CREATE_TRACE_POINTS #include <trace/events/timestamp.h>
@@ -1119,23 +1120,53 @@ static int get_inode_reclaim_batch(int *id, struct list_head *list)return count; } +static void inode_reclaim_update_stat (struct super_block *sb,unsigned int n, +u64 start) +{ +u64 delay; + +if (!sb) +return ; + + delay =div_u64 (ktime_get_ns () - start, n); +/* +* Smooth delay updates with exponential moving average. Updates can +* get lost if workers race but we don't really care. +*/ +WRITE_ONCE (sb->s_deferred_reclaim_delay, + (63 *READ_ONCE (sb->s_deferred_reclaim_delay) + delay) /64 ); + +trace_inode_reclaim_update_stat (sb, n, delay, +READ_ONCE (sb->s_deferred_reclaim_delay)); + +/* +* The elevated s_deferred_reclaim_count keeps sb alive until we drop +* it +*/ +if (!atomic_sub_return (n, &sb->s_deferred_reclaim_count)) +wake_up_var (&sb->s_deferred_reclaim_count); +}static void inode_reclaim_deferred (struct work_struct *work) {struct inode *inode;LIST_HEAD (inode_batch); +u64 start; +int id, count; +struct super_block *sb; +/* * We start with the list corresponding to the work but rolling a dice * would work as well */ -int id = work - deferred_reclaim.work; + id = work - deferred_reclaim.work;while (1 ) {if (!get_inode_reclaim_batch (&id, &inode_batch))break ; -while (!list_empty (&inode_batch)) { -struct super_block *sb; + sb =NULL ; +while (!list_empty (&inode_batch)) {/* * inode_batch list is private and I_FREEING flags * protect us from anybody else trying to remove the
@@ -1143,17 +1174,20 @@ static void inode_reclaim_deferred(struct work_struct *work)*/ inode = list_first_entry(&inode_batch, struct inode, i_lru); +if (inode->i_sb != sb) { +inode_reclaim_update_stat (sb, count, start); + + sb = inode->i_sb; + count =0 ; + start =ktime_get_ns (); + } + count++;list_del_init (&inode->i_lru); - sb = inode->i_sb;evict (inode); -/* -* The elevated s_deferred_reclaim_count keeps sb alive -* until we drop it -*/ -if (atomic_dec_and_test (&sb->s_deferred_reclaim_count)) -wake_up_var (&sb->s_deferred_reclaim_count);cond_resched (); } + +inode_reclaim_update_stat (sb, count, start); } }
@@ -1189,6 +1223,70 @@ static void __init deferred_reclaim_init(void) } } +struct deferred_reclaim_delay { +struct callback_head task_work; +u64 delay; +}; + +static void deferred_delay_work (struct callback_head *work) +{ +struct deferred_reclaim_delay *dd = + container_of(work, struct deferred_reclaim_delay, task_work); + +schedule_timeout_killable (nsecs_to_jiffies (dd->delay)); +} + +/* +* Size of deferred reclaim list from which we start throttling tasks creating +* inodes marked for deferred reclaim. +*/ +#define INODE_DEFERRED_RECLAIM_LIMIT 8192 + +static void throttle_inode_deferred_reclaim (struct inode *inode) +{ +struct super_block *sb = inode->i_sb; +unsigned int len; + +/* +* If inodes with deferred reclaim are accumulating too much, slow down +* tasks creating them. This doesn't provide any kind of guarantee on +* the length of the deferred list since lots of inodes with +* I_DEFER_RECLAIM can be already present in the inode cache and we +* have no control when they reach the deferred list. But if the +* pressure on the deferred list is sustained, the balance should +* eventually be established. +*/ + len =atomic_read (&sb->s_deferred_reclaim_count); +if (len >INODE_DEFERRED_RECLAIM_LIMIT ) { +u64 delay =READ_ONCE (sb->s_deferred_reclaim_delay); +struct deferred_reclaim_delay *dd; + +if (!delay) +return ; + + dd =kmalloc_obj (*dd,GFP_NOFS | __GFP_NORETRY); +/* +* If the allocation failed, don't bother. Throttling is +* approximate anyway and next time we'll throttle more. +*/ +if (!dd) +return ; + +/* +* Scale the delay based on how much we exceed the limit. Wait +* at most 4x as long as estimated time to reclaim the inode. +*/ + len =min (len,5 *INODE_DEFERRED_RECLAIM_LIMIT ); + delay =div_u64 (delay * (len -INODE_DEFERRED_RECLAIM_LIMIT ), +INODE_DEFERRED_RECLAIM_LIMIT ); +trace_mark_inode_reclaim_deferred_throttle (inode, len, delay); + + dd->delay = delay; +init_task_work (&dd->task_work, deferred_delay_work); +task_work_add (current, &dd->task_work,TWA_RESUME ); + } +} +/* * Mark inode for deferred reclaim. When slab reclaim path will be freeing the * inode, it will not call evict() directly but rather offload it to a worker
@@ -1205,12 +1303,20 @@ static void __init deferred_reclaim_init(void)*/ void mark_inode_reclaim_deferred (struct inode *inode) { +bool throttle = false; +if (inode_state_read_once (inode) &I_DEFER_RECLAIM )return ;spin_lock (&inode->i_lock); -inode_state_set (inode,I_DEFER_RECLAIM ); +if (!(inode_state_read (inode) &I_DEFER_RECLAIM )) { +inode_state_set (inode,I_DEFER_RECLAIM ); + throttle = true; + }spin_unlock (&inode->i_lock); + +if (throttle) +throttle_inode_deferred_reclaim (inode); }EXPORT_SYMBOL_GPL (mark_inode_reclaim_deferred);
include/linux/fs/super_types.h +2 -0
@@ -234,6 +234,8 @@ struct super_block {/* Number of inodes queued for deferred reclaim */ atomic_t s_deferred_reclaim_count; +/* Average time to reclaim one deferred inode */ +u64 s_deferred_reclaim_delay;/* Number of inodes with nlink == 0 but still referenced */ atomic_long_t s_remove_count;
include/trace/events/writeback.h +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 6/7 fs: Protect deferred inode reclaim from freezing
In reply to Jan KaraSometimes inode reclaim needs writing to the filesystem. One prominent case is when inode is unlinked and we are deleting it, another case is when XFS needs to trim its speculative block preallocation. When inode reclaim runs on frozen filesystem, this will block the deferred inode reclaim worker so eventually all deferred inode reclaim could be stalled in the system because all workers are waiting for some filesystem to thaw.
Avoid this problem by protecting running deferred inode reclaim from filesystem freezing (the SB_FREEZE_INTWRITE level) and by just skipping starting new deferred inode reclaim for inodes on frozen filesystem.
--- fs/inode.c | 89 ++++++++++++++++++++++++++++------ fs/super.c | 14 ++++++ include/linux/fs/super.h | 1 + include/linux/fs/super_types.h | 2 + 4 files changed, 90 insertions(+), 16 deletions(-)
fs/inode.c +73 -16
@@ -1022,20 +1022,11 @@ static int get_deferred_sb_id(struct super_block *sb)return hash_ptr (sb,32 ) % deferred_reclaim.workers; } -static void queue_deferred_reclaim (struct super_block *sb,int count, -struct list_head *list) +void wake_deferred_reclaim_workers (struct super_block *sb,int count) {int id =get_deferred_sb_id (sb); -struct deferred_reclaim_queue *queue = - &deferred_reclaim.queues[id];int i, wake_count; -spin_lock (&queue->lock); -list_splice_tail (list, &queue->list); -atomic_add (count, &sb->s_deferred_reclaim_count); -spin_unlock (&queue->lock); - -/* Queue works to process inodes we've added to the list */ wake_count =DIV_ROUND_UP (count,INODE_RECLAIM_BATCH_SIZE );if (wake_count > deferred_reclaim.workers) wake_count = deferred_reclaim.workers;
@@ -1046,6 +1037,28 @@ static void queue_deferred_reclaim(struct super_block *sb, int count, } } +static void queue_deferred_reclaim (struct super_block *sb,int count, +struct list_head *list) +{ +int id =get_deferred_sb_id (sb); +struct deferred_reclaim_queue *queue = + &deferred_reclaim.queues[id]; + +spin_lock (&queue->lock); +list_splice_tail (list, &queue->list); +atomic_add (count, &sb->s_deferred_reclaim_count); +spin_unlock (&queue->lock); + +/* +* Skip waking up workers for frozen filesystem. They'd skip the inodes +* anyway. +*/ +if (atomic_read (&sb->s_deferred_reclaim_running) <0 ) +return ; + +wake_deferred_reclaim_workers (sb, count); +} +/* * Walk the superblock inode LRU for freeable inodes and attempt to free them. * This is called from the superblock shrinker function with a number of inodes
@@ -1084,18 +1097,24 @@ static void __get_inode_reclaim_batch(int *start, int end,for (i = *start; i < end; i++) {struct deferred_reclaim_queue *queue = &deferred_reclaim.queues[i]; -struct list_head *entry, tmp; +struct inode *inode, *next;if (list_empty_careful (&queue->list))continue ;spin_lock (&queue->lock); -list_for_each (entry, &queue->list) { -if (*count >=INODE_RECLAIM_BATCH_SIZE ) +list_for_each_entry_safe (inode, next, &queue->list, i_lru) { +/* +* Skip inodes on frozen fs as we'd likely block. We +* always accept inodes from read-only fs as there +* eviction cannot require writing. +*/ +if (!sb_rdonly (inode->i_sb) && + inode->i_sb->s_writers.frozen !=SB_UNFROZEN ) +continue ; +list_move_tail (&inode->i_lru, list); +if (++(*count) >=INODE_RECLAIM_BATCH_SIZE )break ; - (*count)++; } -list_cut_before (&tmp, &queue->list, entry); -list_splice_tail (&tmp, list);spin_unlock (&queue->lock);if (*count >=INODE_RECLAIM_BATCH_SIZE )break ;
@@ -1147,6 +1166,35 @@ static void inode_reclaim_update_stat(struct super_block *sb, unsigned int n,wake_up_var (&sb->s_deferred_reclaim_count); } +/* Put inodes from sb back to reclaim queue */ +static void inode_reclaim_skip_inodes (struct list_head *list, +struct super_block *sb) +{ +struct inode *inode; +struct deferred_reclaim_queue *queue; +struct list_head tmp; + + queue = &deferred_reclaim.queues[get_deferred_sb_id (sb)]; +list_for_each_entry (inode, list, i_lru) +if (inode->i_sb != sb) +break ; +list_cut_before (&tmp, list, &inode->i_lru); +spin_lock (&queue->lock); +list_splice_tail (&tmp, &queue->list); +spin_unlock (&queue->lock); +} + +static bool sb_deferred_reclaim_get (struct super_block *sb) +{ +return atomic_inc_unless_negative (&sb->s_deferred_reclaim_running); +} + +static void sb_deferred_reclaim_put (struct super_block *sb) +{ +if (sb &&atomic_dec_and_test (&sb->s_deferred_reclaim_running)) +wake_up_var (&sb->s_deferred_reclaim_running); +} +static void inode_reclaim_deferred (struct work_struct *work) {struct inode *inode;
@@ -1176,10 +1224,18 @@ static void inode_reclaim_deferred(struct work_struct *work) i_lru);if (inode->i_sb != sb) {inode_reclaim_update_stat (sb, count, start); +sb_deferred_reclaim_put (sb); sb = inode->i_sb; count =0 ; start =ktime_get_ns (); +/* Protect us from filesystem freezing */ +if (!sb_deferred_reclaim_get (sb)) { +inode_reclaim_skip_inodes (&inode_batch, + sb); + sb =NULL ; +continue ; + } } count++;list_del_init (&inode->i_lru);
@@ -1188,6 +1244,7 @@ static void inode_reclaim_deferred(struct work_struct *work) }inode_reclaim_update_stat (sb, count, start); +sb_deferred_reclaim_put (sb); } }
fs/super.c +14 -0
@@ -2308,6 +2308,16 @@ int freeze_super(struct super_block *sb, enum freeze_holder who, const void *frereturn ret; } +/* +* Block deferred reclaim so that workers don't hang on frozen fs. We +* wait for outstanding deferred reclaim here. Because the filesystem +* is already partially frozen, workers will not start new deferred +* reclaim for this sb. +*/ +while (atomic_cmpxchg (&sb->s_deferred_reclaim_running,0 ,-1 )) +wait_var_event (&sb->s_deferred_reclaim_running, +atomic_read (&sb->s_deferred_reclaim_running) ==0 ); +/* Now wait for internal filesystem counter */ sb->s_writers.frozen =SB_FREEZE_FS ;sb_wait_write (sb,SB_FREEZE_FS );
@@ -2388,6 +2398,10 @@ static int thaw_super_locked(struct super_block *sb, enum freeze_holder who, sb->s_writers.freeze_owner =NULL ;wake_up_var (&sb->s_writers.frozen);sb_freeze_unlock (sb,SB_FREEZE_FS ); +/* Unblock deferred inode reclaim and wake workers */ +atomic_set (&sb->s_deferred_reclaim_running,0 ); +wake_deferred_reclaim_workers (sb, +atomic_read (&sb->s_deferred_reclaim_count)); out_deactivate:deactivate_locked_super (sb);return 0 ;
include/linux/fs/super.h +1 -0
@@ -235,6 +235,7 @@ int freeze_super(struct super_block *super, enum freeze_holder who,int thaw_super (struct super_block *super,enum freeze_holder who,const void *freeze_owner); +void wake_deferred_reclaim_workers (struct super_block *sb,int count);int sb_init_dio_done_wq (struct super_block *sb);struct file ;
include/linux/fs/super_types.h +2 -0
@@ -234,6 +234,8 @@ struct super_block {/* Number of inodes queued for deferred reclaim */ atomic_t s_deferred_reclaim_count; +/* Number of inodes currently undergoing direct reclaim */ +atomic_t s_deferred_reclaim_running;/* Average time to reclaim one deferred inode */ u64 s_deferred_reclaim_delay;
Signature · 1 line
2.51.0
Jan Kara
maintainerPATCH v3 7/7 ext4: Defer inode reclaim if it has preallocations
In reply to Jan KaraWhen we have to free preallocations during inode eviction, we need to load block bitmaps and run transaction to modify them. This takes time and also requires GFP_NOFAIL allocations. Mark inodes with preallocated blocks as needing offloading of inode reclaim to a workqueue so that we don't block reclaim for long and potentially deadlock MM subsystem.
--- fs/ext4/mballoc.c | 7 +++++++ 1 file changed, 7 insertions(+)
fs/ext4/mballoc.c +7 -0
@@ -3115,6 +3115,13 @@ ext4_mb_regular_allocator(struct ext4_allocation_context *ac)if (ac->ac_prefetch_nr)ext4_mb_prefetch_fini (sb, ac->ac_prefetch_grp, ac->ac_prefetch_nr); +/* +* Freeing preallocations requires loading bitmaps and running +* transactions. Defer inode reclaim to a workqueue. +*/ +if (!RB_EMPTY_ROOT (&EXT4_I (ac->ac_inode)->i_prealloc_node)) +mark_inode_reclaim_deferred (ac->ac_inode); +return err; }
Signature · 1 line
2.51.0
Jan Kara
maintainerHi!
FWIW Sashiko still found some stupid mistakes I've made so I'll have to respin this. However none of the issues Sashiko found are fundamental (just some memory leaks, wrong memory ordering, ...) so the overall design should stay so if someone has any comment to that, they will be still valid.
Honza
Quoted 76 lines · Jan Kara
Hello,
here is the third revision of my patches implementing deferred inode reclaim to deal with MM warnings due to
GFP_NOFAILallocations from reclaim paths. This happens because to reclaim some inodes, filesystems have to do IO including complex operations using journalling and forward progress of these depends on successful memory allocations (which is impossible to guarantee from reclaim context).Since the last revision the most notable changes include addition of automatic deferal of inode eviction from
iput()when inode link-count is 0 and we are in reclaim (PF_MEMALLOCflag set). This path gets triggered e.g. by overlayfs whose inodes hold dentry references on upper filesystem and thus slab reclaim of overlayfs inodes can trigger the lastdput()(and thusiput()) of already unlinked files. Since inode deletion is practically guaranteed to require IO (and thus possibly e.g. loading allocation information into memory) I've decided to unconditionally defer reclaim in this case instead of marking inodes in each filesystem.Another bigger change in this series is that we now protect deferred inode reclaim from filesystem freezing. Otherwise inode eviction could block the reclaim worker on frozen filesystem which could stall deferred inode reclaim on the whole system. Deferred reclaim workers now skip reclaiming inodes on frozen filesystem and we block filesystem freezing while the worker is evicting current batch of its inodes.
The last bigger change is that
mark_inode_reclaim_deferred()does not block when the queue of deferred inodes grows over limit anymore. Instead it queues task work to delay the task appropriately when returning to userspace. This solves possible lockups when tasks would get throttled when callingmark_inode_reclaim_deferred()with some locks or transaction started.The patches have survived full fstests run on ext4 when marking all loaded ext4 inodes for deferred reclaim to stress the deferred reclaim paths. I have also tested (when marking all inodes for deferred reclaim) the patches by creating a memcg with 128m memory limit and then scanning from 10 processes 1m inodes in total which heavily exercises deferred inode reclaim paths. This also verified efficiency of throttling of marking of inodes for deferred reclaim. Basically we cannot guarantee any particular limit on the number of inodes queued for deferred reclaim as that is bound only by their number in memory and memory pressure. But as the number of queued inodes grows, the tasks creating them get slowed down so eventually some equilibrium is hit. In my VM this was at ~10k inodes (only slightly above the 8k limit when throttling kicks in) but this all very much depends on the reclaim pressure, speed of deferred reclaim etc. so I don't think this is some representative number.
The first patch is a pure fix for a problem I've hit when running fstests. The second patch deals with lazy timestamp updates which I've decided to handle better directly inside the writeback infrastructure instead of deferring reclaim. The remaining patches implement the inode reclaim deferal. Ext4 use of deferred inode reclaim is there mostly as a demonstration. Other filesystems need to determine which inodes need deferred reclaim and mark them as such which generally requires good internal knowledge of the filesystem. I'm hoping that once the infrastructure is there, fs developers prompted by MM warnings will start using it :). I know for a fact that besides ext4 e.g. gfs2 wants to use this infrastructure.
Honza
Changes since v2:
* Some cleanups and simplifications from Andreas Grunbacher
* Fixed couple of bugs pointed out by Sashiko
* Added support for auto-deferring eviction fromiput()when inode has 0 link count and we are in reclaim path
* Added protection of deferred reclaim from filesystem freezing
* Throttling now doesn't happen directly inmark_inode_reclaim_deferred()but on exit to userspaceChanges since v1:
* Multiple workers to reclaim deferred inodes, related to that I've moved lists of deferred inodes from per-sb to global ones
* Added missed lazytime sync fixPrevious versions:
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Christian Brauner
maintainerQuoted 6 lines · Jan Kara
Hi!
FWIW Sashiko still found some stupid mistakes I've made so I'll have to respin this. However none of the issues Sashiko found are fundamental (just some memory leaks, wrong memory ordering, ...) so the overall design should stay so if someone has any comment to that, they will be still valid.
I do like the direction of this design!
Quoted 26 lines · Jan Kara
Hello,
here is the third revision of my patches implementing deferred inode reclaim to deal with MM warnings due to
GFP_NOFAILallocations from reclaim paths. This happens because to reclaim some inodes, filesystems have to do IO including complex operations using journalling and forward progress of these depends on successful memory allocations (which is impossible to guarantee from reclaim context).Since the last revision the most notable changes include addition of automatic deferal of inode eviction from
iput()when inode link-count is 0 and we are in reclaim (PF_MEMALLOCflag set). This path gets triggered e.g. by overlayfs whose inodes hold dentry references on upper filesystem and thus slab reclaim of overlayfs inodes can trigger the lastdput()(and thusiput()) of already unlinked files. Since inode deletion is practically guaranteed to require IO (and thus possibly e.g. loading allocation information into memory) I've decided to unconditionally defer reclaim in this case instead of marking inodes in each filesystem.Another bigger change in this series is that we now protect deferred inode reclaim from filesystem freezing. Otherwise inode eviction could block the reclaim worker on frozen filesystem which could stall deferred inode reclaim on the whole system. Deferred reclaim workers now skip reclaiming inodes on frozen filesystem and we block filesystem freezing while the worker is evicting current batch of its inodes.
Quoted 5 lines
The last bigger change is that
mark_inode_reclaim_deferred()does not block when the queue of deferred inodes grows over limit anymore. Instead it queues task work to delay the task appropriately when returning to userspace. This solves possible lockups when tasks would get throttled when callingmark_inode_reclaim_deferred()with some locks or transaction started.
Hmmm. This re-implements the XFS inodegc infrastructure at the VFs level, but without all the internal XFS knowledge of when background inodegc processing is required. It looks rather complex compared to the XFS inodegc code....
Whilst I agree that the VFS should be able to do this for filesystems like ext4, we really don't want to be deferring inodegc processing twice. Hence for XFS we really want the VFS to call evict() immediately for all inodes so that the filesystem can correctly decide if deferred processing is necessary or not...
Quoted 5 lines
The patches have survived full fstests run on ext4 when marking all loaded ext4 inodes for deferred reclaim to stress the deferred reclaim paths. I have also tested (when marking all inodes for deferred reclaim) the patches by creating a memcg with 128m memory limit and then scanning from 10 processes 1m inodes in total which heavily exercises deferred inode reclaim paths. This also verified
Quoted 8 lines
efficiency of throttling of marking of inodes for deferred reclaim. Basically we cannot guarantee any particular limit on the number of inodes queued for deferred reclaim as that is bound only by their number in memory and memory pressure. But as the number of queued inodes grows, the tasks creating them get slowed down so eventually some equilibrium is hit. In my VM this was at ~10k inodes (only slightly above the 8k limit when throttling kicks in) but this all very much depends on the reclaim pressure, speed of deferred reclaim etc. so I don't think this is some representative number.
XFS has a "almost-hard" queue limit is 256 queued inodes per CPU. Anything larger than that sees reduced performance under sustained inode cache pressure and latency problems when eviction processing of individual inodes takes a long time (e.g. deferred unlink of a set of inodes with millions of extents each).
In normal operations, however, the low queue limit means triggers the scheduler to move the reclaiming thread to another CPU to allow the queued cleanup work to run on the same CPU it was last touched on. Performance testing showed that moving the reclaiming task was much, much better than pushing the reclaiming work to other CPUs whilst blocking on the throttle. Deep queues (like many thousands of inodes) caused other problems with ongoing (non-reclaim) operations then contending with resources with queued background inodegc.
i.e. for XFS it has proven better to have short queues, short backoffs and schedule/process small batches concurrently than it is to have deep queues, spreading the work out many kworkers and having "end-of-queueing" throttling behaviour...
Quoted 10 lines
The first patch is a pure fix for a problem I've hit when running fstests. The second patch deals with lazy timestamp updates which I've decided to handle better directly inside the writeback infrastructure instead of deferring reclaim. The remaining patches implement the inode reclaim deferal. Ext4 use of deferred inode reclaim is there mostly as a demonstration. Other filesystems need to determine which inodes need deferred reclaim and mark them as such which generally requires good internal knowledge of the filesystem. I'm hoping that once the infrastructure is there, fs developers prompted by MM warnings will start using it :). I know for a fact that besides ext4 e.g. gfs2 wants to use this infrastructure.
But XFS doesn't, so it needs to be optional. :)
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jan Kara
maintainerQuoted 31 lines · Dave Chinner
Quoted 24 lines · Jan Kara
here is the third revision of my patches implementing deferred inode reclaim to deal with MM warnings due to
GFP_NOFAILallocations from reclaim paths. This happens because to reclaim some inodes, filesystems have to do IO including complex operations using journalling and forward progress of these depends on successful memory allocations (which is impossible to guarantee from reclaim context).Since the last revision the most notable changes include addition of automatic deferal of inode eviction from
iput()when inode link-count is 0 and we are in reclaim (PF_MEMALLOCflag set). This path gets triggered e.g. by overlayfs whose inodes hold dentry references on upper filesystem and thus slab reclaim of overlayfs inodes can trigger the lastdput()(and thusiput()) of already unlinked files. Since inode deletion is practically guaranteed to require IO (and thus possibly e.g. loading allocation information into memory) I've decided to unconditionally defer reclaim in this case instead of marking inodes in each filesystem.Another bigger change in this series is that we now protect deferred inode reclaim from filesystem freezing. Otherwise inode eviction could block the reclaim worker on frozen filesystem which could stall deferred inode reclaim on the whole system. Deferred reclaim workers now skip reclaiming inodes on frozen filesystem and we block filesystem freezing while the worker is evicting current batch of its inodes.
Quoted 5 lines
The last bigger change is that
mark_inode_reclaim_deferred()does not block when the queue of deferred inodes grows over limit anymore. Instead it queues task work to delay the task appropriately when returning to userspace. This solves possible lockups when tasks would get throttled when callingmark_inode_reclaim_deferred()with some locks or transaction started.
Quoted 4 lines
Hmmm. This re-implements the XFS inodegc infrastructure at the VFs level, but without all the internal XFS knowledge of when background inodegc processing is required. It looks rather complex compared to the XFS inodegc code....
Yes, I took some inspiration from it. One big difference though is that deferred inode reclaim will be applied only to inode evictions done from SLAB reclaim and only for inodes explicitely marked for deferred reclaim by the filesystem. Just there's this exception that currently VFS would auto-offload (i.e. without explicit marking by fs) in the case SLAB reclaim happens to drop the last inode reference of unlinked inode. So in that case VFS could defer reclaim even for XFS inode but that is mostly performance irrelevant cornercase (in vast majority of cases unlink or file close does the last iput() of unlinked inodes and in that case we never offload in VFS). So I'm not sure we care about avoiding the extra offload for XFS for that case.
Quoted 5 lines
Whilst I agree that the VFS should be able to do this for filesystems like ext4, we really don't want to be deferring inodegc processing twice. Hence for XFS we really want the VFS to call
evict()immediately for all inodes so that the filesystem can correctly decide if deferred processing is necessary or not...
Yep, that should mostly happen. See above.
Quoted 29 lines
Quoted 5 lines
The patches have survived full fstests run on ext4 when marking all loaded ext4 inodes for deferred reclaim to stress the deferred reclaim paths. I have also tested (when marking all inodes for deferred reclaim) the patches by creating a memcg with 128m memory limit and then scanning from 10 processes 1m inodes in total which heavily exercises deferred inode reclaim paths. This also verified
Quoted 8 lines
efficiency of throttling of marking of inodes for deferred reclaim. Basically we cannot guarantee any particular limit on the number of inodes queued for deferred reclaim as that is bound only by their number in memory and memory pressure. But as the number of queued inodes grows, the tasks creating them get slowed down so eventually some equilibrium is hit. In my VM this was at ~10k inodes (only slightly above the 8k limit when throttling kicks in) but this all very much depends on the reclaim pressure, speed of deferred reclaim etc. so I don't think this is some representative number.
XFS has a "almost-hard" queue limit is 256 queued inodes per CPU. Anything larger than that sees reduced performance under sustained inode cache pressure and latency problems when eviction processing of individual inodes takes a long time (e.g. deferred unlink of a set of inodes with millions of extents each).
In normal operations, however, the low queue limit means triggers the scheduler to move the reclaiming thread to another CPU to allow the queued cleanup work to run on the same CPU it was last touched on. Performance testing showed that moving the reclaiming task was much, much better than pushing the reclaiming work to other CPUs whilst blocking on the throttle. Deep queues (like many thousands of inodes) caused other problems with ongoing (non-reclaim) operations then contending with resources with queued background inodegc.
Quoted 4 lines
i.e. for XFS it has proven better to have short queues, short backoffs and schedule/process small batches concurrently than it is to have deep queues, spreading the work out many kworkers and having "end-of-queueing" throttling behaviour...
Yes, I was considering XFS inode reclaim design as well. The big difference to XFS case which offloads everything is that we expect to only have to defer inode reclaim if something expensive (like allocating blocks) happened to the inode. At least all usecases I have in mind are like that. Also except of rare cornercase races the reclaim of unlinked inodes will not be handled by deferred reclaim workers but directly inline in iput(). That is why I didn't bother with per-cpu thing as I believe the performance of the offloading mechanism itself doesn't matter as much at that point. But I could be wrong...
Quoted 11 lines
Quoted 10 lines
The first patch is a pure fix for a problem I've hit when running fstests. The second patch deals with lazy timestamp updates which I've decided to handle better directly inside the writeback infrastructure instead of deferring reclaim. The remaining patches implement the inode reclaim deferal. Ext4 use of deferred inode reclaim is there mostly as a demonstration. Other filesystems need to determine which inodes need deferred reclaim and mark them as such which generally requires good internal knowledge of the filesystem. I'm hoping that once the infrastructure is there, fs developers prompted by MM warnings will start using it :). I know for a fact that besides ext4 e.g. gfs2 wants to use this infrastructure.
Quoted 1 line
But XFS doesn't, so it needs to be optional. :)
If you still want the rare cases where offload could happen for XFS optional after above explanations, I can do a filesystem_type flag for that.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR