Skip to content

fs: rework inode reference counting

Josef Bacik linux-btrfs linux-ext4 linux-fsdevel linux-xfs lkml syzbot 2 versions – Applied

Status from this message: Applied to the vfs-6.18.inode.refcount.preliminaries branch of the vfs/vfs.git tree.

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest54 patches 46 repliesApplied
  2. v150 patches 33 repliesApplied

Changes in v2 from the cover letter

- Fixed all the things that Christian pointed out.
- Re-ordered some of the patches to the front in case Christian wants to take those first.
- Added a new patch for reading the current i_count and propagated that everywhere.
- Fixed the cifs build breakage.
- Removed I_REFERENCED since it's no longer needed.
- Remove I_LRU_ISOLATING since it's no longer needed.
- Reworked the drop_nlink/clear_nlink part to simply remove the inode from the LRU in the unlink path, and made this its own patch to make the behavior change clear.
- NOTE: I'm re-running fstests on this now, there was a slight issue with removing the drop_nlink/clear_nlink patch and so I had to add the unlink/rmdir patch to resolve it. I assume everything will be fine but just an FYI.
- NOTE #2: I reordered stuff, and I did a rebase and rebuild at every step, but I noticed this morning I still missed an odd rebase artifact, so by all means validate I didn't make any silly mistakes on the in-between patches.

Patches in v2 Applied

#Subject and filesSizeRepliesApplied
1/54fs: make the i_state flags an enuminclude/linux/fs.h+117 -1120 replies
2/54fs: add an icount_read helperarch/powerpc/platforms/cell/spufs/file.c fs/btrfs/inode.c fs/ceph/mds_client.c +13 more+25 -202 replies
3/54fs: rework iput logicfs/inode.c+14 -99 replies
4/54fs: add an i_obj_count refcount to the inodefs/inode.c include/linux/fs.h+32 -00 replies
5/54fs: hold an i_obj_count reference in wait_sb_inodesfs/fs-writeback.c+2 -00 replies
6/54fs: hold an i_obj_count reference for the i_wb_listfs/fs-writeback.c+14 -00 replies
7/54fs: hold an i_obj_count reference for the i_io_listfs/fs-writeback.c+21 -40 replies
8/54fs: hold an i_obj_count reference in writeback_sb_inodesfs/fs-writeback.c+5 -40 replies
9/54fs: hold an i_obj_count reference while on the hashtablefs/inode.c+9 -00 replies
10/54fs: hold an i_obj_count reference while on the LRU listfs/inode.c+17 -30 replies
11/54fs: hold an i_obj_count reference while on the sb inode listfs/inode.c+2 -00 replies
12/54fs: stop accessing ->i_count directly in f2fs and gfs2fs/f2fs/super.c fs/gfs2/ops_fstype.c+3 -30 replies
13/54fs: hold an i_obj_count when we have an i_count referencefs/btrfs/inode.c fs/fs-writeback.c fs/inode.c +1 more+13 -220 replies
14/54fs: add an I_LRU flag to the inodefs/inode.c include/linux/fs.h include/trace/events/writeback.h+16 -20 replies
15/54fs: maintain a list of pinned inodesfs/fs-writeback.c fs/inode.c fs/internal.h +3 more+122 -83 replies
16/54fs: delete the inode from the LRU list on lookupfs/inode.c+3 -03 replies
17/54fs: remove the inode from the LRU list on unlink/rmdirfs/namei.c+25 -58 replies
18/54fs: change evict_inodes to use iput instead of evict directlyfs/inode.c+11 -151 reply
19/54fs: hold a full ref while the inode is on a LRUfs/inode.c+40 -471 reply
20/54fs: disallow 0 reference count inodesfs/inode.c+41 -202 replies
21/54fs: make evict_inodes add to the dispose list under the i_lockfs/inode.c+1 -10 replies
22/54fs: convert i_count to refcount_tfs/btrfs/inode.c fs/inode.c include/linux/fs.h+9 -81 reply
23/54fs: use refcount_inc_not_zero in igrabfs/inode.c include/linux/fs.h+43 -132 replies
24/54fs: use inode_tryget in find_inode*fs/inode.c+9 -100 replies
25/54fs: update find_inode_*rcu to check the i_count countfs/inode.c+3 -30 replies
26/54fs: use igrab in insert_inode_lockedfs/inode.c+3 -51 reply
27/54fs: remove I_WILL_FREE|I_FREEING check from __inode_add_lrufs/inode.c+0 -20 replies
28/54fs: remove I_WILL_FREE|I_FREEING check in inode_pin_lru_isolatingfs/inode.c+1 -10 replies
29/54fs: use inode_tryget in evict_inodesfs/inode.c+6 -20 replies
30/54fs: change evict_dentries_for_decrypted_inodes to use refcountfs/crypto/keyring.c+5 -23 replies
31/54block: use igrab in sync_bdevsblock/bdev.c+5 -30 replies
32/54bcachefs: use the refcount instead of I_WILL_FREE|I_FREEINGfs/bcachefs/fs.c+1 -20 replies
33/54btrfs: don't check I_WILL_FREE|I_FREEINGfs/btrfs/inode.c+1 -10 replies
34/54fs: use igrab in drop_pagecache_sbfs/drop_caches.c+4 -70 replies
35/54fs: stop checking I_FREEING in d_find_alias_rcufs/dcache.c+2 -20 replies
36/54ext4: stop checking I_WILL_FREE|IFREEING in ext4_check_map_extents_envfs/ext4/inode.c+1 -10 replies
37/54fs: remove I_WILL_FREE|I_FREEING from fs-writeback.cfs/fs-writeback.c+26 -210 replies
38/54gfs2: remove I_WILL_FREE|I_FREEING usagefs/gfs2/ops_fstype.c+13 -40 replies
39/54fs: remove I_WILL_FREE|I_FREEING check from dquot.cfs/quota/dquot.c+4 -21 reply
40/54notify: remove I_WILL_FREE|I_FREEING checks in fsnotify_unmount_inodesfs/notify/fsnotify.c+4 -220 replies
41/54xfs: remove I_FREEING checkfs/xfs/xfs_bmap_util.c+1 -10 replies
42/54landlock: remove I_FREEING|I_WILL_FREE checksecurity/landlock/fs.c+4 -180 replies
43/54fs: change inode_is_dirtytime_only to use refcountinclude/linux/fs.h+7 -72 replies
44/54btrfs: remove references to I_FREEINGfs/btrfs/inode.c+1 -20 replies
45/54ext4: remove reference to I_FREEING in inode.cfs/ext4/inode.c+3 -30 replies
46/54ext4: remove reference to I_FREEING in orphan.cfs/ext4/orphan.c+4 -20 replies
47/54pnfs: use i_count refcount to determine if the inode is going awayfs/nfs/pnfs.c+1 -10 replies
48/54fs: remove some spurious I_FREEING references in inode.cfs/inode.c+7 -71 reply
49/54xfs: remove reference to I_FREEING|I_WILL_FREEfs/xfs/scrub/common.c+1 -20 replies
50/54ocfs2: do not set I_WILL_FREEfs/ocfs2/inode.c+0 -40 replies
51/54fs: remove I_FREEING|I_WILL_FREEfs/inode.c include/linux/fs.h include/trace/events/writeback.h+25 -411 reply
52/54fs: remove I_REFERENCEDfs/inode.c include/linux/fs.h include/trace/events/writeback.h+17 -421 reply
53/54fs: remove I_LRU_ISOLATING flagfs/inode.c include/linux/fs.h include/trace/events/writeback.h+17 -692 replies
54/54fs: add documentation explaining the reference count rules for inodesDocumentation/filesystems/vfs.rst+86 -00 replies

Reviews trailers per patch

3 of 54 patches have a Reviewed-by or Acked-by.

PatchChristian BraunerEric BiggersJan Kara
1/54R-b
2/54
3/54R-b
4/54
5/54
6/54
7/54
8/54
9/54
10/54
11/54
12/54
13/54
14/54
15/54
16/54
17/54
18/54
19/54
20/54
21/54
22/54
23/54
24/54
25/54
26/54
27/54
28/54
29/54
30/54R-b
31/54
32/54
33/54
34/54
35/54
36/54
37/54
38/54
39/54
40/54
41/54
42/54
43/54
44/54
45/54
46/54
47/54
48/54
49/54
50/54
51/54
52/54
53/54
54/54

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v2 thread

Cover letter [PATCH v2 00/54] fs: rework inode reference counting

v1: https://lore.kernel.org/linux-fsdevel/[email protected]/

v1->v2:
- Fixed all the things that Christian pointed out.
- Re-ordered some of the patches to the front in case Christian wants to take those first.
- Added a new patch for reading the current i_count and propagated that everywhere.
- Fixed the cifs build breakage.
- Removed I_REFERENCED since it's no longer needed.
- Remove I_LRU_ISOLATING since it's no longer needed.
- Reworked the drop_nlink/clear_nlink part to simply remove the inode from the LRU in the unlink path, and made this its own patch to make the behavior change clear.
- NOTE: I'm re-running fstests on this now, there was a slight issue with removing the drop_nlink/clear_nlink patch and so I had to add the unlink/rmdir patch to resolve it. I assume everything will be fine but just an FYI.
- NOTE #2: I reordered stuff, and I did a rebase and rebuild at every step, but I noticed this morning I still missed an odd rebase artifact, so by all means validate I didn't make any silly mistakes on the in-between patches.

--- Original email ---

Hello,

This series is the first part of a larger body of work geared towards solving a variety of scalability issues in the VFS.

We have historically had a variety of foot-guns related to inode freeing. We have I_WILL_FREE and I_FREEING flags that indicated when the inode was in the different stages of being reclaimed. This lead to confusion, and bugs in cases where one was checked but the other wasn't. Additionally, it's frankly confusing to have both of these flags and to deal with them in practice.

However, this exists because we have an odd behavior with inodes, we allow them to have a 0 reference count and still be usable. This again is a pretty unfun footgun, because generally speaking we want reference counts to be meaningful.

The problem with the way we reference inodes is the final iput(). The majority of file systems do their final truncate of a unlinked inode in their ->evict_inode() callback, which happens when the inode is actually being evicted. This can be a long process for large inodes, and thus isn't safe to happen in a variety of contexts. Btrfs, for example, has an entire delayed iput infrastructure to make sure that we do not do the final iput() in a dangerous context. We cannot expand the use of this reference count to all the places the inode is used, because there are cases where we would need to iput() in an IRQ context (end folio writeback) or other unsafe context, which is not allowed.

To that end, resolve this by introducing a new i_obj_count reference count. This will be used to control when we can actually free the inode. We then can use this reference count in all the places where we may reference the inode. This removes another huge footgun, having ways to access the inode itself without having an actual reference to it. The writeback code is one of the main places where we see this. Inodes end up on all sorts of lists here without a proper reference count. This allows us to protect the inode from being freed by giving this an other code mechanisms to protect their access to the inode.

With this we can separate the concept of the inode being usable, and the inode being freed. The next part of the patch series is to stop allowing for inodes to have an i_count of 0 and still be viable. This comes with some warts. The biggest wart is now if we choose to cache inodes in the LRU list we have to remove the inode from the LRU list if we access it once it's on the LRU list. This will result in more contention on the lru list lock, but in practice we rarely have inodes that do not have a dentry, and if we do that inode is not long for this world.

With not allowing inodes to hit a refcount of 0, we can take advantage of that common pattern of using refcount_inc_not_zero() in all of the lockless places where we do inode lookup in cache. From there we can change all the users who check I_WILL_FREE or I_FREEING to simply check the i_count. If it is 0 then they aren't allowed to do their work, othrwise they can proceed as normal.

With all of that in place we can finally remove these two flags.

This is a large series, but it is mostly mechanical. I've kept the patches very small, to make it easy to review and logic about each change. I have run this through fstests for btrfs and ext4, xfs is currently going. I wanted to get this out for review to make sure this big design changes are reasonable to everybody.

The series is based on vfs/vfs.all branch, which is based on 6.9-rc1. Thanks,

Josef

Josef Bacik (54):
fs: make the i_state flags an enum
fs: add an icount_read helper
fs: rework iput logic
fs: add an i_obj_count refcount to the inode
fs: hold an i_obj_count reference in wait_sb_inodes
fs: hold an i_obj_count reference for the i_wb_list
fs: hold an i_obj_count reference for the i_io_list
fs: hold an i_obj_count reference in writeback_sb_inodes
fs: hold an i_obj_count reference while on the hashtable
fs: hold an i_obj_count reference while on the LRU list
fs: hold an i_obj_count reference while on the sb inode list
fs: stop accessing ->i_count directly in f2fs and gfs2
fs: hold an i_obj_count when we have an i_count reference
fs: add an I_LRU flag to the inode
fs: maintain a list of pinned inodes
fs: delete the inode from the LRU list on lookup
fs: remove the inode from the LRU list on unlink/rmdir
fs: change evict_inodes to use iput instead of evict directly
fs: hold a full ref while the inode is on a LRU
fs: disallow 0 reference count inodes
fs: make evict_inodes add to the dispose list under the i_lock
fs: convert i_count to refcount_t
fs: use refcount_inc_not_zero in igrab
fs: use inode_tryget in find_inode*
fs: update find_inode_*rcu to check the i_count count
fs: use igrab in insert_inode_locked
fs: remove I_WILL_FREE|I_FREEING check from __inode_add_lru
fs: remove I_WILL_FREE|I_FREEING check in inode_pin_lru_isolating
fs: use inode_tryget in evict_inodes
fs: change evict_dentries_for_decrypted_inodes to use refcount
block: use igrab in sync_bdevs
bcachefs: use the refcount instead of I_WILL_FREE|I_FREEING
btrfs: don't check I_WILL_FREE|I_FREEING
fs: use igrab in drop_pagecache_sb
fs: stop checking I_FREEING in d_find_alias_rcu
ext4: stop checking I_WILL_FREE|IFREEING in ext4_check_map_extents_env
fs: remove I_WILL_FREE|I_FREEING from fs-writeback.c
gfs2: remove I_WILL_FREE|I_FREEING usage
fs: remove I_WILL_FREE|I_FREEING check from dquot.c
notify: remove I_WILL_FREE|I_FREEING checks in fsnotify_unmount_inodes
xfs: remove I_FREEING check
landlock: remove I_FREEING|I_WILL_FREE check
fs: change inode_is_dirtytime_only to use refcount
btrfs: remove references to I_FREEING
ext4: remove reference to I_FREEING in inode.c
ext4: remove reference to I_FREEING in orphan.c
pnfs: use i_count refcount to determine if the inode is going away
fs: remove some spurious I_FREEING references in inode.c
xfs: remove reference to I_FREEING|I_WILL_FREE
ocfs2: do not set I_WILL_FREE
fs: remove I_FREEING|I_WILL_FREE
fs: remove I_REFERENCED
fs: remove I_LRU_ISOLATING flag
fs: add documentation explaining the reference count rules for inodes

 Documentation/filesystems/vfs.rst        |  86 +++++
 arch/powerpc/platforms/cell/spufs/file.c |   2 +-
 block/bdev.c                             |   8 +-
 fs/bcachefs/fs.c                         |   3 +-
 fs/btrfs/inode.c                         |  11 +-
 fs/ceph/mds_client.c                     |   2 +-
 fs/crypto/keyring.c                      |   7 +-
 fs/dcache.c                              |   4 +-
 fs/drop_caches.c                         |  11 +-
 fs/ext4/ialloc.c                         |   4 +-
 fs/ext4/inode.c                          |   8 +-
 fs/ext4/orphan.c                         |   6 +-
 fs/f2fs/super.c                          |   4 +-
 fs/fs-writeback.c                        | 105 ++++--
 fs/gfs2/ops_fstype.c                     |  17 +-
 fs/hpfs/inode.c                          |   2 +-
 fs/inode.c                               | 422 ++++++++++++++---------
 fs/internal.h                            |   1 +
 fs/namei.c                               |  30 +-
 fs/nfs/inode.c                           |   4 +-
 fs/nfs/pnfs.c                            |   2 +-
 fs/notify/fsnotify.c                     |  26 +-
 fs/ocfs2/inode.c                         |   4 -
 fs/quota/dquot.c                         |   6 +-
 fs/smb/client/inode.c                    |   2 +-
 fs/super.c                               |   3 +
 fs/ubifs/super.c                         |   2 +-
 fs/xfs/scrub/common.c                    |   3 +-
 fs/xfs/xfs_bmap_util.c                   |   2 +-
 fs/xfs/xfs_inode.c                       |   2 +-
 fs/xfs/xfs_trace.h                       |   2 +-
 include/linux/fs.h                       | 285 ++++++++-------
 include/trace/events/filelock.h          |   2 +-
 include/trace/events/writeback.h         |   6 +-
 security/landlock/fs.c                   |  22 +-
 35 files changed, 684 insertions(+), 422 deletions(-)
Signature · 1 line
2.49.0