ceph: fix deadlock bugs by making iput() calls asynchronous

fs/ceph/inode.c +19 -0

+/**
+ * Queue an asynchronous iput() call in a worker thread.  Use this
+ * instead of iput() in contexts where evicting the inode is unsafe.
+ * For example, inode eviction may cause deadlocks in
+ * inode_wait_for_writeback() (when called from within writeback) or
+ * in netfs_wait_for_outstanding_io() (when called from within the
+ * Ceph messenger).
+ *
+ * @n: how many references to put
+ */
+void ceph_iput_n_async(struct inode *inode, int n)
+{
+       if (unlikely(!inode))
+               return;
+
+       if (likely(atomic_sub_return(n, &inode->i_count) > 0))
+               /* somebody else is holding another reference -
+                * nothing left to do for us
+                */

fs/ceph/inode.c +13 -0

+       /* simply queue a ceph_inode_work() without setting
+        * i_work_mask bit; other than putting the reference, there is
+        * nothing to do
+        */
+       WARN_ON_ONCE(!queue_work(ceph_inode_to_fs_client(inode)->inode_wq,
+                                &ceph_inode(inode)->i_work));
+
+       /* note: queue_work() cannot fail; it i_work were already
+        * queued, then it would be holding another reference, but no
+        * such reference exists
+        */
+}
+

On Wed, Sep 17, 2025 at 2:44 PM Max Kellermann <[email protected]> wrote:

I don't know about ceph internals, so no comment on that front.

Quoted 19 lines

fs/ceph/inode.c +19 -0

+/**
+ * Queue an asynchronous iput() call in a worker thread.  Use this
+ * instead of iput() in contexts where evicting the inode is unsafe.
+ * For example, inode eviction may cause deadlocks in
+ * inode_wait_for_writeback() (when called from within writeback) or
+ * in netfs_wait_for_outstanding_io() (when called from within the
+ * Ceph messenger).
+ *
+ * @n: how many references to put
+ */
+void ceph_iput_n_async(struct inode *inode, int n)
+{
+       if (unlikely(!inode))
+               return;
+
+       if (likely(atomic_sub_return(n, &inode->i_count) > 0))
+               /* somebody else is holding another reference -
+                * nothing left to do for us
+                */

fs/ceph/inode.c void ceph_queue_inode_work(struct inode *inode, int work_bit) · line 2222

Quoted 10 lines

fs/ceph/inode.c +10 -0

+               return;
+
+       doutc(ceph_inode_to_fs_client(inode)->client, "%p %llx.%llx\n", inode, ceph_vinop(inode));
+
+       /* the reference counter is now 0, i.e. nobody else is holding
+        * a reference to this inode; restore it to 1 and donate it to
+        * ceph_inode_work() which will call iput() at the end
+        */
+       atomic_set(&inode->i_count, 1);
+

That loop over iput() indeed asks for a variant which grabs an explicit count to subtract.

However, you cannot legally transition to 0 without ->i_lock held. By API contract the ->drop_inode routine needs to be called when you get here and other CPUs are prevented from refing the inode.

While it is true nobody *refs* the inode, it is still hanging out on the superblock list where it can get picked up by forced unmount and on the inode hash where it can get picked up by lookup. With a refcount of 0, ->i_lock not held and no flags added, from their POV this is a legally cached inode they can do whatever they want with.

So that force setting of refcount to 1 might be a use-after-free if this raced against another iput or it might be losing a reference picked up by someone else.

If you got the idea to bring back one frem from iput() in the stock kernel:

        if (atomic_dec_and_lock(&inode->i_count, &inode->i_lock)) {
                if (inode->i_nlink && (inode->i_state & I_DIRTY_TIME)) {
                        atomic_inc(&inode->i_count);

Note this guy still makes sure to take the lock first. As a side note this weird deref to 0 + ref back to 1 business is going away [1].

I don't know what's the handy Linux way to sub an arbitrary amount as long as the target is not x, I guess worst case one can just write a cmpxchg loop by hand.

Given that this is a reliability fix I would forego optimizations of the sort.

Does the patch convert literally all iput calls within ceph into the async variant? I would be worried that mandatory deferral of literally all final iputs may be a regression from perf standpoint.

I see you are mentioning another deadlock, perhaps being in danger of deadlocking is something you could track with a flag within ceph (just like it happens for writeback)? Then the local iput variant could check on both. I have no idea if this is feasible at all for the netfs thing.

No matter what though, it looks like the way forward concerning ->i_count is to make sure it does not drop to 0 within the new primitive.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs-6.18.inode.refcount.preliminaries&id=9e70e985bdc2[1[1c6fe7a160e4d59ddd7c0a39bc077

Quoted 13 lines

fs/ceph/inode.c +13 -0

+       /* simply queue a ceph_inode_work() without setting
+        * i_work_mask bit; other than putting the reference, there is
+        * nothing to do
+        */
+       WARN_ON_ONCE(!queue_work(ceph_inode_to_fs_client(inode)->inode_wq,
+                                &ceph_inode(inode)->i_work));
+
+       /* note: queue_work() cannot fail; it i_work were already
+        * queued, then it would be holding another reference, but no
+        * such reference exists
+        */
+}
+

Given that this is a reliability fix I would forego optimizations of the sort.

Does the patch convert literally all iput calls within ceph into the async variant? I would be worried that mandatory deferral of literally all final iputs may be a regression from perf standpoint.

Quoted 3 lines · Mateusz Guzik

Does the patch convert literally all iput calls within ceph into the async variant? I would be worried that mandatory deferral of literally all final iputs may be a regression from perf standpoint.

I don't think this affects performance at all. It almost never happens that the last reference gets dropped by somebody other than dcache (which only happens under memory pressure).

(Forgot to reply to this part)
No, I changed just the ones that are called from Writeback+Messenger.

I don't think this affects performance at all. It almost never happens that the last reference gets dropped by somebody other than dcache (which only happens under memory pressure).
It was very difficult to reproduce this bug:
- "echo 2 >drop_caches" in a loop
- a kernel patch that adds msleep() to several functions
- another kernel patch that allows me to disconnect the Ceph server via ioctl The latter was to free inode references that are held by Ceph caps. For this deadlock to occur, all references other than writeback/messenger must be gone already.
(It did happen on our production servers, crashing all of them a few days ago causing a major service outage, but apparently in all these years we're the first ones to observe this deadlock bug.)

The iput() function is a dangerous one - if the reference counter goes to zero, the function may block for a long time due to:

- inode_wait_for_writeback() waits until writeback on this inode completes

- the filesystem-specific "evict_inode" callback can do similar things; e.g. all netfs-based filesystems will call netfs_wait_for_outstanding_io() which is similar to inode_wait_for_writeback()

Therefore, callers must carefully evaluate the context they're in and check whether invoking iput() is a good idea at all.

Most of the time, this is not a problem because the dcache holds references to all inodes, and the dcache is usually the one to release the last reference. But this assumption is fragile. For example, under (memcg) memory pressure, the dcache shrinker is more likely to release inode references, moving the inode eviction to contexts where that was extremely unlikely to occur.

Our production servers "found" at least two deadlock bugs in the Ceph filesystem that were caused by this iput() behavior:

1. Writeback may lead to iput() calls in Ceph (e.g. from ceph_put_wrbuffer_cap_refs()) which deadlocks in inode_wait_for_writeback(). Waiting for writeback completion from within writeback will obviously never be able to make any progress. This leads to blocked kworkers like this:

    INFO: task kworker/u777:6:1270802 blocked for more than 122 seconds.
          Not tainted 6.16.7-i1-es #773
    task:kworker/u777:6  state:D stack:0 pid:1270802 tgid:1270802 ppid:2
          task_flags:0x4208060 flags:0x00004000
    Workqueue: writeback wb_workfn (flush-ceph-3)
    Call Trace:
     <TASK>
     __schedule+0x4ea/0x17d0
     schedule+0x1c/0xc0
     inode_wait_for_writeback+0x71/0xb0
     evict+0xcf/0x200
     ceph_put_wrbuffer_cap_refs+0xdd/0x220
     ceph_invalidate_folio+0x97/0xc0
     ceph_writepages_start+0x127b/0x14d0
     do_writepages+0xba/0x150
     __writeback_single_inode+0x34/0x290
     writeback_sb_inodes+0x203/0x470
     __writeback_inodes_wb+0x4c/0xe0
     wb_writeback+0x189/0x2b0
     wb_workfn+0x30b/0x3d0
     process_one_work+0x143/0x2b0
     worker_thread+0x30a/0x450

2. In the Ceph messenger thread (net/ceph/messenger*.c), any iput() call may invoke ceph_evict_inode() which will deadlock in netfs_wait_for_outstanding_io(); since this blocks the messenger thread, completions from the Ceph servers will not ever be received and handled.

It looks like these deadlock bugs have been in the Ceph filesystem code since forever (therefore no "Fixes" tag in this patch). There may be various ways to solve this:

- make iput() asynchronous and defer the actual eviction like fput() (may add overhead)

- make iput() only asynchronous if I_SYNC is set (doesn't solve random things happening inside the "evict_inode" callback)

- add iput_deferred() to make this asynchronous behavior/overhead optional and explicit

- refactor Ceph to avoid iput() calls from within writeback and messenger (if that is even possible)

- add a Ceph-specific workaround

fs/ceph/inode.c +18 -0

index f67025465de0..385d5261632d 100644
@@ -2191,6 +2191,48 @@ void ceph_queue_inode_work(struct inode *inode, int work_bit)
 	}
 }
 
+/**
+ * Queue an asynchronous iput() call in a worker thread.  Use this
+ * instead of iput() in contexts where evicting the inode is unsafe.
+ * For example, inode eviction may cause deadlocks in
+ * inode_wait_for_writeback() (when called from within writeback) or
+ * in netfs_wait_for_outstanding_io() (when called from within the
+ * Ceph messenger).
+ *
+ * @n: how many references to put
+ */
+void ceph_iput_n_async(struct inode *inode, int n)
+{
+	if (unlikely(!inode))
+		return;
+
+	if (likely(atomic_sub_return(n, &inode->i_count) > 0))
+		/* somebody else is holding another reference -
+		 * nothing left to do for us

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.

All that adding a whacky async iput work around does right now is make it harder for Josef to land the patchset that makes this problem go away entirely....

Quoted 8 lines · Dave Chinner

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.

All that adding a whacky async iput work around does right now is make it harder for Josef to land the patchset that makes this problem go away entirely....

Per Max this is a problem present on older kernels as well, something of this sort is needed to cover it regardless of what happens in mainline.

As for mainline, I don't believe Josef's patchset addresses the problem.

The newly added refcount now taken by writeback et al only gates the inode getting freed, it does not gate almost any of iput/evict processing. As in with the patchset writeback does not hold a real reference.

So ceph can still iput from writeback and find itself waiting in inode_wait_for_writeback, unless the filesystem can be converted to use the weaker refcounts and iobj_put instead (but that's not something I would be betting on).

Quoted 22 lines · Mateusz Guzik

Quoted 8 lines · Dave Chinner

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.

All that adding a whacky async iput work around does right now is make it harder for Josef to land the patchset that makes this problem go away entirely....

Per Max this is a problem present on older kernels as well, something of this sort is needed to cover it regardless of what happens in mainline.

As for mainline, I don't believe Josef's patchset addresses the problem.

The newly added refcount now taken by writeback et al only gates the inode getting freed, it does not gate almost any of iput/evict processing. As in with the patchset writeback does not hold a real reference.

Quoted 4 lines

So ceph can still iput from writeback and find itself waiting in inode_wait_for_writeback, unless the filesystem can be converted to use the weaker refcounts and iobj_put instead (but that's not something I would be betting on).

To further elaborate, an extra count which only gates the struct being freed has limited usefulness. Notably it does not help filesystems which need the inode to be valid for use the entire time as evict() is only stalled *after* ->evict_inode(), which might have destroyed the vital parts.

Or to put it differently, the patchset tries to fit btrfs's needs which don't necessarily line up with other filesystems.

For example it
may be ceph needs the full reference in writeback,

then the new ref is
of no use here. But for the sake of argument let's say ceph will get away with the ligher ref instead. Even then this is on the clock for a different filesystem to show up which can't do it and needs an async iput and then its developers are looking at "whacky work arounds".

The actual generic async iput is the actual async iput, not an arbitrary chunk of it after the inode is partway through processing. But then any form of extra refcounting is of no significance.

To that end a non-whacky mechanism to defer iput would be most welcome, presumably provided by the vfs layer itself. Per remarks by Al elsewhere, care needs to be taken to make sure all inodes are sorted out before the super block gets destroyed.

This suggests expanding the super_block to track all of the deferred iputs and drain them early in sb destruction. The current struct inode on LP64 has 2 * 4 byte holes and llist linkage is only 8 bytes, so this can be added without growing the struct above stock kernel.

I would argue it would be good if the work could be deffered to task_work if possible (fput-style). Waiting for these should be easy enough, but arguably the thread which is supposed to get to them can be stalled elsewhere indefinitely, so perhaps this bit is a no-go.

Quoted 28 lines · Mateusz Guzik

Quoted 22 lines · Mateusz Guzik

Quoted 8 lines · Dave Chinner

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.

All that adding a whacky async iput work around does right now is make it harder for Josef to land the patchset that makes this problem go away entirely....

Per Max this is a problem present on older kernels as well, something of this sort is needed to cover it regardless of what happens in mainline.

As for mainline, I don't believe Josef's patchset addresses the problem.

The newly added refcount now taken by writeback et al only gates the inode getting freed, it does not gate almost any of iput/evict processing. As in with the patchset writeback does not hold a real reference.

Quoted 4 lines

So ceph can still iput from writeback and find itself waiting in inode_wait_for_writeback, unless the filesystem can be converted to use the weaker refcounts and iobj_put instead (but that's not something I would be betting on).

Quoted 5 lines

To further elaborate, an extra count which only gates the struct being freed has limited usefulness. Notably it does not help filesystems which need the inode to be valid for use the entire time as evict() is only stalled *after* ->evict_inode(), which might have destroyed the vital parts.

Not sure I follow you - ->evict_inode() comes after waiting for writeback and other VFS operations to complete. There's nothing in the VFS eviction code that actually blocks after ->evict_inode() is called.

Quoted 2 lines

For example it
may be ceph needs the full reference in writeback,

IMO, we should always hold a full reference in writeback, because doing so addresses the underlying eviction vs writeback race condition that exists. i.e. we currently handle the lack of reference counts in writeback by blocking on I_SYNC in eviction to prevent a UAF.

If we have an active reference for writeback in the first place then eviction doesn't need to block on writeback because, by definition, we cannot be performing writeback and eviction at the same time....

Quoted 5 lines

then the new ref is
of no use here. But for the sake of argument let's say ceph will get away with the ligher ref instead. Even then this is on the clock for a different filesystem to show up which can't do it and needs an async iput and then its developers are looking at "whacky work arounds".

Right, that's because we haven't addressed the underlying problem.

That is, we need to hold the right references at the VFS level such that filesystems can't drop the last reference to the inode whilst high level VFS inode operations (such as writeback) are in progress on that inode.

Done properly, eviction can then be done asynchronously for all inodes because we've guaranteed there are no active or pending active users of the inode....

.... and at that point, all the custom async inode garbage collection stuff that XFS does goes away because it is native VFS functionality :)

Quoted 8 lines

The actual generic async iput is the actual async iput, not an arbitrary chunk of it after the inode is partway through processing. But then any form of extra refcounting is of no significance.

To that end a non-whacky mechanism to defer iput would be most welcome, presumably provided by the vfs layer itself. Per remarks by Al elsewhere, care needs to be taken to make sure all inodes are sorted out before the super block gets destroyed.

Yes, but first we have to get the reference counting right, such that inode eviction only occurs after we've guaranteed there are no other active users of the inode. Page cache residency and dirty inodes are still in active use, we should account for them that way.

Quoted 4 lines

This suggests expanding the super_block to track all of the deferred iputs and drain them early in sb destruction. The current struct inode on LP64 has 2 * 4 byte holes and llist linkage is only 8 bytes, so this can be added without growing the struct above stock kernel.

Right, we already do this lockless llist based async garbage collection under ->destroy_inode with XFS.

Quoted 4 lines

I would argue it would be good if the work could be deffered to task_work if possible (fput-style). Waiting for these should be easy enough, but arguably the thread which is supposed to get to them can be stalled elsewhere indefinitely, so perhaps this bit is a no-go.

If the reference counting is right, nothing expect a new lookup on the inode should stall on an inode queued for eviction...

Quoted 8 lines · Dave Chinner

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.

All that adding a whacky async iput work around does right now is make it harder for Josef to land the patchset that makes this problem go away entirely....

Per Max this is a problem present on older kernels as well, something of this sort is needed to cover it regardless of what happens in mainline.

As for mainline, I don't believe Josef's patchset addresses the problem.

The newly added refcount now taken by writeback et al only gates the inode getting freed, it does not gate almost any of iput/evict processing. As in with the patchset writeback does not hold a real reference.

So ceph can still iput from writeback and find itself waiting in inode_wait_for_writeback, unless the filesystem can be converted to use the weaker refcounts and iobj_put instead (but that's not something I would be betting on).

- wait for Josef to finish his inode refcount rework patchset that gets rid of this whole "writeback doesn't hold an inode reference" problem that is the root cause of this the deadlock.