Skip to contentSkip to conversation

xfs: drain inodegc before quota teardown on mount failure

PATCH Superseded Zhang Cen 2 participants 6 messages May 14 – Jun 2 XFS filesystem

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Zhang Cen

author
#0 lore raw

PATCH 1/1 xfs: drain inodegc before quota teardown on mount failure

xfs_mountfs() starts background inodegc before mount completion. If a later setup step fails, the out_agresv unwind can release the quota inodes via xfs_qm_unmount_quotas() while m_qflags still leave dqattach enabled, and the common failure path can destroy mp->m_quotainfo via xfs_qm_unmount() before the first common inodegc drain.

That leaves two teardown windows for an inodegc worker that resumes xfs_inactive(): dqget can still need qi_*quotaip after xfs_qm_unmount_quotas(), and the quota cache lookup still dereferences mp->m_quotainfo until xfs_qm_unmount() finishes. Normal unmount and the quotacheck abort path already drain inodegc before those teardown stages.

Fix this by draining inodegc twice on the mount failure path: once on out_agresv before dropping AG reservations and calling xfs_qm_unmount_quotas(), and again on the common teardown path before xfs_qm_unmount(). Keep xfs_unmount_flush_inodes() in place afterwards so that reclaim queued by quota, realtime, root, or metadir teardown still runs after xfs_qm_unmount() releases quota inodes on partially initialized mounts.

The buggy scenario involves two paths, with each column showing the order within that path:

path A label: mount failure unwind path B label: inodegc worker
1. xfs_mountfs() starts inodegc and 1. Recovery or eviction leaves an
later jumps into out_agresv. ordinary inode queued for
background inactivation.
2. out_agresv used to reach 2. xfs_inodegc_worker() runs
xfs_qm_unmount_quotas(), which xfs_inactive() on that inode.
releases qi_*quotaip while 3. xfs_inactive() calls
m_qflags still allow dqattach. xfs_qm_dqattach().
3. The common unwind later reached 4. dqattach follows m_qflags into

   xfs_qm_unmount(), which purged          dqget, which may need the quota
   dquots and destroyed                    inodes and mp->m_quotainfo being
   mp->m_quotainfo.                        torn down by the unwind.

4. The first common inodegc drain
used to happen only after quota
teardown had started.

Sanitizer validation reported:

general protection fault
Call trace:
  xfs_qm_dqget_cache_lookup() (?:?)
  trace_function() (kernel/trace/trace.h:806)
  xfs_qm_dqget_inode() (?:?)
  ring_buffer_unlock_commit() (kernel/trace/ring_buffer.c:4306)
  xfs_qm_dqattach_locked() (?:?)
  srso_alias_return_thunk() (arch/x86/include/asm/nospec-branch.h:375)
  xfs_qm_dqattach() (?:?)
  xfs_free_eofblocks() (?:?)
  xfs_inactive() (?:?)
  xfs_inodegc_worker() (?:?)
  process_one_work() (kernel/workqueue.c:3200)
  worker_thread() (?:?)
  kthread() (?:?)
  _raw_spin_unlock_irq() (kernel/locking/spinlock.c:204)
  ret_from_fork() (?:?)
  __switch_to() (?:?)
  ret_from_fork_asm() (?:?)
Fixes ab23a7768739 ("xfs: per-cpu deferred inode inactivation queues")

---

fs/xfs/xfs_mount.c +14 -7
index b24195f570cd..e8a870482d1a 100644
@@ -1228,6 +1228,12 @@ xfs_mountfs(
 	return 0;
 
  out_agresv:
+	/*
+	 * Drain ordinary inodegc before we drop per-AG reservations or tear
+	 * down quota state.  Background inactivation can still attach dquots
+	 * here, and dqget can still need the quota inodes.
+	 */
+	xfs_inodegc_flush(mp);
 	xfs_fs_unreserve_ag_blocks(mp);
 	xfs_qm_unmount_quotas(mp);
 	if (xfs_has_zoned(mp))
@@ -1236,24 +1242,25 @@ xfs_mountfs(
 	xfs_rtunmount_inodes(mp);
  out_rele_rip:
 	xfs_irele(rip);
-	/* Clean out dquots that might be in memory after quotacheck. */
-	xfs_qm_unmount(mp);
  out_free_metadir:
 	if (mp->m_metadirip)
 		xfs_irele(mp->m_metadirip);
 
 	/*
-	 * Inactivate all inodes that might still be in memory after a log
-	 * intent recovery failure so that reclaim can free them.  Metadata
-	 * inodes and the root directory shouldn't need inactivation, but the
-	 * mount failed for some reason, so pull down all the state and flee.
+	 * Flush inodegc before we destroy quotainfo.  This is the first drain
+	 * for paths that bypass out_agresv, and it also catches any reclaim
+	 * queued by the quota, realtime, root, or metadir teardown above.
 	 */
 	xfs_inodegc_flush(mp);
 
+	/* Clean out dquots that might be in memory after quotacheck. */
+	xfs_qm_unmount(mp);
+
 	/*
 	 * Flush all inode reclamation work and flush the log.
 	 * We have to do this /after/ rtunmount and qm_unmount because those
-	 * two will have scheduled delayed reclaim for the rt/quota inodes.
+	 * will have scheduled delayed reclaim for the rt/quota/root/metadir
+	 * inodes.
 	 *
 	 * This is slightly different from the unmountfs call sequence
 	 * because we could be tearing down a partially set up mount.  In
#1 lore raw
Quoted 1 line · Zhang Cen

Sanitizer validation reported:

How did you run into this? Normal xfstests under a new sanitizer?

Quoted 3 lines
Fixes ab23a7768739 ("xfs: per-cpu deferred inode inactivation queues")

Nit: no empty line between the Fixes and the Signoff or other tags, please.

Quoted 5 lines
fs/xfs/xfs_mount.c +0 -0
index b24195f570cd..e8a870482d1a 100644

fs/xfs/xfs_mount.c xfs_mountfs( · line 1236

Quoted 9 lines
fs/xfs/xfs_mount.c +6 -0
@@ -1228,6 +1228,12 @@ xfs_mountfs(
 	return 0;
 
  out_agresv:
+	/*
+	 * Drain ordinary inodegc before we drop per-AG reservations or tear
+	 * down quota state.  Background inactivation can still attach dquots
+	 * here, and dqget can still need the quota inodes.
+	 */
+	xfs_inodegc_flush(mp);

This looks good.

Quoted 7 lines
fs/xfs/xfs_mount.c +0 -0
  out_free_metadir:
 	if (mp->m_metadirip)
 		xfs_irele(mp->m_metadirip);
fs/xfs/xfs_mount.c +0 -2
 	/*
-	 * Inactivate all inodes that might still be in memory after a log
-	 * intent recovery failure so that reclaim can free them.  Metadata

fs/xfs/xfs_mount.c xfs_mountfs( · line 1257

Quoted 10 lines
fs/xfs/xfs_mount.c +5 -2
-	 * inodes and the root directory shouldn't need inactivation, but the
-	 * mount failed for some reason, so pull down all the state and flee.
+	 * Flush inodegc before we destroy quotainfo.  This is the first drain
+	 * for paths that bypass out_agresv, and it also catches any reclaim
+	 * queued by the quota, realtime, root, or metadir teardown above.
 	 */
 	xfs_inodegc_flush(mp);
 
+	/* Clean out dquots that might be in memory after quotacheck. */
+	xfs_qm_unmount(mp);

This is later in the teardown order than where I'd expect it. The first thing that can bring in quotainodes is xfs_qm_newmount. I'd expect a new label placed above out_rtunmount to handle this. Is there a reason I'm missing why this won't work?

author
#2 lore raw

Hi Christoph,

Sorry, I missed this reply earlier and only noticed it now from the public archive.

Quoted 1 line

How did you run into this? Normal xfstests under a new sanitizer?

This was not from normal xfstests. I reproduced it in a KASAN-enabled QEMU guest with a disposable quota-enabled XFS image, then forced a late mount failure after quota setup while tracing the inodegc and quota teardown path. The failing ordering was:

xfs_qm_mount_quotas()
late mount failure
xfs_qm_unmount_quotas()
xfs_qm_unmount() / xfs_qm_destroy_quotainfo()
xfs_inodegc_flush()

with inodegc still able to reach the dqattach/dqget side during the teardown window.

Quoted 1 line

This looks good.

Thanks.

Quoted 4 lines

This is later in the teardown order than where I'd expect it. The first thing that can bring in quotainodes is xfs_qm_newmount. I'd expect a new label placed above out_rtunmount to handle this. Is there a reason I'm missing why this won't work?

No, I don't think you are missing anything. A separate label above out_rtunmount is the cleaner place for this. I'll send a v2 that keeps the out_agresv inodegc drain before xfs_qm_unmount_quotas(), adds an out_qm_unmount label above out_rtunmount, and routes the later post-quota failure paths through that label.

I'll also fix the trailer spacing in v2.

Thanks,
Zhang

#3 lore raw
Quoted 5 lines · Zhang Cen

This was not from normal xfstests. I reproduced it in a KASAN-enabled QEMU guest with a disposable quota-enabled XFS image, then forced a late mount failure after quota setup while tracing the inodegc and quota teardown path. The failing ordering was:

Quoted 8 lines

xfs_qm_mount_quotas()
late mount failure
xfs_qm_unmount_quotas()
xfs_qm_unmount() / xfs_qm_destroy_quotainfo()
xfs_inodegc_flush()

with inodegc still able to reach the dqattach/dqget side during the teardown window.

Can you come up with an xfstest for this? I guess you'd need a new error injection point, but that should not be horrible.

author
#4 lore raw

Hi Christoph,

Quoted 2 lines

Can you come up with an xfstest for this? I guess you'd need a new error injection point, but that should not be horrible.

I tried to turn this into an xfstest.

In my local attempts, a mount-failure errortag after xfs_qm_mount_quotas() was not enough by itself, because ordinary test setup did not reliably leave an inodegc item that would enter the dqattach/eofblocks path during the mount-failure unwind.

The draft I have locally uses two DEBUG-only errortags:

- mount_fail_after_qm_mount_quotas

This injects -EIO immediately after xfs_qm_mount_quotas().

- force_inodegc_eofblocks

This makes a regular inode released after quotacheck scanning go through inodegc/eofblocks cleanup, and delays that inodegc work so the mount-failure unwind can reach quota teardown first.

The xfstest creates one regular scratch file without quotas enabled, unmounts, and then remounts with:

-o uquota,errortag=force_inodegc_eofblocks,errortag=mount_fail_after_qm_mount_quotas

On a kernel with the errortags but without the inodegc drain fix, this reproduces the same teardown ordering problem:

  KASAN: null-ptr-deref
  Workqueue: xfs-inodegc/vdc xfs_inodegc_worker
  xfs_qm_dqget_cache_lookup()
  xfs_qm_dqget_inode()
  xfs_qm_dqattach_locked()
  xfs_qm_dqattach()
  xfs_free_eofblocks()
  xfs_inactive()
  xfs_inodegc_worker()

With the drain fix applied, the same xfstest passes and the filesystem can be mounted again cleanly afterwards.

I do not have much experience with XFS-specific fstests, so I wanted to check whether this is the right shape before posting it formally. If this approach looks acceptable, I can send a small kernel series adding the DEBUG errortags and the mount-failure fix, followed by a separate xfstests patch.

Thanks,
Zhang Cen

#5 lore raw
Quoted 9 lines · Cen Zhang

In my local attempts, a mount-failure errortag after xfs_qm_mount_quotas() was not enough by itself, because ordinary test setup did not reliably leave an inodegc item that would enter the dqattach/eofblocks path during the mount-failure unwind.

The draft I have locally uses two DEBUG-only errortags:

- mount_fail_after_qm_mount_quotas

Quoted 7 lines

This injects -EIO immediately after xfs_qm_mount_quotas().

- force_inodegc_eofblocks

This makes a regular inode released after quotacheck scanning go through inodegc/eofblocks cleanup, and delays that inodegc work so the mount-failure unwind can reach quota teardown first.

This sounds pretty useful, but I actually have a different question:

Quoted 8 lines
  Workqueue: xfs-inodegc/vdc xfs_inodegc_worker
  xfs_qm_dqget_cache_lookup()
  xfs_qm_dqget_inode()
  xfs_qm_dqattach_locked()
  xfs_qm_dqattach()
  xfs_free_eofblocks()
  xfs_inactive()
  xfs_inodegc_worker()

What kind of inode do we actually get here? I guess your test case reproduces an unclean shutdown and this processes inodes that otherwise would have been processed on unmount?

I'm kinda tempted to just make xfs_can_free_eofblocks return false during early mount (or rather during mount failure), as trying to fix all the corner cases here seems like an uphill battle ( also see the sashiko review)