Skip to content

xfs: log recovery fixes

Dave Chinner linux-xfs oe-kbuild oe-kbuild-all 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest7 patches 4 repliesIn review
  2. v37 patches 21 repliesSuperseded
  3. v14 patches 2 repliesSuperseded

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7xfs: log worker needs to start before intent/unlink recoveryfs/xfs/xfs_log.c+1 -20 replies
2/7xfs: check buffer pin state after locking in delwri_submitfs/xfs/xfs_buf.c+3 -20 replies
3/7xfs: xfs_ail_push_all_sync() stalls when racing with updatesfs/xfs/xfs_trans_ail.c+16 -51 reply
4/7xfs: async CIL flushes need pending pushes to be made stablefs/xfs/xfs_log_cil.c+19 -31 reply
5/7xfs: log items should have a xlog pointer, not a mountfs/xfs/xfs_bmap_item.c fs/xfs/xfs_buf_item.c fs/xfs/xfs_extfree_item.c +7 more+14 -120 replies
6/7xfs: AIL should be log centricfs/xfs/xfs_trans.c fs/xfs/xfs_trans_ail.c fs/xfs/xfs_trans_priv.h+16 -150 replies
7/7xfs: xfs_is_shutdown vs xlog_is_shutdown cage fightfs/xfs/xfs_buf.c fs/xfs/xfs_icache.c fs/xfs/xfs_inode.c +2 more+70 -152 replies

Reviews trailers per patch

All 7 patches have a Reviewed-by or Acked-by.

PatchChandan Babu RDarrick J. WongMatthew Wilcox
1/7R-b R-b
2/7R-b R-b
3/7R-b R-b
4/7R-b R-b Reported-and-tested-by
5/7R-b R-b
6/7R-b R-b
7/7R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v4 thread

Cover letter [PATCH 0/7 v4] xfs: log recovery fixes

Willy reported generic/530 had started hanging on his test machines and I've tried to reproduce the problem he reported. While I haven't reproduced the exact hang he's been having, I've found a couple of others while running g/530 in a tight loop on a couple of test machines.

The first 3 patches are defensive fixes - the log worker acts as a watchdog, and the issues in patch 2 and 3 were triggered on my testing of g/530 and lead to 30s delays that the log worker watchdog caught. Without the watchdog, these may actually be deadlock triggers.

The 4th patch is the one that fixes the problem Willy reported. It is a regression from conversion of the AIL pushing to use non-blocking CIL flushes. It is unknown why this suddenly started showing up on Willy's test machine right now, and why only on that machine, but it is clearly a problem. This patch catches the state that leads to the deadlock and breaks it with an immediate log force to flush any pending iclogs.

In testing these patches, I found generic/388 was causing frequent failures with recovery failing because inode clusters were being found uninitialised in log recovery. That turns out to be a zero day race condition in the forced shutdown code, and that is fixed over the patches 5-7. In short, we can't abort writeback of log items before we shut down the log (because that's separate to -mount- shutdown) as removing aborted log items can move the tail of the log forward and that can be propagated to the on-disk log and corrupt it if timing is just right.

Fixing this takes failures of g/388 from 1 in 5-10 runs to 1 in 100 runs. There is a change in patch 7 that I mention "I'm not sure how this can happen here, but it's handled elsewhere like this" that avoids a double remove of an aborted inode from the AIL that results in an ASSERT failure. I *think* I now know how that can occur, but fixing it is another set of patches, and it may be a recent regression rather than a long standing issue.

Version 4:
- rebase on 5.17-rc8
- commit message fixups.
- more comments for non-obvious xlog_is_shutdown() use cases.

Version 3:
- https://lore.kernel.org/linux-xfs/[email protected]/
- added fixes for generic/388 failures.

Version 2:
- https://lore.kernel.org/linux-xfs/[email protected]/
- updated to 5.17-rc7
- tested by Willy.

Version 1:
- https://lore.kernel.org/linux-xfs/[email protected]/