Skip to contentSkip to conversation

fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

PATCH Superseded Srikanth C S 3 participants 3 messages Nov 2, 2022

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can put the machine into rescue shell. To address this problem, mount and umount the fs before running xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the resuce shell if repair detects any corruptions

---
 fsck/xfs_fsck.sh | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +21 -2
index 6af0f22..4ef61db 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,24 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ -n "$REPAIR" ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 

--
1.8.3.1

#1 lore raw
Quoted 10 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

Quoted 5 lines

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can put the machine into rescue shell. To address this problem, mount and umount the fs before running xfs_repair.

"This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair."

Quoted 3 lines

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the resuce shell if repair detects

s/resuce/rescue/

Quoted 3 lines

any corruptions

Ah good, your email works again.

Quoted 23 lines
---
 fsck/xfs_fsck.sh | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +3 -1
index 6af0f22..4ef61db 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done

fsck/xfs_fsck.sh fi · line 70

Quoted 6 lines
fsck/xfs_fsck.sh +2 -1
@@ -64,7 +66,24 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ -n "$REPAIR" ]; then

test -n checks that its argument "$REPAIR" is nonzero length. Since you set REPAIR=false above, this test will always return success. I think you wanted:

	if [ $error -eq 2 ] && [ $REPAIR = true ]; then

here?

Quoted 6 lines
fsck/xfs_fsck.sh +6 -0
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"

fsck/xfs_fsck.sh fi · line 86

Quoted 10 lines
fsck/xfs_fsck.sh +10 -0
+                               ;;
+                       esac
+               done
+               mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error

The rest of the logic looks ok to me. The new behavior needs to be documented in the manpage. Here's a fugly troff snippet that could be added towards the end of man/man8/fsck.xfs.8:

If the system administrator adds "fsck.mode=force fsck.repair=yes" to the kernel command line,
.B fsck.xfs
will detect a dirty log and mount and unmount the filesystem to clean the log before running
.BR xfs_repair (8).

--D

Quoted 5 lines
fsck/xfs_fsck.sh +0 -1
        exit $?
 fi
 
--

1.8.3.1

#2 lore raw
In reply to Srikanth C S
Quoted 48 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can put the machine into rescue shell. To address this problem, mount and umount the fs before running xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the resuce shell if repair detects any corruptions

---
 fsck/xfs_fsck.sh | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +3 -1
index 6af0f22..4ef61db 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,24 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV

fsck/xfs_fsck.sh fi · line 77

Quoted 10 lines
fsck/xfs_fsck.sh +9 -1
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ -n "$REPAIR" ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;

What if fsck is being called on all devices (i.e. -a) or something other than the root device? Don't we have to match the root flags to the root dev? It's likely that there will be a root=<dev> parameter on the CLI, so we'd want to grab that and check that it matches $DEV before using ROOTFLAGS, right?

Otherwise this looks OK.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]