fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
PATCH v3 1/1 fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.
fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.
Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.
--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +29 -2
@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done
@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$? + if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o ${x#rootflags=}" + ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 + else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi
Signature · 1 line
1.8.3.1
Hi.
Did you plan to resend V3 again, or is this supposed to be V4?
Quoted 81 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell becausexfs_fsck.shdoes not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunningxfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)fsck/xfs_fsck.sh +29
-3@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$? + if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o ${x#rootflags=}" + ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 + else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi --1.8.3.1
Signature · 1 line
Carlos Maiolino
A message that is not in the archive (sent off-list or not yet received): c-vuqhpmmrL6JSN0ZRnqX7c1BUcXw5gJ9L2UZ2lG3H8hCJRNIn_uan2rVHLDUPwgY24Nv3WZpiBt2nflhVadtA==@protonmail.internalid
A message that is not in the archive (sent off-list or not yet received): CY4PR10MB1479D19A047EAB8558445EC7A30C9@CY4PR10MB1479.namprd10.prod.outlook.com
Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 3 lines · Srikanth C S
Hi
I resent the same patch as I did not see any review comments.
Unless I'm looking at the wrong patch, there were comments on your previous submission:
https://lore.kernel.org/linux-xfs/[email protected]/T/#t
Am I missing something?
Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:
[RESEND PATCH] <subject>
Cheers.
Quoted 104 lines
-Srikanth
__________________________________________________________________From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected] <[email protected]>
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairHi.
Did you plan to resend V3 again, or is this supposed to be V4?Quoted 10 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.Quoted 3 lines
fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop themachine
Quoted 1 line
into the rescue shell because
xfs_fsck.shdoes not know how to cleanthe
Quoted 37 lines
log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning
xfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)fsck/xfs_fsck.sh +4
-2@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$?fsck/xfs_fsck.sh fi · line 79
Quoted 10 lines
fsck/xfs_fsck.sh +10
-0+ if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o${x#rootflags=}"
fsck/xfs_fsck.sh fi
Quoted 6 lines
fsck/xfs_fsck.sh +6
-0+ ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit1
Quoted 13 lines
fsck/xfs_fsck.sh +9
-1+ else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi --Quoted 1 line
1.8.3.1
--
Carlos Maiolino
Signature · 1 line
Carlos Maiolino
RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 21 lines
-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairQuoted 3 lines · Srikanth C S
Hi
I resent the same patch as I did not see any review comments.
Unless I'm looking at the wrong patch, there were comments on your previous
submission:
Quoted 7 lines
https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M- 0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z- 2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF v_lA1Zs_jsUgr$Am I missing something?
All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.
So I am still wondering on how to proceed with this patch. Any comments would be helpful.
Quoted 6 lines
Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:
[RESEND PATCH] <subject>
Thanks for the info. Didn't know this.
Quoted 107 lines
Cheers.
Quoted 4 lines
-Srikanth
__________________________________________________________
________Quoted 8 lines
From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]Quoted 89 lines
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running
xfs_repairHi.
Did you plan to resend V3 again, or is this supposed to be V4?Quoted 10 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.Quoted 3 lines
fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop themachine
Quoted 1 line
into the rescue shell because
xfs_fsck.shdoes not know how to cleanthe
Quoted 37 lines
log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning
xfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)fsck/xfs_fsck.sh +4
-2@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$?fsck/xfs_fsck.sh fi · line 79
Quoted 10 lines
fsck/xfs_fsck.sh +10
-0+ if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o${x#rootflags=}"
fsck/xfs_fsck.sh fi
Quoted 6 lines
fsck/xfs_fsck.sh +6
-0+ ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit1
Quoted 12 lines
fsck/xfs_fsck.sh +9
-0+ else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
--
1.8.3.1--
Carlos Maiolino--
Carlos Maiolino
Regards,
Srikanth
Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 21 lines · Srikanth C S
Quoted 21 lines
-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairQuoted 3 lines · Srikanth C S
Hi
I resent the same patch as I did not see any review comments.
Unless I'm looking at the wrong patch, there were comments on your previous
submission:
Quoted 7 lines
Quoted 7 lines
https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M- 0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z- 2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF v_lA1Zs_jsUgr$Am I missing something?
Err.... whose comments, Joseph's or Gao's?
Quoted 8 lines
All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.
So I am still wondering on how to proceed with this patch. Any comments would be helpful.
Same here, though the long holiday weekend probably didn't help.
--D
Quoted 123 lines
Quoted 5 lines
Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:
[RESEND PATCH] <subject>
Thanks for the info. Didn't know this.
Quoted 107 lines
Cheers.
Quoted 4 lines
-Srikanth
__________________________________________________________
________Quoted 8 lines
From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]Quoted 89 lines
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running
xfs_repairHi.
Did you plan to resend V3 again, or is this supposed to be V4?Quoted 10 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.Quoted 3 lines
fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop themachine
Quoted 1 line
into the rescue shell because
xfs_fsck.shdoes not know how to cleanthe
Quoted 37 lines
log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning
xfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)fsck/xfs_fsck.sh +4
-2@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$?fsck/xfs_fsck.sh fi · line 79
Quoted 10 lines
fsck/xfs_fsck.sh +10
-0+ if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o${x#rootflags=}"
fsck/xfs_fsck.sh fi
Quoted 6 lines
fsck/xfs_fsck.sh +6
-0+ ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit1
Quoted 12 lines
fsck/xfs_fsck.sh +9
-0+ else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fiQuoted 7 lines
Quoted 4 lines
Quoted 2 lines
--
1.8.3.1--
Carlos Maiolino--
Carlos MaiolinoRegards,
Srikanth
RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 40 lines
-----Original Message-----
From: Darrick J. Wong <[email protected]>
Sent: 29 November 2022 04:34 AM
To: Srikanth C S <[email protected]>
Cc: Carlos Maiolino <[email protected]>; [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairQuoted 18 lines · Srikanth C S
Quoted 18 lines
-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>;
Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairQuoted 3 lines · Srikanth C S
Hi
I resent the same patch as I did not see any review comments.
Quoted 8 lines
Quoted 8 lines
Unless I'm looking at the wrong patch, there were comments on your previous
submission:https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M-
0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z-
Quoted 16 lines
2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF
Quoted 3 lines
Quoted 3 lines
v_lA1Zs_jsUgr$
Am I missing something?
Err.... whose comments, Joseph's or Gao's?
Quoted 8 lines
All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.
So I am still wondering on how to proceed with this patch. Any comments would be helpful.
@Carlos Maiolino, Any comments or thoughts on this patch?
-Srikanth
Quoted 131 lines
Same here, though the long holiday weekend probably didn't help.
--D
Quoted 3 lines
Quoted 3 lines
Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the
patch tag:
Quoted 11 lines
Quoted 2 lines
[RESEND PATCH] <subject>
Thanks for the info. Didn't know this.
Quoted 8 lines
Cheers.
Quoted 4 lines
-Srikanth
__________________________________________________________
Quoted 6 lines
Quoted 6 lines
________
Quoted 5 lines
From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; DarrickWong
Quoted 20 lines
Quoted 20 lines
Quoted 3 lines
<[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Quoted 16 lines
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running
xfs_repairHi.
Did you plan to resend V3 again, or is this supposed to be V4?Quoted 10 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop themachine
Quoted 1 line
into the rescue shell because
xfs_fsck.shdoes not know how toclean
Quoted 78 lines
Quoted 68 lines
Quoted 68 lines
the > log. Since the administrator told us to force repairs, address the > deficiency by cleaning the log and rerunning xfs_repair. > > Run xfs_repair -e when fsck.mode=force and repair=auto or yes. > Replay the logs only if fsck.mode=force and fsck.repair=yes. For > other option -fa and -f drop to the rescue shell if repair detects > any corruptions. > > Signed-off-by: Srikanth C S <[email protected]> > --- > fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- > 1 file changed, 29 insertions(+), 2 deletions(-) > > diff --git a/fsck/xfs_fsck.sh b/fsck/xfs_fsck.sh > index 6af0f22..62a1e0b 100755 > --- a/fsck/xfs_fsck.sh > +++ b/fsck/xfs_fsck.sh > @@ -31,10 +31,12 @@ repair2fsck_code() { > > AUTO=false > FORCE=false > +REPAIR=false > while getopts ":aApyf" c > do > case $c in > - a|A|p|y) AUTO=true;; > + a|A|p) AUTO=true;; > + y) REPAIR=true;; > f) FORCE=true;; > esac > done > @@ -64,7 +66,32 @@ fi > > if $FORCE; then > xfs_repair -e $DEV > - repair2fsck_code $? > + error=$? > + if [ $error -eq 2 ] && [ $REPAIR = true ]; then > + echo "Replaying log for $DEV" > + mkdir -p /tmp/repair_mnt || exit 1 > + for x in $(cat /proc/cmdline); do > + case $x in > + root=*) > + ROOT="${x#root=}" > + ;; > + rootflags=*) > + ROOTFLAGS="-o ${x#rootflags=}" > + ;; > + esac > + done > + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) > + if [ $(basename $DEV) = $(basename $ROOT) ]; then > + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 > + else > + mount $DEV /tmp/repair_mnt || exit 1 > + fi > + umount /tmp/repair_mnt > + xfs_repair -e $DEV > + error=$? > + rm -d /tmp/repair_mnt > + fi > + repair2fsck_code $error > exit $? > fi >Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
--
1.8.3.1--
Carlos Maiolino--
Carlos MaiolinoRegards,
Srikanth
Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
So I am still wondering on how to proceed with this patch. Any comments would be helpful.
@Carlos Maiolino, Any comments or thoughts on this patch?
Sorry, didn't have time to look into it yet.
Signature · 1 line
Carlos Maiolino
Quoted 17 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell becausexfs_fsck.shdoes not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunningxfs_repair.
Quoted 6 lines
Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Signed-off-by Srikanth C S <[email protected]>
Apologies it took so long, the patch seems fine to me. Will test.
Quoted 58 lines
--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)fsck/xfs_fsck.sh +29
-3@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$? + if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o ${x#rootflags=}" + ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 + else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi --1.8.3.1
Signature · 1 line
Carlos Maiolino
Hi Srikanth.
Quoted 22 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell becausexfs_fsck.shdoes not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunningxfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.
Quoted 4 lines
Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
Did you by any chance wrote this patch on top of something else you have in your tree?
It doesn't apply to the tree without tweaking it, and the last changes we've in the fsck/xfs_fsck.sh file are from 2018, so I assume you have something before this patch in your tree.
Could you please rebase this patch against xfsprogs for-next and resend it? Feel free to keep my RwB as long as you don't change the code semantics.
Cheers.
Quoted 55 lines
fsck/xfs_fsck.sh +29
-3@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$? + if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o ${x#rootflags=}" + ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 + else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi --1.8.3.1
Signature · 1 line
Carlos Maiolino
RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair
Quoted 41 lines
-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 13 December 2022 03:10 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before runningxfs_repairHi Srikanth.
Quoted 21 lines · Srikanth C S
After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off
xfs_repairif the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.fsck.xfs does
xfs_repair-e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell becausexfs_fsck.shdoes not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunningxfs_repair.Run
xfs_repair-e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.Quoted 4 lines
Signed-off-by Srikanth C S <[email protected]>--- fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
Quoted 7 lines
Did you by any chance wrote this patch on top of something else you have in your tree?
It doesn't apply to the tree without tweaking it, and the last changes we've in the
fsck/xfs_fsck.shfile are from 2018, so I assume you have something before this patch in your tree.
Sorry for the inconvenience, will verify this.
Quoted 3 lines
Could you please rebase this patch against xfsprogs for-next and resend it? Feel free to keep my RwB as long as you don't change the code semantics.
Let me rebase the patch and resend it. Thanks for the Reviewed by.
Quoted 55 lines
Cheers.
Quoted 53 lines
fsck/xfs_fsck.sh index renamed from fsck/xfs_fsck.sh +0
-06af0f22..62a1e0b 100755
fsck/xfs_fsck.sh +29
-2@@ -31,10 +31,12 @@ repair2fsck_code() { AUTO=false FORCE=false +REPAIR=false while getopts ":aApyf" c do case $c in - a|A|p|y) AUTO=true;; + a|A|p) AUTO=true;; + y) REPAIR=true;; f) FORCE=true;; esac done@@ -64,7 +66,32 @@ fi if $FORCE; then xfs_repair -e $DEV - repair2fsck_code $? + error=$? + if [ $error -eq 2 ] && [ $REPAIR = true ]; then + echo "Replaying log for $DEV" + mkdir -p /tmp/repair_mnt || exit 1 + for x in $(cat /proc/cmdline); do + case $x in + root=*) + ROOT="${x#root=}" + ;; + rootflags=*) + ROOTFLAGS="-o ${x#rootflags=}" + ;; + esac + done + test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device) + if [ $(basename $DEV) = $(basename $ROOT) ]; then + mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1 + else + mount $DEV /tmp/repair_mnt || exit 1 + fi + umount /tmp/repair_mnt + xfs_repair -e $DEV + error=$? + rm -d /tmp/repair_mnt + fi + repair2fsck_code $error exit $? fi
Quoted 5 lines
Quoted 2 lines
--
1.8.3.1--
Carlos Maiolino
Thanks,
Srikanth