Skip to contentSkip to conversation

fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

PATCH Superseded Srikanth C S 3 participants 10 messages Nov 23, 2022 – Dec 13, 2022 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +29 -2
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
Signature · 1 line
1.8.3.1
#1 lore raw

Hi.

Did you plan to resend V3 again, or is this supposed to be V4?

Quoted 81 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +29 -3
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
--

1.8.3.1

Signature · 1 line
Carlos Maiolino

A message that is not in the archive (sent off-list or not yet received): c-vuqhpmmrL6JSN0ZRnqX7c1BUcXw5gJ9L2UZ2lG3H8hCJRNIn_uan2rVHLDUPwgY24Nv3WZpiBt2nflhVadtA==@protonmail.internalid

A message that is not in the archive (sent off-list or not yet received): CY4PR10MB1479D19A047EAB8558445EC7A30C9@CY4PR10MB1479.namprd10.prod.outlook.com

#4 lore raw

Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 3 lines · Srikanth C S

Hi

I resent the same patch as I did not see any review comments.

Unless I'm looking at the wrong patch, there were comments on your previous submission:

https://lore.kernel.org/linux-xfs/[email protected]/T/#t

Am I missing something?

Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:

[RESEND PATCH] <subject>

Cheers.

Quoted 104 lines

-Srikanth

     __________________________________________________________________

From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected] <[email protected]>
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Hi.
Did you plan to resend V3 again, or is this supposed to be V4?

Quoted 10 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

Quoted 3 lines

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the

machine

Quoted 1 line

into the rescue shell because xfs_fsck.sh does not know how to clean

the

Quoted 37 lines

log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +4 -2
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?

fsck/xfs_fsck.sh fi · line 79

Quoted 10 lines
fsck/xfs_fsck.sh +10 -0
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o

${x#rootflags=}"

fsck/xfs_fsck.sh fi

Quoted 6 lines
fsck/xfs_fsck.sh +6 -0
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit

1

Quoted 13 lines
fsck/xfs_fsck.sh +9 -1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
--
Quoted 1 line

1.8.3.1

--
Carlos Maiolino

Signature · 1 line
Carlos Maiolino
#5 lore raw

RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 21 lines

-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 3 lines · Srikanth C S

Hi

I resent the same patch as I did not see any review comments.

Unless I'm looking at the wrong patch, there were comments on your previous
submission:

Quoted 7 lines

https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M- 0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z- 2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF v_lA1Zs_jsUgr$

Am I missing something?

All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.

So I am still wondering on how to proceed with this patch. Any comments would be helpful.

Quoted 6 lines

Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:

[RESEND PATCH] <subject>

Thanks for the info. Didn't know this.

Quoted 107 lines

Cheers.

Quoted 4 lines

-Srikanth

__________________________________________________________
________

Quoted 8 lines

From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]

<[email protected]>

Quoted 89 lines

Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Hi.
Did you plan to resend V3 again, or is this supposed to be V4?

Quoted 10 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

Quoted 3 lines

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the

machine

Quoted 1 line

into the rescue shell because xfs_fsck.sh does not know how to clean

the

Quoted 37 lines

log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +4 -2
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?

fsck/xfs_fsck.sh fi · line 79

Quoted 10 lines
fsck/xfs_fsck.sh +10 -0
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o

${x#rootflags=}"

fsck/xfs_fsck.sh fi

Quoted 6 lines
fsck/xfs_fsck.sh +6 -0
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit

1

Quoted 12 lines
fsck/xfs_fsck.sh +9 -0
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

--
1.8.3.1

--
Carlos Maiolino

--
Carlos Maiolino

Regards,
Srikanth

#6 lore raw

Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 21 lines · Srikanth C S
Quoted 21 lines

-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 3 lines · Srikanth C S

Hi

I resent the same patch as I did not see any review comments.

Unless I'm looking at the wrong patch, there were comments on your previous
submission:

Quoted 7 lines
Quoted 7 lines

https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M- 0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z- 2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF v_lA1Zs_jsUgr$

Am I missing something?

Err.... whose comments, Joseph's or Gao's?

Quoted 8 lines

All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.

So I am still wondering on how to proceed with this patch. Any comments would be helpful.

Same here, though the long holiday weekend probably didn't help.

--D

Quoted 123 lines
Quoted 5 lines

Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the patch tag:

[RESEND PATCH] <subject>

Thanks for the info. Didn't know this.

Quoted 107 lines

Cheers.

Quoted 4 lines

-Srikanth

__________________________________________________________
________

Quoted 8 lines

From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]

<[email protected]>

Quoted 89 lines

Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Hi.
Did you plan to resend V3 again, or is this supposed to be V4?

Quoted 10 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

Quoted 3 lines

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the

machine

Quoted 1 line

into the rescue shell because xfs_fsck.sh does not know how to clean

the

Quoted 37 lines

log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +4 -2
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?

fsck/xfs_fsck.sh fi · line 79

Quoted 10 lines
fsck/xfs_fsck.sh +10 -0
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o

${x#rootflags=}"

fsck/xfs_fsck.sh fi

Quoted 6 lines
fsck/xfs_fsck.sh +6 -0
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit

1

Quoted 12 lines
fsck/xfs_fsck.sh +9 -0
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

--
1.8.3.1

--
Carlos Maiolino

--
Carlos Maiolino

Regards,
Srikanth

#7 lore raw

RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 40 lines

-----Original Message-----
From: Darrick J. Wong <[email protected]>
Sent: 29 November 2022 04:34 AM
To: Srikanth C S <[email protected]>
Cc: Carlos Maiolino <[email protected]>; [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 18 lines · Srikanth C S
Quoted 18 lines

-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 23 November 2022 05:53 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>;
Junxiao Bi <[email protected]>; [email protected]
Subject: Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 3 lines · Srikanth C S

Hi

I resent the same patch as I did not see any review comments.

Quoted 8 lines
Quoted 8 lines

Unless I'm looking at the wrong patch, there were comments on your previous
submission:

https://urldefense.com/v3/__https://lore.kernel.org/linux-
xfs/Y2ie54fcHDx5bcG4@B-P7TQMD6M-
0146.local/T/*t__;Iw!!ACWV5N9M2RV99hQ!J2Z-

Quoted 16 lines

2NThyyDm__z9ivhioF9QoHsaHh4Tk733jtNbVMPGeA2vbmbw3h4ZGxOywQF

Quoted 3 lines
Quoted 3 lines

v_lA1Zs_jsUgr$

Am I missing something?

Err.... whose comments, Joseph's or Gao's?

Quoted 8 lines

All the previous comments addressing this patch were about having journal replay code in the userspace. But Darricks comments indicate that this requires making the log endian safe because of kernel's inability to recover a log from a platform with a different endianness.

So I am still wondering on how to proceed with this patch. Any comments would be helpful.

@Carlos Maiolino, Any comments or thoughts on this patch?

-Srikanth

Quoted 131 lines

Same here, though the long holiday weekend probably didn't help.

--D

Quoted 3 lines
Quoted 3 lines

Also, if you are sending the same patch, you can 'flag' it as a resend, so, it's easier to identify you are simply resending the same patch. You can do it by appending/prepending 'RESEND', to the

patch tag:

Quoted 11 lines
Quoted 2 lines

[RESEND PATCH] <subject>

Thanks for the info. Didn't know this.

Quoted 8 lines

Cheers.

Quoted 4 lines

-Srikanth

__________________________________________________________

Quoted 6 lines
Quoted 6 lines

________

Quoted 5 lines

From: Carlos Maiolino <[email protected]>
Sent: Wednesday, November 23, 2022 2:06 PM
To: Srikanth C S <[email protected]>
Cc: [email protected] <[email protected]>; Darrick

Wong

Quoted 20 lines
Quoted 20 lines
Quoted 3 lines

<[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]

<[email protected]>

Quoted 16 lines

Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Hi.
Did you plan to resend V3 again, or is this supposed to be V4?

Quoted 10 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the

machine

Quoted 1 line

into the rescue shell because xfs_fsck.sh does not know how to

clean

Quoted 78 lines
Quoted 68 lines
Quoted 68 lines
   the
   > log. Since the administrator told us to force repairs, address the
   > deficiency by cleaning the log and rerunning xfs_repair.
   >
   > Run xfs_repair -e when fsck.mode=force and repair=auto or yes.
   > Replay the logs only if fsck.mode=force and fsck.repair=yes. For
   > other option -fa and -f drop to the rescue shell if repair detects
   > any corruptions.
   >
   > Signed-off-by: Srikanth C S <[email protected]>
   > ---
   >  fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
   >  1 file changed, 29 insertions(+), 2 deletions(-)
   >
   > diff --git a/fsck/xfs_fsck.sh b/fsck/xfs_fsck.sh
   > index 6af0f22..62a1e0b 100755
   > --- a/fsck/xfs_fsck.sh
   > +++ b/fsck/xfs_fsck.sh
   > @@ -31,10 +31,12 @@ repair2fsck_code() {
   >
   >  AUTO=false
   >  FORCE=false
   > +REPAIR=false
   >  while getopts ":aApyf" c
   >  do
   >         case $c in
   > -       a|A|p|y)        AUTO=true;;
   > +       a|A|p)          AUTO=true;;
   > +       y)              REPAIR=true;;
   >         f)              FORCE=true;;
   >         esac
   >  done
   > @@ -64,7 +66,32 @@ fi
   >
   >  if $FORCE; then
   >         xfs_repair -e $DEV
   > -       repair2fsck_code $?
   > +       error=$?
   > +       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
   > +               echo "Replaying log for $DEV"
   > +               mkdir -p /tmp/repair_mnt || exit 1
   > +               for x in $(cat /proc/cmdline); do
   > +                       case $x in
   > +                               root=*)
   > +                                       ROOT="${x#root=}"
   > +                               ;;
   > +                               rootflags=*)
   > +                                       ROOTFLAGS="-o
   ${x#rootflags=}"
   > +                               ;;
   > +                       esac
   > +               done
   > +               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
   > +               if [ $(basename $DEV) = $(basename $ROOT) ]; then
   > +                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit
   1
   > +               else
   > +                       mount $DEV /tmp/repair_mnt || exit 1
   > +               fi
   > +               umount /tmp/repair_mnt
   > +               xfs_repair -e $DEV
   > +               error=$?
   > +               rm -d /tmp/repair_mnt
   > +       fi
   > +       repair2fsck_code $error
   >         exit $?
   >  fi
   >
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

--
1.8.3.1

--
Carlos Maiolino

--
Carlos Maiolino

Regards,
Srikanth

#8 lore raw

Re: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

So I am still wondering on how to proceed with this patch. Any comments would be helpful.

@Carlos Maiolino, Any comments or thoughts on this patch?

Sorry, didn't have time to look into it yet.

Signature · 1 line
Carlos Maiolino
Reviewed-by on v3 1/1
#9 lore raw
In reply to Srikanth C S
Quoted 17 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Quoted 6 lines

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

Apologies it took so long, the patch seems fine to me. Will test.

Quoted 58 lines
---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
fsck/xfs_fsck.sh +29 -3
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
--

1.8.3.1

Signature · 1 line
Carlos Maiolino
#10 lore raw
In reply to Srikanth C S

Hi Srikanth.

Quoted 22 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

Quoted 4 lines
---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

Did you by any chance wrote this patch on top of something else you have in your tree?

It doesn't apply to the tree without tweaking it, and the last changes we've in the fsck/xfs_fsck.sh file are from 2018, so I assume you have something before this patch in your tree.

Could you please rebase this patch against xfsprogs for-next and resend it? Feel free to keep my RwB as long as you don't change the code semantics.

Cheers.

Quoted 55 lines
fsck/xfs_fsck.sh +29 -3
index 6af0f22..62a1e0b 100755
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
--

1.8.3.1

Signature · 1 line
Carlos Maiolino
#11 lore raw

RE: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Quoted 41 lines

-----Original Message-----
From: Carlos Maiolino <[email protected]>
Sent: 13 December 2022 03:10 PM
To: Srikanth C S <[email protected]>
Cc: [email protected]; Darrick Wong <[email protected]>; Rajesh Sivaramasubramaniom <[email protected]>; Junxiao Bi <[email protected]>; [email protected]
Subject: [External] : Re: [PATCH v3] fsck.xfs: mount/umount xfs fs to replay log before running xfs_repair

Hi Srikanth.

Quoted 21 lines · Srikanth C S

After a recent data center crash, we had to recover root filesystems on several thousands of VMs via a boot time fsck. Since these machines are remotely manageable, support can inject the kernel command line with 'fsck.mode=force fsck.repair=yes' to kick off xfs_repair if the machine won't come up or if they suspect there might be deeper issues with latent errors in the fs metadata, which is what they did to try to get everyone running ASAP while anticipating any future problems. But, fsck.xfs does not address the journal replay in case of a crash.

fsck.xfs does xfs_repair -e if fsck.mode=force is set. It is possible that when the machine crashes, the fs is in inconsistent state with the journal log not yet replayed. This can drop the machine into the rescue shell because xfs_fsck.sh does not know how to clean the log. Since the administrator told us to force repairs, address the deficiency by cleaning the log and rerunning xfs_repair.

Run xfs_repair -e when fsck.mode=force and repair=auto or yes. Replay the logs only if fsck.mode=force and fsck.repair=yes. For other option -fa and -f drop to the rescue shell if repair detects any corruptions.

Quoted 4 lines
---
 fsck/xfs_fsck.sh | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
Quoted 7 lines

Did you by any chance wrote this patch on top of something else you have in your tree?

It doesn't apply to the tree without tweaking it, and the last changes we've in the fsck/xfs_fsck.sh file are from 2018, so I assume you have something before this patch in your tree.

Sorry for the inconvenience, will verify this.

Quoted 3 lines

Could you please rebase this patch against xfsprogs for-next and resend it? Feel free to keep my RwB as long as you don't change the code semantics.

Let me rebase the patch and resend it. Thanks for the Reviewed by.

Quoted 55 lines

Cheers.

Quoted 53 lines
fsck/xfs_fsck.sh index renamed from fsck/xfs_fsck.sh +0 -0

6af0f22..62a1e0b 100755

fsck/xfs_fsck.sh +29 -2
@@ -31,10 +31,12 @@ repair2fsck_code() {
 
 AUTO=false
 FORCE=false
+REPAIR=false
 while getopts ":aApyf" c
 do
        case $c in
-       a|A|p|y)        AUTO=true;;
+       a|A|p)          AUTO=true;;
+       y)              REPAIR=true;;
        f)              FORCE=true;;
        esac
 done
@@ -64,7 +66,32 @@ fi
 
 if $FORCE; then
        xfs_repair -e $DEV
-       repair2fsck_code $?
+       error=$?
+       if [ $error -eq 2 ] && [ $REPAIR = true ]; then
+               echo "Replaying log for $DEV"
+               mkdir -p /tmp/repair_mnt || exit 1
+               for x in $(cat /proc/cmdline); do
+                       case $x in
+                               root=*)
+                                       ROOT="${x#root=}"
+                               ;;
+                               rootflags=*)
+                                       ROOTFLAGS="-o ${x#rootflags=}"
+                               ;;
+                       esac
+               done
+               test -b "$ROOT" || ROOT=$(blkid -t "$ROOT" -o device)
+               if [ $(basename $DEV) = $(basename $ROOT) ]; then
+                       mount $DEV /tmp/repair_mnt $ROOTFLAGS || exit 1
+               else
+                       mount $DEV /tmp/repair_mnt || exit 1
+               fi
+               umount /tmp/repair_mnt
+               xfs_repair -e $DEV
+               error=$?
+               rm -d /tmp/repair_mnt
+       fi
+       repair2fsck_code $error
        exit $?
 fi
 
Quoted 5 lines
Quoted 2 lines

--
1.8.3.1

--
Carlos Maiolino

Thanks,
Srikanth