New: [xfstests generic/447] xfs_scrub always complains fs corruption
https://bugzilla.kernel.org/show_bug.cgi?id=216486
Bug ID: 216486
Summary: [xfstests generic/447] xfs_scrub always complains fs
corruption
Product: File System
Version: 2.5
Kernel Version: 6.0.0-rc4+
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: XFS
Assignee: [email protected]
Reporter: [email protected]
Regression: NoRecently xfstests generic/447 always fails[1][2][3] on latest xfs kernel with xfsprogs. It's reproducible on 1k blocksize and rmapbt enabled XFS (-b size=1024 -m rmapbt=1). Not sure if it's a kernel bug or a xfsprogs issue, or an expected failure.
[1]
SECTION -- default FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 hp-xxxxxxxx-01
6.0.0-0.rc4.20220906git53e99dcff61e.32.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Sep 7 07:51:49 UTC 2022MKFS_OPTIONS -- -f -b size=1024 -m rmapbt=1 /dev/sda3 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/sda3 /mnt/scratch
generic/447 246s ... _check_xfs_filesystem: filesystem on /dev/sda3 failed scrub
(see /root/git/xfstests/results//default/generic/447.full for details)
[2]
# cat results//default/generic/447.full
meta-data=/dev/sda3 isize=512 agcount=16, agsize=3276544 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=0
data = bsize=1024 blocks=52424704, imaxpct=25
= sunit=256 swidth=256 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=1024 blocks=65536, version=2
= sectsz=512 sunit=256 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
creating 2097152 blocks...
wrote 2147483648/2147483648 bytes at offset 0
2.000 GiB, 512 ops; 0:00:07.59 (269.766 MiB/sec and 67.4414 ops/sec)
Punching file2...
...done
_check_xfs_filesystem: filesystem on /dev/sda3 failed scrub
*** xfs_scrub -v -d -n output ***
EXPERIMENTAL xfs_scrub program in use! Use at your own risk!
Phase 1: Find filesystem geometry.
/mnt/scratch: using 1 threads to scrub.
Phase 2: Check internal metadata.
Corruption: AG 0 reference count btree: Repairs are required. (scrub.c line
196)
Info: AG 1 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 2 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 3 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 4 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 5 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 6 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 7 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 8 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 9 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 10 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 11 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 12 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 13 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 14 superblock: Optimization is possible. (scrub.c line 212)
Info: AG 15 superblock: Optimization is possible. (scrub.c line 212)
Phase 3: Scan all inodes.
Info: inode 512 (0/512) inode record: Cross-referencing failed. (scrub.c line
117)
Info: inode 515 (0/515) inode record: Cross-referencing failed. (scrub.c line
117)
Info: inode 517 (0/517) inode record: Cross-referencing failed. (scrub.c line
117)
Info: inode 517 (0/517) data block map: Cross-referencing failed. (scrub.c line
117)
Info: /mnt/scratch: Optimizations of inode record are possible. (scrub.c line
253)
Phase 5: Check directory tree.
Info: /mnt/scratch: Filesystem has errors, skipping connectivity checks.
(phase5.c line 392)
Phase 7: Check summary counters.
5.2GiB data used; 6 inodes used.
1.1GiB data found; 5 inodes found.
5 inodes counted; 6 inodes checked.
/mnt/scratch: corruptions found: 1
/mnt/scratch: Re-run xfs_scrub without -n.
*** end xfs_scrub output[3]
# dmesg [329558.995550] run fstests generic/447 at 2022-09-13 14:01:24 [329560.019866] systemd[1]: Started fstests-generic-447.scope - /usr/bin/bash -c test -w /proc/self/oom_score_adj && echo 250 > /proc/self/oom_score_adj; exec ./tests/generic/447. [329561.466573] XFS (sda3): Mounting V5 Filesystem [329561.542655] XFS (sda3): Ending clean mount [329561.596681] XFS (sda3): Unmounting Filesystem [329561.598209] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329562.183863] XFS (sda3): Mounting V5 Filesystem [329562.265873] XFS (sda3): Ending clean mount [329727.320231] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329729.160375] XFS (sda3): Unmounting Filesystem [329730.480159] XFS (sda3): Mounting V5 Filesystem [329730.559529] XFS (sda3): Ending clean mount [329730.595342] systemd[1]: fstests-generic-447.scope: Deactivated successfully. [329730.597524] systemd[1]: fstests-generic-447.scope: Consumed 2min 44.321s CPU time. [329730.641904] XFS (sda5): Unmounting Filesystem [329730.644716] systemd[1]: mnt-test.mount: Deactivated successfully. [329730.899455] XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk! [329743.405813] XFS (sda3): Corruption detected during scrub. [329743.922150] XFS (sda3): Corruption detected during scrub. [329744.438304] XFS (sda3): Corruption detected during scrub. [329744.956067] XFS (sda3): Corruption detected during scrub. [329745.472617] XFS (sda3): Corruption detected during scrub. [329745.988849] XFS (sda3): Corruption detected during scrub. [329746.505812] XFS (sda3): Corruption detected during scrub. [329747.022342] XFS (sda3): Corruption detected during scrub. [329747.538927] XFS (sda3): Corruption detected during scrub. [329748.055586] XFS (sda3): Corruption detected during scrub. [329748.572338] XFS (sda3): Corruption detected during scrub. [329911.911869] XFS (sda3): Unmounting Filesystem [329911.913058] XFS (sda3): Uncorrected metadata errors detected; please run xfs_repair. [329911.913588] systemd[1]: mnt-scratch.mount: Deactivated successfully.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Quoted 19 lines · On Wed, Sep 14, 2022 at 08:12:56AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216486
Bug ID: 216486 Summary: [xfstests generic/447] xfs_scrub always complains fs corruption Product: File System Version: 2.5 Kernel Version: 6.0.0-rc4+ Hardware: All OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: XFS Assignee: [email protected] Reporter: [email protected] Regression: No
Quoted 4 lines
Recently xfstests generic/447 always fails[1][2][3] on latest xfs kernel with xfsprogs. It's reproducible on 1k blocksize and rmapbt enabled XFS (-b size=1024 -m rmapbt=1). Not sure if it's a kernel bug or a xfsprogs issue, or an expected failure.
It's an expected failure that is one of the many things fixed by the online fsck patchset. The solution I came up with is described here:
https://djwong.org/docs/xfs-online-fsck-design/#eventual-consistency-vs-online-fsck
The TLDR is that scrub is probably racing with a thread that's in the middle of doing a file mapping change that involves both an rmap and a refcount update. This is possible because we don't hold the AGF buffer between work items in a defer ops chain.
--D
Quoted 120 lines
[1]
SECTION -- default FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 hp-xxxxxxxx-016.0.0-0.rc4.20220906git53e99dcff61e.32.fc38.x86_64 #1 SMP
PREEMPT_DYNAMICWed Sep 7 07:51:49 UTC 2022MKFS_OPTIONS-- -f -b size=1024 -m rmapbt=1/dev/sda3MOUNT_OPTIONS-- -o context=system_u:object_r:root_t:s0/dev/sda3/mnt/scratchgeneric/447 246s ...
_check_xfs_filesystem: filesystem on/dev/sda3failed scrub
(see/root/git/xfstests/results//default/generic/447.fullfor details)[2]
# cat results//default/generic/447.full meta-data=/dev/sda3 isize=512 agcount=16, agsize=3276544 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=0 data = bsize=1024 blocks=52424704, imaxpct=25 = sunit=256 swidth=256 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=1024 blocks=65536, version=2 = sectsz=512 sunit=256 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 creating 2097152 blocks... wrote 2147483648/2147483648 bytes at offset 0 2.000 GiB, 512 ops; 0:00:07.59 (269.766 MiB/sec and 67.4414 ops/sec) Punching file2... ...done _check_xfs_filesystem: filesystem on /dev/sda3 failed scrub *** xfs_scrub -v -d -n output *** EXPERIMENTAL xfs_scrub program in use! Use at your own risk! Phase 1: Find filesystem geometry. /mnt/scratch: using 1 threads to scrub. Phase 2: Check internal metadata. Corruption: AG 0 reference count btree: Repairs are required. (scrub.c line 196) Info: AG 1 superblock: Optimization is possible. (scrub.c line 212) Info: AG 2 superblock: Optimization is possible. (scrub.c line 212) Info: AG 3 superblock: Optimization is possible. (scrub.c line 212) Info: AG 4 superblock: Optimization is possible. (scrub.c line 212) Info: AG 5 superblock: Optimization is possible. (scrub.c line 212) Info: AG 6 superblock: Optimization is possible. (scrub.c line 212) Info: AG 7 superblock: Optimization is possible. (scrub.c line 212) Info: AG 8 superblock: Optimization is possible. (scrub.c line 212) Info: AG 9 superblock: Optimization is possible. (scrub.c line 212) Info: AG 10 superblock: Optimization is possible. (scrub.c line 212) Info: AG 11 superblock: Optimization is possible. (scrub.c line 212) Info: AG 12 superblock: Optimization is possible. (scrub.c line 212) Info: AG 13 superblock: Optimization is possible. (scrub.c line 212) Info: AG 14 superblock: Optimization is possible. (scrub.c line 212) Info: AG 15 superblock: Optimization is possible. (scrub.c line 212) Phase 3: Scan all inodes. Info: inode 512 (0/512) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 515 (0/515) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 517 (0/517) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 517 (0/517) data block map: Cross-referencing failed. (scrub.c line 117) Info: /mnt/scratch: Optimizations of inode record are possible. (scrub.c line 253) Phase 5: Check directory tree. Info: /mnt/scratch: Filesystem has errors, skipping connectivity checks. (phase5.c line 392) Phase 7: Check summary counters. 5.2GiB data used; 6 inodes used. 1.1GiB data found; 5 inodes found. 5 inodes counted; 6 inodes checked. /mnt/scratch: corruptions found: 1 /mnt/scratch: Re-run xfs_scrub without -n. *** end xfs_scrub output[3]
# dmesg [329558.995550] run fstests generic/447 at 2022-09-13 14:01:24 [329560.019866] systemd[1]: Started fstests-generic-447.scope - /usr/bin/bash -c test -w /proc/self/oom_score_adj && echo 250 > /proc/self/oom_score_adj; exec ./tests/generic/447. [329561.466573] XFS (sda3): Mounting V5 Filesystem [329561.542655] XFS (sda3): Ending clean mount [329561.596681] XFS (sda3): Unmounting Filesystem [329561.598209] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329562.183863] XFS (sda3): Mounting V5 Filesystem [329562.265873] XFS (sda3): Ending clean mount [329727.320231] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329729.160375] XFS (sda3): Unmounting Filesystem [329730.480159] XFS (sda3): Mounting V5 Filesystem [329730.559529] XFS (sda3): Ending clean mount [329730.595342] systemd[1]: fstests-generic-447.scope: Deactivated successfully. [329730.597524] systemd[1]: fstests-generic-447.scope: Consumed 2min 44.321s CPU time. [329730.641904] XFS (sda5): Unmounting Filesystem [329730.644716] systemd[1]: mnt-test.mount: Deactivated successfully. [329730.899455] XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk! [329743.405813] XFS (sda3): Corruption detected during scrub. [329743.922150] XFS (sda3): Corruption detected during scrub. [329744.438304] XFS (sda3): Corruption detected during scrub. [329744.956067] XFS (sda3): Corruption detected during scrub. [329745.472617] XFS (sda3): Corruption detected during scrub. [329745.988849] XFS (sda3): Corruption detected during scrub. [329746.505812] XFS (sda3): Corruption detected during scrub. [329747.022342] XFS (sda3): Corruption detected during scrub. [329747.538927] XFS (sda3): Corruption detected during scrub. [329748.055586] XFS (sda3): Corruption detected during scrub. [329748.572338] XFS (sda3): Corruption detected during scrub. [329911.911869] XFS (sda3): Unmounting Filesystem [329911.913058] XFS (sda3): Uncorrected metadata errors detected; please run xfs_repair. [329911.913588] systemd[1]: mnt-scratch.mount: Deactivated successfully.Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216486] [xfstests generic/447] xfs_scrub always complains fs corruption
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216486
--- Comment #1 from Darrick J. Wong ([email protected]) ---
Quoted 19 lines · On Wed, Sep 14, 2022 at 08:12:56AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216486
Bug ID: 216486 Summary: [xfstests generic/447] xfs_scrub always complains fs corruption Product: File System Version: 2.5 Kernel Version: 6.0.0-rc4+ Hardware: All OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: XFS Assignee: [email protected] Reporter: [email protected] Regression: No
Quoted 4 lines
Recently xfstests generic/447 always fails[1][2][3] on latest xfs kernel with xfsprogs. It's reproducible on 1k blocksize and rmapbt enabled XFS (-b size=1024 -m rmapbt=1). Not sure if it's a kernel bug or a xfsprogs issue, or an expected failure.
It's an expected failure that is one of the many things fixed by the online fsck patchset. The solution I came up with is described here:
https://djwong.org/docs/xfs-online-fsck-design/#eventual-consistency-vs-online-fsck
The TLDR is that scrub is probably racing with a thread that's in the middle of doing a file mapping change that involves both an rmap and a refcount update. This is possible because we don't hold the AGF buffer between work items in a defer ops chain.
--D
Quoted 123 lines
[1]
SECTION -- default FSTYP -- xfs (non-debug) PLATFORM -- Linux/x86_64 hp-xxxxxxxx-016.0.0-0.rc4.20220906git53e99dcff61e.32.fc38.x86_64 #1 SMP
PREEMPT_DYNAMICWed Sep 7 07:51:49 UTC 2022MKFS_OPTIONS-- -f -b size=1024 -m rmapbt=1/dev/sda3MOUNT_OPTIONS-- -o context=system_u:object_r:root_t:s0/dev/sda3/mnt/scratchgeneric/447 246s ...
_check_xfs_filesystem: filesystem on/dev/sda3failed scrub
(see/root/git/xfstests/results//default/generic/447.fullfor details)[2]
# cat results//default/generic/447.full meta-data=/dev/sda3 isize=512 agcount=16, agsize=3276544 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=0 data = bsize=1024 blocks=52424704, imaxpct=25 = sunit=256 swidth=256 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=1024 blocks=65536, version=2 = sectsz=512 sunit=256 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 creating 2097152 blocks... wrote 2147483648/2147483648 bytes at offset 0 2.000 GiB, 512 ops; 0:00:07.59 (269.766 MiB/sec and 67.4414 ops/sec) Punching file2... ...done _check_xfs_filesystem: filesystem on /dev/sda3 failed scrub *** xfs_scrub -v -d -n output *** EXPERIMENTAL xfs_scrub program in use! Use at your own risk! Phase 1: Find filesystem geometry. /mnt/scratch: using 1 threads to scrub. Phase 2: Check internal metadata. Corruption: AG 0 reference count btree: Repairs are required. (scrub.c line 196) Info: AG 1 superblock: Optimization is possible. (scrub.c line 212) Info: AG 2 superblock: Optimization is possible. (scrub.c line 212) Info: AG 3 superblock: Optimization is possible. (scrub.c line 212) Info: AG 4 superblock: Optimization is possible. (scrub.c line 212) Info: AG 5 superblock: Optimization is possible. (scrub.c line 212) Info: AG 6 superblock: Optimization is possible. (scrub.c line 212) Info: AG 7 superblock: Optimization is possible. (scrub.c line 212) Info: AG 8 superblock: Optimization is possible. (scrub.c line 212) Info: AG 9 superblock: Optimization is possible. (scrub.c line 212) Info: AG 10 superblock: Optimization is possible. (scrub.c line 212) Info: AG 11 superblock: Optimization is possible. (scrub.c line 212) Info: AG 12 superblock: Optimization is possible. (scrub.c line 212) Info: AG 13 superblock: Optimization is possible. (scrub.c line 212) Info: AG 14 superblock: Optimization is possible. (scrub.c line 212) Info: AG 15 superblock: Optimization is possible. (scrub.c line 212) Phase 3: Scan all inodes. Info: inode 512 (0/512) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 515 (0/515) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 517 (0/517) inode record: Cross-referencing failed. (scrub.c line 117) Info: inode 517 (0/517) data block map: Cross-referencing failed. (scrub.c line 117) Info: /mnt/scratch: Optimizations of inode record are possible. (scrub.c line 253) Phase 5: Check directory tree. Info: /mnt/scratch: Filesystem has errors, skipping connectivity checks. (phase5.c line 392) Phase 7: Check summary counters. 5.2GiB data used; 6 inodes used. 1.1GiB data found; 5 inodes found. 5 inodes counted; 6 inodes checked. /mnt/scratch: corruptions found: 1 /mnt/scratch: Re-run xfs_scrub without -n. *** end xfs_scrub output[3]
# dmesg [329558.995550] run fstests generic/447 at 2022-09-13 14:01:24 [329560.019866] systemd[1]: Started fstests-generic-447.scope - /usr/bin/bash -c test -w /proc/self/oom_score_adj && echo 250 > /proc/self/oom_score_adj; exec ./tests/generic/447. [329561.466573] XFS (sda3): Mounting V5 Filesystem [329561.542655] XFS (sda3): Ending clean mount [329561.596681] XFS (sda3): Unmounting Filesystem [329561.598209] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329562.183863] XFS (sda3): Mounting V5 Filesystem [329562.265873] XFS (sda3): Ending clean mount [329727.320231] systemd[1]: mnt-scratch.mount: Deactivated successfully. [329729.160375] XFS (sda3): Unmounting Filesystem [329730.480159] XFS (sda3): Mounting V5 Filesystem [329730.559529] XFS (sda3): Ending clean mount [329730.595342] systemd[1]: fstests-generic-447.scope: Deactivated successfully. [329730.597524] systemd[1]: fstests-generic-447.scope: Consumed 2min 44.321s CPU time. [329730.641904] XFS (sda5): Unmounting Filesystem [329730.644716] systemd[1]: mnt-test.mount: Deactivated successfully. [329730.899455] XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk! [329743.405813] XFS (sda3): Corruption detected during scrub. [329743.922150] XFS (sda3): Corruption detected during scrub. [329744.438304] XFS (sda3): Corruption detected during scrub. [329744.956067] XFS (sda3): Corruption detected during scrub. [329745.472617] XFS (sda3): Corruption detected during scrub. [329745.988849] XFS (sda3): Corruption detected during scrub. [329746.505812] XFS (sda3): Corruption detected during scrub. [329747.022342] XFS (sda3): Corruption detected during scrub. [329747.538927] XFS (sda3): Corruption detected during scrub. [329748.055586] XFS (sda3): Corruption detected during scrub. [329748.572338] XFS (sda3): Corruption detected during scrub. [329911.911869] XFS (sda3): Unmounting Filesystem [329911.913058] XFS (sda3): Uncorrected metadata errors detected; please run xfs_repair. [329911.913588] systemd[1]: mnt-scratch.mount: Deactivated successfully.Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.
[Bug 216486] [xfstests generic/447] xfs_scrub always complains fs corruption
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=216486
--- Comment #2 from Zorro Lang ([email protected]) --- (In reply to Darrick J. Wong from comment #1)
Quoted 32 lines · On Wed, Sep 14, 2022 at 08:12:56AM +0000, [email protected] wrote:
Quoted 19 lines
https://bugzilla.kernel.org/show_bug.cgi?id=216486
Bug ID: 216486 Summary: [xfstests generic/447] xfs_scrub always complains fs corruption Product: File System Version: 2.5 Kernel Version: 6.0.0-rc4+ Hardware: All OS: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: XFS Assignee: [email protected] Reporter: [email protected] Regression: NoQuoted 1 line
Recently xfstests generic/447 always fails[1][2][3] on latest xfs kernel
with
Quoted 2 lines
xfsprogs. It's reproducible on 1k blocksize and rmapbt enabled XFS (-b size=1024 -m rmapbt=1). Not sure if it's a kernel bug or a xfsprogs issue,
or
Quoted 1 line
an expected failure.
It's an expected failure that is one of the many things fixed by the online fsck patchset. The solution I came up with is described here:
https://djwong.org/docs/xfs-online-fsck-design/#eventual-consistency-vs-
online-fsck
Quoted 4 lines
The TLDR is that scrub is probably racing with a thread that's in the middle of doing a file mapping change that involves both an rmap and a refcount update. This is possible because we don't hold the AGF buffer between work items in a defer ops chain.
Thanks Darrick, for your reply and confirmation!
Signature · 4 lines
You may reply to this email to add a comment. You are receiving this mail because: You are watching the assignee of the bug.