xfs: check sb_meta_uuid for dabuf buffer recovery
PATCH 1/1 xfs: check sb_meta_uuid for dabuf buffer recovery
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected at xfs_dir3_leaf_check_int+0x99/0xf0, xfs_dir3_leaf1 block 0x12faa158 XFS (loop0): Unmount and run xfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:
00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....
XFS (loop0): Corruption of in-memory data (0x8) detected at xfs_do_force_shutdown+0x1a/0x20 (fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failed
Tracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.
The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the sb_meta_uuid, not sb_uuid. This filesystem has sb_uuid != sb_meta_uuid (which is fine), and the buffer has the correct matching sb_meta_uuid in it, it's just the code checked it against the wrong superblock uuid.
The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
This was missed back in 2015 during an audit of incorrect sb_uuid usage that resulted in commit fcfbe2c4ef42 ("xfs: log recovery needs to validate against sb_meta_uuid") that fixed the magic32 buffers to validate against sb_meta_uuid instead of sb_uuid. It missed the magicda buffers....
--- fs/xfs/xfs_buf_item_recover.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/xfs/xfs_buf_item_recover.c +1 -1
@@ -816,7 +816,7 @@ xlog_recover_get_buf_lsn( }if (lsn != (xfs_lsn_t)-1 ) { -if (!uuid_equal (&mp->m_sb.sb_uuid, uuid)) +if (!uuid_equal (&mp->m_sb.sb_meta_uuid, uuid)) goto recover_immediately;return lsn; }
Signature · 1 line
2.33.0
Quoted 47 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failedTracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
Quoted 8 lines
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>
Kind of amazing it took so long to turn up and get found, eh?
Thanks Dave,
Quoted 17 lines
--- fs/xfs/xfs_buf_item_recover.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/xfs/xfs_buf_item_recover.c +1
-1@@ -816,7 +816,7 @@ xlog_recover_get_buf_lsn( }if (lsn != (xfs_lsn_t)-1 ) { -if (!uuid_equal (&mp->m_sb.sb_uuid, uuid)) +if (!uuid_equal (&mp->m_sb.sb_meta_uuid, uuid)) goto recover_immediately;return lsn; }
Quoted 54 lines · Eric Sandeen
Quoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failedTracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....
Quoted 4 lines
Quoted 2 lines
Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>Kind of amazing it took so long to turn up and get found, eh?
Especially considering that I use sb_uuid != sb_meta_uuid for all the test device filesystems I run fstests on and have for the past 5 years. I think that means we *never* run log recovery tests on test devices in fstests.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 25 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failed
Quoted 6 lines
Tracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the
^^^^^^^^^^^^^^^^^^^^^^^^^^
xlog_recover_get_buf_lsn()
Quoted 45 lines
buffer.
The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_buf_item_recover.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/xfs/xfs_buf_item_recover.c +1
-1@@ -816,7 +816,7 @@ xlog_recover_get_buf_lsn( }if (lsn != (xfs_lsn_t)-1 ) { -if (!uuid_equal (&mp->m_sb.sb_uuid, uuid)) +if (!uuid_equal (&mp->m_sb.sb_meta_uuid, uuid)) goto recover_immediately;return lsn; }Signature · 1 line
2.33.0
Signature · 2 lines
Dave Chinner [email protected]
Quoted 47 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failedTracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
Quoted 8 lines
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>
Makes sense. Please send a slimmed down version of this to fstests@.
--D
Quoted 20 lines
--- fs/xfs/xfs_buf_item_recover.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/xfs/xfs_buf_item_recover.c +1
-1@@ -816,7 +816,7 @@ xlog_recover_get_buf_lsn( }if (lsn != (xfs_lsn_t)-1 ) { -if (!uuid_equal (&mp->m_sb.sb_uuid, uuid)) +if (!uuid_equal (&mp->m_sb.sb_meta_uuid, uuid)) goto recover_immediately;return lsn; }Signature · 1 line
2.33.0
Quoted 54 lines · Darrick J. Wong
Quoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failedTracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....
Quoted 4 lines
Quoted 2 lines
Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>Makes sense. Please send a slimmed down version of this to fstests@.
<sigh> "...of a reproducer for this to fstests@."
--D
Quoted 24 lines
Reviewed-by Darrick J. Wong <[email protected]>--D
Quoted 20 lines
--- fs/xfs/xfs_buf_item_recover.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/xfs/xfs_buf_item_recover.c +1
-1@@ -816,7 +816,7 @@ xlog_recover_get_buf_lsn( }if (lsn != (xfs_lsn_t)-1 ) { -if (!uuid_equal (&mp->m_sb.sb_uuid, uuid)) +if (!uuid_equal (&mp->m_sb.sb_meta_uuid, uuid)) goto recover_immediately;return lsn; }Signature · 1 line
2.33.0
Quoted 55 lines · Darrick J. Wong
Quoted 54 lines · Darrick J. Wong
Quoted 53 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Got a report that a repeated crash test of a container host would eventually fail with a log recovery error preventing the system from mounting the root filesystem. It manifested as a directory leaf node corruption on writeback like so:
XFS (loop0): Mounting V5 Filesystem
XFS (loop0): Starting recovery (logdev: internal)
XFS (loop0): Metadata corruption detected atxfs_dir3_leaf_check_int+0x99/0xf0,xfs_dir3_leaf1block0x12faa158XFS (loop0): Unmount and runxfs_repair
XFS (loop0): First 128 bytes of corrupted metadata buffer:00000000: 00 00 00 00 00 00 00 00 3d f1 00 00 e1 9e d5 8b ........=....... 00000010: 00 00 00 00 12 fa a1 58 00 00 00 29 00 00 1b cc .......X...).... 00000020: 91 06 78 ff f7 7e 4a 7d 8d 53 86 f2 ac 47 a8 23 ..x..~J}.S...G.# 00000030: 00 00 00 00 17 e0 00 80 00 43 00 00 00 00 00 00 .........C...... 00000040: 00 00 00 2e 00 00 00 08 00 00 17 2e 00 00 00 0a ................ 00000050: 02 35 79 83 00 00 00 30 04 d3 b4 80 00 00 01 50 .5y....0.......P 00000060: 08 40 95 7f 00 00 02 98 08 41 fe b7 00 00 02 d4 [email protected]...... 00000070: 0d 62 ef a7 00 00 01 f2 14 50 21 41 00 00 00 0c .b.......P!A....XFS (loop0): Corruption of in-memory data (
0x8) detected atxfs_do_force_shutdown+0x1a/0x20(fs/xfs/xfs_buf.c:1514). Shutting down. XFS (loop0): Please unmount the filesystem and rectify the problem(s)
XFS (loop0): log mount/recovery failed: error -117
XFS (loop0): log mount failedTracing indicated that we were recovering changes from a transaction at LSN 0x29/0x1c16 into a buffer that had an LSN of 0x29/0x1d57. That is, log recovery was overwriting a buffer with newer changes on disk than was in the transaction. Tracing indicated that we were hitting the "recovery immediately" case in
xfs_buf_log_recovery_lsn(), and hence it was ignoring the LSN in the buffer.The code was extracting the LSN correctly, then ignoring it because the UUID in the buffer did not match the superblock UUID. The problem arises because the UUID check uses the wrong UUID - it should be checking the
sb_meta_uuid, notsb_uuid. This filesystem hassb_uuid!=sb_meta_uuid(which is fine), and the buffer has the correct matchingsb_meta_uuidin it, it's just the code checked it against the wrong superblock uuid.The is no corruption in the filesystem, and failing to recover the buffer due to a write verifier failure means the recovery bug did not propagate the corruption to disk. Hence there is no corruption before or after this bug has manifested, the impact is limited simply to an unmountable filesystem....
This was missed back in 2015 during an audit of incorrect
sb_uuidusage that resulted in commitfcfbe2c4ef42("xfs: log recovery needs to validate againstsb_meta_uuid") that fixed the magic32 buffers to validate againstsb_meta_uuidinstead ofsb_uuid. It missed the magicda buffers....
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Fixes ce748eaa65f2 ("xfs: create new metadata UUID field and incompat flag")Signed-off-by Dave Chinner <[email protected]>Makes sense. Please send a slimmed down version of this to fstests@.
<sigh> "...of a reproducer for this to fstests@."
I don't have one. All I have a pre-recovery metadump that has the issue already in it, and I traced the mount and looked at log print output to find the issue.
Eric, it seems like getting generic coverage of sb_meta_uuid != sb_uuid for at least the recoveryloop group tests would be a goof thing for the QE team to add to fstests? Alternatively, randomly decide whether to set a different UUID during each scratch_mkfs call?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Looks good,