xfs: fix recovery corruption on s390 w/ nrext64
cover xfs: fix recovery corruption on s390 w/ nrext64
Hi folks,
The following are two patches to fix the corruption bug that Zorro recently recently reported on big endian platforms when nrext64 is enabled.
It is caused by recovery zeroing a v2 inode field even on v3 inodes. Before nrext64, this wasn't an issue because the field in v3 inodes was just padding (i.e. zeroes) and so writing zero to part of it was never noticed. However, nrext64 uses this field for the 64 bit data fork extent count, so zeroing part of it becomes a problem.
The zeroing is done to the xfs_log_dinode, which is held in host format. Because of the layout of this field, on little endian machines the di_flush_iter field overlaps the high 16 bits of di_big_nextents, and so we didn't notice that recovery was overwriting the the high bits of the extent count because it's always going to be zero bytes (needs to go over 2^48 extents before we'd notice) in testing contexts.
However, on big endian machines, the di_flushiter field overlays the lower 16 bits of the di_big_extents field, and so zeroing di_flushiter essentially rounds it down to the nearest 64k. For typical testing contexts, that's effectively zeroing the extent count.
The fix is two parts. The first patch adds detection of this specific corruption to the xfs_dinode verifier, and adds a call to the verifier after recovery has recovered the inode. This catches corruptions at recovery time, rather than silently writing them back and not noticing them until the inode is re-read from disk at runtime.
The second patch fixes the recovery bug. It simply moves the resetting of the di_flushiter field into the v2 inode specific flushiter handling branch. That's always been wrong (i.e. since v3 inodes were introduced) but it hasn't mattered until now, so it marked as fixing the introduction of the nrext64 feature....
-Dave.
PATCH 1/2 xfs: inode recovery does not validate the recovered inode
From: Dave Chinner <[email protected]>
Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.
The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.
--- fs/xfs/libxfs/xfs_inode_buf.c | 3 +++ fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-)
fs/xfs/libxfs/xfs_inode_buf.c +3 -0
@@ -508,6 +508,9 @@ xfs_dinode_verify(if (mode && nextents + naextents > nblocks)return __this_address; +if (nextents + naextents ==0 && nblocks !=0 ) +return __this_address; +if (S_ISDIR (mode) && nextents > mp->m_dir_geo->max_extents)return __this_address;
fs/xfs/xfs_inode_item_recover.c +13 -1
@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(struct xfs_log_dinode *ldip;uint isize;int need_free =0 ; +xfs_failaddr_t fa;if (item->ri_buf[0 ].i_len ==sizeof (struct xfs_inode_log_format )) { in_f = item->ri_buf[0 ].i_addr;
@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2( (dip->di_mode != 0)) error =xfs_recover_inode_owner_change (mp, dip, in_f, buffer_list); -/* re-generate the checksum. */ +/* re-generate the checksum and validate the recovered inode. */ xfs_dinode_calc_crc (log->l_mp, dip); + fa =xfs_dinode_verify (log->l_mp, in_f->ilf_ino, dip); +if (fa) { +XFS_CORRUPTION_ERROR ( +"Bad dinode after recovery" , +XFS_ERRLEVEL_LOW , mp, dip,sizeof (*dip)); +xfs_alert (mp, +"Metadata corruption detected at %pS, inode 0x%llx" , + fa, in_f->ilf_ino); + error = -EFSCORRUPTED ; + goto out_release; + }ASSERT (bp->b_mount == mp); bp->b_flags |= _XBF_LOGRECOVERY;
Signature · 1 line
2.42.0
Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode
Quoted 42 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.
The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/libxfs/xfs_inode_buf.c | 3 +++ fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-)fs/xfs/libxfs/xfs_inode_buf.c +3
-0@@ -508,6 +508,9 @@ xfs_dinode_verify(if (mode && nextents + naextents > nblocks)return __this_address; +if (nextents + naextents ==0 && nblocks !=0 ) +return __this_address; +if (S_ISDIR (mode) && nextents > mp->m_dir_geo->max_extents)return __this_address;fs/xfs/xfs_inode_item_recover.c +1
-0@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(struct xfs_log_dinode *ldip;uint isize;int need_free =0 ; +xfs_failaddr_t fa;if (item->ri_buf[0 ].i_len ==sizeof (struct xfs_inode_log_format )) { in_f = item->ri_buf[0 ].i_addr;
fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536
Quoted 8 lines
fs/xfs/xfs_inode_item_recover.c +3
-1@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2( (dip->di_mode != 0)) error =xfs_recover_inode_owner_change (mp, dip, in_f, buffer_list); -/* re-generate the checksum. */ +/* re-generate the checksum and validate the recovered inode. */ xfs_dinode_calc_crc (log->l_mp, dip); + fa =xfs_dinode_verify (log->l_mp, in_f->ilf_ino, dip); +if (fa) {
Does xlog_recover_dquot_commit_pass2 need to call xfs_dquot_verify as well?
This patch looks good though,
--D
Quoted 15 lines
fs/xfs/xfs_inode_item_recover.c +9
-0+XFS_CORRUPTION_ERROR ( +"Bad dinode after recovery" , +XFS_ERRLEVEL_LOW , mp, dip,sizeof (*dip)); +xfs_alert (mp, +"Metadata corruption detected at %pS, inode 0x%llx" , + fa, in_f->ilf_ino); + error = -EFSCORRUPTED ; + goto out_release; + }ASSERT (bp->b_mount == mp); bp->b_flags |= _XBF_LOGRECOVERY;Signature · 1 line
2.42.0
Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode
Quoted 40 lines · Darrick J. Wong
Quoted 39 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.
The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/libxfs/xfs_inode_buf.c | 3 +++ fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-)fs/xfs/libxfs/xfs_inode_buf.c +3
-0@@ -508,6 +508,9 @@ xfs_dinode_verify(if (mode && nextents + naextents > nblocks)return __this_address; +if (nextents + naextents ==0 && nblocks !=0 ) +return __this_address; +if (S_ISDIR (mode) && nextents > mp->m_dir_geo->max_extents)return __this_address;fs/xfs/xfs_inode_item_recover.c +1
-0@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(struct xfs_log_dinode *ldip;uint isize;int need_free =0 ; +xfs_failaddr_t fa;
Quoted 14 lines
fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536
Quoted 11 lines
fs/xfs/xfs_inode_item_recover.c +3
-1if (item->ri_buf[0 ].i_len ==sizeof (struct xfs_inode_log_format )) { in_f = item->ri_buf[0 ].i_addr;@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2( (dip->di_mode != 0)) error =xfs_recover_inode_owner_change (mp, dip, in_f, buffer_list); -/* re-generate the checksum. */ +/* re-generate the checksum and validate the recovered inode. */ xfs_dinode_calc_crc (log->l_mp, dip); + fa =xfs_dinode_verify (log->l_mp, in_f->ilf_ino, dip); +if (fa) {Does
xlog_recover_dquot_commit_pass2need to callxfs_dquot_verifyas well?
Maybe - I haven't looked closely at that, and it depends what the dquot buffer verifier does. If it's similar to the inode cluster buffer verifier (i.e. only checks for dquots, doesn't verify the dquots) then it should do the same thing. I don't have time to do this right now because I'm OOO for the next week, so maybe you could check this and send a patch for it?
Quoted 2 lines
This patch looks good though,
Reviewed-by Darrick J. Wong <[email protected]>
Thanks!
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 1/2] xfs: inode recovery does not validate the recovered inode
Quoted 56 lines · Dave Chinner
Quoted 40 lines · Darrick J. Wong
Quoted 39 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Discovered when trying to track down a weird recovery corruption issue that wasn't detected at recovery time.
The specific corruption was a zero extent count field when big extent counts are in use, and it turns out the dinode verifier doesn't detect that specific corruption case, either. So fix it too.
Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/libxfs/xfs_inode_buf.c | 3 +++ fs/xfs/xfs_inode_item_recover.c | 14 +++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-)fs/xfs/libxfs/xfs_inode_buf.c +3
-0@@ -508,6 +508,9 @@ xfs_dinode_verify(if (mode && nextents + naextents > nblocks)return __this_address; +if (nextents + naextents ==0 && nblocks !=0 ) +return __this_address; +if (S_ISDIR (mode) && nextents > mp->m_dir_geo->max_extents)return __this_address;fs/xfs/xfs_inode_item_recover.c +1
-0@@ -286,6 +286,7 @@ xlog_recover_inode_commit_pass2(struct xfs_log_dinode *ldip;uint isize;int need_free =0 ; +xfs_failaddr_t fa;Quoted 14 lines
fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 536
Quoted 11 lines
fs/xfs/xfs_inode_item_recover.c +3
-1if (item->ri_buf[0 ].i_len ==sizeof (struct xfs_inode_log_format )) { in_f = item->ri_buf[0 ].i_addr;@@ -529,8 +530,19 @@ xlog_recover_inode_commit_pass2( (dip->di_mode != 0)) error =xfs_recover_inode_owner_change (mp, dip, in_f, buffer_list); -/* re-generate the checksum. */ +/* re-generate the checksum and validate the recovered inode. */ xfs_dinode_calc_crc (log->l_mp, dip); + fa =xfs_dinode_verify (log->l_mp, in_f->ilf_ino, dip); +if (fa) {Does
xlog_recover_dquot_commit_pass2need to callxfs_dquot_verifyas well?
Quoted 6 lines
Maybe - I haven't looked closely at that, and it depends what the dquot buffer verifier does. If it's similar to the inode cluster buffer verifier (i.e. only checks for dquots, doesn't verify the dquots) then it should do the same thing. I don't have time to do this right now because I'm OOO for the next week, so maybe you could check this and send a patch for it?
I tossed on a patch to do this and after a couple of days of generic/388 and generic/475 spinning I haven't noticed any failures.
--D
Quoted 9 lines
Quoted 2 lines
This patch looks good though,
Reviewed-by Darrick J. Wong <[email protected]>Thanks!
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 2/2 xfs: recovery should not clear di_flushiter unconditionally
In reply to Dave ChinnerFrom: Dave Chinner <[email protected]>
Because on v3 inodes, di_flushiter doesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.
This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.
This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing di_flushiter on v2 inodes during recovery.
--- fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-)
fs/xfs/xfs_inode_item_recover.c +17 -15
@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(* superblock flag to determine whether we need to look at di_flushiter * to skip replay when the on disk inode is newer than the log one */ -if (!xfs_has_v3inodes (mp) && - ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { -/* -* Deal with the wrap case, DI_MAX_FLUSH is less -* than smaller numbers -*/ -if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && - ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) { -/* do nothing */ - }else { -trace_xfs_log_recover_inode_skip (log, in_f); - error =0 ; - goto out_release; +if (!xfs_has_v3inodes (mp)) { +if (ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { +/* +* Deal with the wrap case, DI_MAX_FLUSH is less +* than smaller numbers +*/ +if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && + ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) { +/* do nothing */ + }else { +trace_xfs_log_recover_inode_skip (log, in_f); + error =0 ; + goto out_release; + } } + +/* Take the opportunity to reset the flush iteration count */ + ldip->di_flushiter =0 ; } -/* Take the opportunity to reset the flush iteration count */ - ldip->di_flushiter =0 ;if (unlikely (S_ISREG (ldip->di_mode))) { if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&
Signature · 1 line
2.42.0
Re: [PATCH 2/2] xfs: recovery should not clear di_flushiter unconditionally
Quoted 55 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Because on v3 inodes,
di_flushiterdoesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.
This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing
di_flushiteron v2 inodes during recovery.Fixes 9b7d16e34bbe ("xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers")cc [email protected] # 5.19+Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-)fs/xfs/xfs_inode_item_recover.c +8
-13@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(* superblock flag to determine whether we need to look at di_flushiter * to skip replay when the on disk inode is newer than the log one */ -if (!xfs_has_v3inodes (mp) && - ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { -/* -* Deal with the wrap case, DI_MAX_FLUSH is less -* than smaller numbers -*/ -if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && - ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) { -/* do nothing */ - }else { -trace_xfs_log_recover_inode_skip (log, in_f); - error =0 ; - goto out_release; +if (!xfs_has_v3inodes (mp)) { +if (ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { +/* +* Deal with the wrap case, DI_MAX_FLUSH is less +* than smaller numbers +*/ +if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && + ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) {
fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 391
Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +9
-0+/* do nothing */ + }else { +trace_xfs_log_recover_inode_skip (log, in_f); + error =0 ; + goto out_release; + } } + +/* Take the opportunity to reset the flush iteration count */ + ldip->di_flushiter =0 ;
Hmm. Well this fixes the zeroday problem, so thank you for getting the root of this!
Though hch did suggest reducing the amount of indenting here by compressing the if tests together. I can't decide if it's worth rearranging that old V4 code since none of it's scheduled for removal until 2030, but it /is/ legacy code that maybe we just don't care to touch?
<shrug>
--D
Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +0
-2} -/* Take the opportunity to reset the flush iteration count */ - ldip->di_flushiter =0 ;if (unlikely (S_ISREG (ldip->di_mode))) { if ((ldip->di_format != XFS_DINODE_FMT_EXTENTS) &&Signature · 1 line
2.42.0
Re: [PATCH 2/2] xfs: recovery should not clear di_flushiter unconditionally
Quoted 70 lines · Darrick J. Wong
Quoted 55 lines · Dave Chinner
From: Dave Chinner <[email protected]>
Because on v3 inodes,
di_flushiterdoesn't exist. It overlaps with zero padding in the inode, except when NREXT64=1 configurations are in use and the zero padding is no longer padding but holds the 64 bit extent counter.This manifests obviously on big endian platforms (e.g. s390) because the log dinode is in host order and the overlap is the LSBs of the extent count field. It is not noticed on little endian machines because the overlap is at the MSB end of the extent count field and we need to get more than 2^^48 extents in the inode before it manifests. i.e. the heat death of the universe will occur before we see the problem in little endian machines.
This is a zero-day issue for NREXT64=1 configuraitons on big endian machines. Fix it by only clearing
di_flushiteron v2 inodes during recovery.Fixes 9b7d16e34bbe ("xfs: Introduce XFS_DIFLAG2_NREXT64 and associated helpers")cc [email protected] # 5.19+Signed-off-by Dave Chinner <[email protected]>--- fs/xfs/xfs_inode_item_recover.c | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-)fs/xfs/xfs_inode_item_recover.c +8
-13@@ -371,24 +371,26 @@ xlog_recover_inode_commit_pass2(* superblock flag to determine whether we need to look at di_flushiter * to skip replay when the on disk inode is newer than the log one */ -if (!xfs_has_v3inodes (mp) && - ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { -/* -* Deal with the wrap case, DI_MAX_FLUSH is less -* than smaller numbers -*/ -if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && - ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) { -/* do nothing */ - }else { -trace_xfs_log_recover_inode_skip (log, in_f); - error =0 ; - goto out_release; +if (!xfs_has_v3inodes (mp)) { +if (ldip->di_flushiter <be16_to_cpu (dip->di_flushiter)) { +/* +* Deal with the wrap case, DI_MAX_FLUSH is less +* than smaller numbers +*/ +if (be16_to_cpu (dip->di_flushiter) ==DI_MAX_FLUSH && + ldip->di_flushiter < (DI_MAX_FLUSH >>1 )) {fs/xfs/xfs_inode_item_recover.c xlog_recover_inode_commit_pass2( · line 391
Quoted 10 lines
fs/xfs/xfs_inode_item_recover.c +9
-0+/* do nothing */ + }else { +trace_xfs_log_recover_inode_skip (log, in_f); + error =0 ; + goto out_release; + } } + +/* Take the opportunity to reset the flush iteration count */ + ldip->di_flushiter =0 ;Hmm. Well this fixes the zeroday problem, so thank you for getting the root of this!
Quoted 7 lines
Reviewed-by Darrick J. Wong <[email protected]>Though hch did suggest reducing the amount of indenting here by compressing the if tests together. I can't decide if it's worth rearranging that old V4 code since none of it's scheduled for removal until 2030, but it /is/ legacy code that maybe we just don't care to touch?
I'd prefer not to touch it. It's now all isolated in a "!xfs_has_v3inodes()" branch, so I think we can largely ignore the grot for now. If we have to do a larger rework of this code in future, then we can look at reworking it.
But right now, I really don't feel like risking breaking something else by doing unnecessary cleanups on code we haven't needed to touch in over a decade.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]