fix h_size validation
Hi all,
this series fixes the slab out of bounds founds by Sam's fuzzer, and tightens up the h_size validation and log recovery buffer allocation while at it.
Diffstat:
xfs_log_recover.c | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-)
PATCH 1/3 xfs: fix log recovery buffer allocation for the legacy h_size fixup
Commit a70f9fe52daa ("xfs: detect and handle invalid iclog size set by mkfs") added a fixup for incorrect h_size values used for the initial umount record in old xfsprogs versions. But it is not using this fixed up value to size the log recovery buffer, which can lead to an out of bounds access when the incorrect h_size does not come from the old mkfs tool, but a fuzzer.
Fix this by open coding xlog_logrec_hblks and taking the fixed h_size into account for this calculation.
--- fs/xfs/xfs_log_recover.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)
fs/xfs/xfs_log_recover.c +14 -6
@@ -2999,7 +2999,7 @@ xlog_do_recovery_pass(int error =0 , h_size, h_len;int error2 =0 ;int bblks, split_bblks; -int hblks, split_hblks, wrapped_hblks; +int hblks =1 , split_hblks, wrapped_hblks;int i;struct hlist_head rhash[XLOG_RHASH_SIZE ];LIST_HEAD (buffer_list);
@@ -3055,14 +3055,22 @@ xlog_do_recovery_pass(if (error) goto bread_err1; - hblks =xlog_logrec_hblks (log, rhead); -if (hblks !=1 ) { -kvfree (hbp); - hbp =xlog_alloc_buffer (log, hblks); +/* +* This open codes xlog_logrec_hblks so that we can reuse the +* fixed up h_size value calculated above. Without that we'd +* still allocate the buffer based on the incorrect on-disk +* size. +*/ +if (h_size >XLOG_HEADER_CYCLE_SIZE && + (rhead->h_version &cpu_to_be32 (XLOG_VERSION_2 ))) { + hblks =DIV_ROUND_UP (h_size,XLOG_HEADER_CYCLE_SIZE ); +if (hblks >1 ) { +kvfree (hbp); + hbp =xlog_alloc_buffer (log, hblks); + } } }else {ASSERT (log->l_sectBBsize ==1 ); - hblks =1 ; hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; }
Signature · 1 line
2.39.2
Re: [PATCH 1/3] xfs: fix log recovery buffer allocation for the legacy h_size fixup
Quoted 7 lines · Christoph Hellwig
Commit
a70f9fe52daa("xfs: detect and handle invalid iclog size set by mkfs") added a fixup for incorrecth_sizevalues used for the initial umount record in old xfsprogs versions. But it is not using this fixed up value to size the log recovery buffer, which can lead to an out of bounds access when the incorrecth_sizedoes not come from the old mkfs tool, but a fuzzer.
Quoted 7 lines
Fix this by open coding
xlog_logrec_hblksand taking the fixedh_sizeinto account for this calculation.Fixes a70f9fe52daa ("xfs: detect and handle invalid iclog size set by mkfs")Reported-by Sam Sun <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>---
The commit log/fixes tag are incorrect... xlog_logrec_hblks() didn't exist at the time a70f9fe52daa was committed. I suspect this broke later in commit 0c771b99d6c9 ("xfs: clean up calculation of LR header blocks"), but please double check.
Otherwise the code changes look fine to me, so with the commit log fixed up:
Quoted 48 lines
fs/xfs/xfs_log_recover.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)fs/xfs/xfs_log_recover.c +14
-6@@ -2999,7 +2999,7 @@ xlog_do_recovery_pass(int error =0 , h_size, h_len;int error2 =0 ;int bblks, split_bblks; -int hblks, split_hblks, wrapped_hblks; +int hblks =1 , split_hblks, wrapped_hblks;int i;struct hlist_head rhash[XLOG_RHASH_SIZE ];LIST_HEAD (buffer_list);@@ -3055,14 +3055,22 @@ xlog_do_recovery_pass(if (error) goto bread_err1; - hblks =xlog_logrec_hblks (log, rhead); -if (hblks !=1 ) { -kvfree (hbp); - hbp =xlog_alloc_buffer (log, hblks); +/* +* This open codes xlog_logrec_hblks so that we can reuse the +* fixed up h_size value calculated above. Without that we'd +* still allocate the buffer based on the incorrect on-disk +* size. +*/ +if (h_size >XLOG_HEADER_CYCLE_SIZE && + (rhead->h_version &cpu_to_be32 (XLOG_VERSION_2 ))) { + hblks =DIV_ROUND_UP (h_size,XLOG_HEADER_CYCLE_SIZE ); +if (hblks >1 ) { +kvfree (hbp); + hbp =xlog_alloc_buffer (log, hblks); + } } }else {ASSERT (log->l_sectBBsize ==1 ); - hblks =1 ; hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; }Signature · 1 line
2.39.2
Re: [PATCH 1/3] xfs: fix log recovery buffer allocation for the legacy h_size fixup
Quoted 8 lines · Brian Foster
Quoted 3 lines
Reported-by Sam Sun <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>---
The commit log/fixes tag are incorrect...
xlog_logrec_hblks()didn't exist at the timea70f9fe52daawas committed. I suspect this broke later in commit0c771b99d6c9("xfs: clean up calculation of LR header blocks"), but please double check.
Oh, indeed. My (git-)blame was a little too quick.
Looking at that later commit: can xfs_sb_version_haslogv2 and the per-xlog_rec_header XLOG_VERSION_2 ever disagree? Do we need to check for that?
Re: [PATCH 1/3] xfs: fix log recovery buffer allocation for the legacy h_size fixup
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
Commit
a70f9fe52daa("xfs: detect and handle invalid iclog size set by mkfs") added a fixup for incorrecth_sizevalues used for the initial umount record in old xfsprogs versions. But it is not using this fixed up value to size the log recovery buffer, which can lead to an out of bounds access when the incorrecth_sizedoes not come from the old mkfs tool, but a fuzzer.Fix this by open coding
xlog_logrec_hblksand taking the fixedh_sizeinto account for this calculation.Fixes a70f9fe52daa ("xfs: detect and handle invalid iclog size set by mkfs")
Modulo what bfoster said about the tagging,
--D
Quoted 51 lines
Reported-by Sam Sun <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>--- fs/xfs/xfs_log_recover.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-)fs/xfs/xfs_log_recover.c +14
-6@@ -2999,7 +2999,7 @@ xlog_do_recovery_pass(int error =0 , h_size, h_len;int error2 =0 ;int bblks, split_bblks; -int hblks, split_hblks, wrapped_hblks; +int hblks =1 , split_hblks, wrapped_hblks;int i;struct hlist_head rhash[XLOG_RHASH_SIZE ];LIST_HEAD (buffer_list);@@ -3055,14 +3055,22 @@ xlog_do_recovery_pass(if (error) goto bread_err1; - hblks =xlog_logrec_hblks (log, rhead); -if (hblks !=1 ) { -kvfree (hbp); - hbp =xlog_alloc_buffer (log, hblks); +/* +* This open codes xlog_logrec_hblks so that we can reuse the +* fixed up h_size value calculated above. Without that we'd +* still allocate the buffer based on the incorrect on-disk +* size. +*/ +if (h_size >XLOG_HEADER_CYCLE_SIZE && + (rhead->h_version &cpu_to_be32 (XLOG_VERSION_2 ))) { + hblks =DIV_ROUND_UP (h_size,XLOG_HEADER_CYCLE_SIZE ); +if (hblks >1 ) { +kvfree (hbp); + hbp =xlog_alloc_buffer (log, hblks); + } } }else {ASSERT (log->l_sectBBsize ==1 ); - hblks =1 ; hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; }Signature · 1 line
2.39.2
PATCH 2/3 xfs: restrict the h_size fixup in xlog_do_recovery_pass
In reply to Christoph HellwigThe reflink and rmap features require a fixed xfsprogs, so don't allow this fixup for them.
--- fs/xfs/xfs_log_recover.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/xfs/xfs_log_recover.c +5 -1
@@ -3040,10 +3040,14 @@ xlog_do_recovery_pass(* Detect this condition here. Use lsunit for the buffer size as * long as this looks like the mkfs case. Otherwise, return an * error to avoid a buffer overrun. +* +* Reject the invalid size if the file system has new enough +* features that require a fixed mkfs. */ h_size =be32_to_cpu (rhead->h_size); h_len =be32_to_cpu (rhead->h_len); -if (h_len > h_size && h_len <= log->l_mp->m_logbsize && +if (!xfs_has_reflink (log->l_mp) &&xfs_has_reflink (log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize && rhead->h_num_logops ==cpu_to_be32 (1 )) { xfs_warn(log->l_mp, "invalid iclog size (%d bytes), using lsunit (%d bytes)",
Signature · 1 line
2.39.2
Re: [PATCH 2/3] xfs: restrict the h_size fixup in xlog_do_recovery_pass
Quoted 5 lines · Christoph Hellwig
The reflink and rmap features require a fixed xfsprogs, so don't allow this fixup for them.
Signed-off-by Christoph Hellwig <[email protected]>---
Seems reasonable..
Quoted 10 lines
fs/xfs/xfs_log_recover.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)fs/xfs/xfs_log_recover.c +0
-0@@ -3040,10 +3040,14 @@ xlog_do_recovery_pass(* Detect this condition here. Use lsunit for the buffer size as * long as this looks like the mkfs case. Otherwise, return an
fs/xfs/xfs_log_recover.c xlog_do_recovery_pass( · line 3050
Quoted 10 lines
fs/xfs/xfs_log_recover.c +5
-1* error to avoid a buffer overrun. +* +* Reject the invalid size if the file system has new enough +* features that require a fixed mkfs. */ h_size =be32_to_cpu (rhead->h_size); h_len =be32_to_cpu (rhead->h_len); - if (h_len > h_size && h_len <= log->l_mp->m_logbsize && + if (!xfs_has_reflink(log->l_mp) && xfs_has_reflink(log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize &&
... but I'm going to assume this hasn't been tested. ;) Do you mean to also check !rmapbt here?
Can you please also just double check that we still handle the original mkfs problem correctly after these changes? I think that just means mkfs from a sufficiently old xfsprogs using a larger log stripe unit, and confirm the fs mounts (with a warning).
Brian
Quoted 7 lines
fs/xfs/xfs_log_recover.c +0
-0rhead->h_num_logops == cpu_to_be32(1)) { xfs_warn(log->l_mp, "invalid iclog size (%d bytes), using lsunit (%d bytes)",Signature · 1 line
2.39.2
Re: [PATCH 2/3] xfs: restrict the h_size fixup in xlog_do_recovery_pass
Quoted 6 lines · Brian Foster
fs/xfs/xfs_log_recover.c xlog_do_recovery_pass( · line 3050
Quoted 3 lines
fs/xfs/xfs_log_recover.c +2
-1- if (h_len > h_size && h_len <= log->l_mp->m_logbsize && + if (!xfs_has_reflink(log->l_mp) && xfs_has_reflink(log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize &&... but I'm going to assume this hasn't been tested. ;) Do you mean to also check !rmapbt here?
Heh. Well, it has been tested in that we don't do the fixup for the reproducer fixed by the previous patch and in that xfstests still passes. I guess nothing in there hits the old mkfs fixup, which isn't surprising.
Quoted 4 lines
Can you please also just double check that we still handle the original mkfs problem correctly after these changes? I think that just means mkfs from a sufficiently old xfsprogs using a larger log stripe unit, and confirm the fs mounts (with a warning).
Yeah. Is there any way to commit a fs image to xfstests so that we actually regularly test for it?
Re: [PATCH 2/3] xfs: restrict the h_size fixup in xlog_do_recovery_pass
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Brian Foster
fs/xfs/xfs_log_recover.c xlog_do_recovery_pass( · line 3050
Quoted 3 lines
fs/xfs/xfs_log_recover.c +2
-1- if (h_len > h_size && h_len <= log->l_mp->m_logbsize && + if (!xfs_has_reflink(log->l_mp) && xfs_has_reflink(log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize &&... but I'm going to assume this hasn't been tested. ;) Do you mean to also check !rmapbt here?
Quoted 4 lines
Heh. Well, it has been tested in that we don't do the fixup for the reproducer fixed by the previous patch and in that xfstests still passes. I guess nothing in there hits the old mkfs fixup, which isn't surprising.
Yeah.. (sorry, just teasing about the testing.. ;).
Quoted 8 lines
Quoted 4 lines
Can you please also just double check that we still handle the original mkfs problem correctly after these changes? I think that just means mkfs from a sufficiently old xfsprogs using a larger log stripe unit, and confirm the fs mounts (with a warning).
Yeah. Is there any way to commit a fs image to xfstests so that we actually regularly test for it?
Not sure.. ideally we could fuzz the log record header somehow or another to test for these various scenarios, since we clearly broke this once already.
I don't quite recall if I looked into that at the time of the original workaround. To Darrick's point, I wonder if there would be some use to an expert logformat command or something that allowed for some bonkers parameters (assuming something like that doesn't exist already).
I'm out on PTO for (at least) today, but I can take a closer look at that once I'm back and caught up...
Brian
Re: [PATCH 2/3] xfs: restrict the h_size fixup in xlog_do_recovery_pass
Quoted 29 lines · Brian Foster
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Brian Foster
fs/xfs/xfs_log_recover.c xlog_do_recovery_pass( · line 3050
Quoted 3 lines
fs/xfs/xfs_log_recover.c +2
-1- if (h_len > h_size && h_len <= log->l_mp->m_logbsize && + if (!xfs_has_reflink(log->l_mp) && xfs_has_reflink(log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize &&... but I'm going to assume this hasn't been tested. ;) Do you mean to also check !rmapbt here?
Quoted 4 lines
Heh. Well, it has been tested in that we don't do the fixup for the reproducer fixed by the previous patch and in that xfstests still passes. I guess nothing in there hits the old mkfs fixup, which isn't surprising.
Yeah.. (sorry, just teasing about the testing.. ;).
Quoted 8 lines
Quoted 4 lines
Can you please also just double check that we still handle the original mkfs problem correctly after these changes? I think that just means mkfs from a sufficiently old xfsprogs using a larger log stripe unit, and confirm the fs mounts (with a warning).
Yeah. Is there any way to commit a fs image to xfstests so that we actually regularly test for it?
Not sure.. ideally we could fuzz the log record header somehow or another to test for these various scenarios, since we clearly broke this once already.
Quoted 8 lines
I don't quite recall if I looked into that at the time of the original workaround. To Darrick's point, I wonder if there would be some use to an expert logformat command or something that allowed for some bonkers parameters (assuming something like that doesn't exist already).
I'm out on PTO for (at least) today, but I can take a closer look at that once I'm back and caught up...
I've had a chance to poke at this a bit and so far I don't think it's as straightforward as I'd hoped. The logformat command already exists, which makes it pretty easy to malformat a log record, but the recovery code only runs into this path on a dirty log. I suspect the original issue that prompted this logic was something like a crash-recovery test leading to log record trimming (i.e. simulated torn log writes) and then happening onto previously mkfs-formatted log records that way, but that is a guess at this point.
I did play around a bit with fuzzing h_size for those sorts of tests (i.e. xfs/141), but that ran into other issues. I went back to using xfsprogs v4.3 or so and that eventually also produces an unmountable filesystem with a similar error (even with the h_size fix patch). It fails to locate the head/tail, which is obviously necessary in order to process the log and perform record validation, so I'm wondering if something else changed on the kernel side to further gate this scenario. Of course it's also possible I'm looking at things wrong and this is just orthogonal to the original problem.
But given all of that, I am a little skeptical on the value of retaining this logic at all. Does whatever test case that recently uncovered the h_size problem leave a mountable filesystem, or does it just work around bad behavior to provide a more graceful failure condition?
Brian
Re: [PATCH 2/3] xfs: restrict the h_size fixup in xlog_do_recovery_pass
In reply to Christoph HellwigQuoted 16 lines · Christoph Hellwig
The reflink and rmap features require a fixed xfsprogs, so don't allow this fixup for them.
Signed-off-by Christoph Hellwig <[email protected]>--- fs/xfs/xfs_log_recover.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)fs/xfs/xfs_log_recover.c +0
-0@@ -3040,10 +3040,14 @@ xlog_do_recovery_pass(* Detect this condition here. Use lsunit for the buffer size as * long as this looks like the mkfs case. Otherwise, return an * error to avoid a buffer overrun.
fs/xfs/xfs_log_recover.c xlog_do_recovery_pass( · line 3051
Quoted 10 lines
fs/xfs/xfs_log_recover.c +5
-1+* +* Reject the invalid size if the file system has new enough +* features that require a fixed mkfs. */ h_size =be32_to_cpu (rhead->h_size); h_len =be32_to_cpu (rhead->h_len); -if (h_len > h_size && h_len <= log->l_mp->m_logbsize && +if (!xfs_has_reflink (log->l_mp) &&xfs_has_reflink (log->l_mp) && + h_len > h_size && h_len <= log->l_mp->m_logbsize && rhead->h_num_logops ==cpu_to_be32 (1 )) {
Same comment about do you want to test for rmap and reflink here?
I also wonder if this multiline predicate should turn into a static inline helper. I nearly wrote you one, but then I realize that I don't remember enough about the xfsprogs problem to know if the problem was limited to mkfs, or if xfs_repair zeroing the log would also write out a bad h_size?
--D
Quoted 6 lines
fs/xfs/xfs_log_recover.c +0
-0xfs_warn(log->l_mp, "invalid iclog size (%d bytes), using lsunit (%d bytes)",Signature · 1 line
2.39.2
PATCH 3/3 xfs: clean up buffer allocation in xlog_do_recovery_pass
In reply to Christoph HellwigMerge the initial xlog_alloc_buffer calls, and pass the variable designating the length that is initialized to 1 above instead of passing the open coded 1 directly.
--- fs/xfs/xfs_log_recover.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
fs/xfs/xfs_log_recover.c +6 -7
@@ -3010,6 +3010,10 @@ xlog_do_recovery_pass(for (i =0 ; i <XLOG_RHASH_SIZE ; i++)INIT_HLIST_HEAD (&rhash[i]); + hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; +/* * Read the header of the tail block and get the iclog buffer size from * h_size. Use this to tell how many sectors make up the log header.
@@ -3020,10 +3024,6 @@ xlog_do_recovery_pass(* iclog header and extract the header size from it. Get a * new hbp that is the correct size. */ - hbp =xlog_alloc_buffer (log,1 ); -if (!hbp) -return -ENOMEM ; - error =xlog_bread (log, tail_blk,1 , hbp, &offset);if (error) goto bread_err1;
@@ -3071,16 +3071,15 @@ xlog_do_recovery_pass(if (hblks >1 ) {kvfree (hbp); hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; } } }else {ASSERT (log->l_sectBBsize ==1 ); - hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; } -if (!hbp) -return -ENOMEM ; dbp =xlog_alloc_buffer (log,BTOBB (h_size));if (!dbp) {kvfree (hbp);
Signature · 1 line
2.39.2
Re: [PATCH 3/3] xfs: clean up buffer allocation in xlog_do_recovery_pass
Quoted 6 lines · Christoph Hellwig
Merge the initial
xlog_alloc_buffercalls, and pass the variable designating the length that is initialized to 1 above instead of passing the open coded 1 directly.Signed-off-by Christoph Hellwig <[email protected]>---
Quoted 52 lines
fs/xfs/xfs_log_recover.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)fs/xfs/xfs_log_recover.c +6
-7@@ -3010,6 +3010,10 @@ xlog_do_recovery_pass(for (i =0 ; i <XLOG_RHASH_SIZE ; i++)INIT_HLIST_HEAD (&rhash[i]); + hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; +/* * Read the header of the tail block and get the iclog buffer size from * h_size. Use this to tell how many sectors make up the log header. @@ -3020,10 +3024,6 @@ xlog_do_recovery_pass(* iclog header and extract the header size from it. Get a * new hbp that is the correct size. */ - hbp =xlog_alloc_buffer (log,1 ); -if (!hbp) -return -ENOMEM ; - error =xlog_bread (log, tail_blk,1 , hbp, &offset);if (error) goto bread_err1;@@ -3071,16 +3071,15 @@ xlog_do_recovery_pass(if (hblks >1 ) {kvfree (hbp); hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; } } }else {ASSERT (log->l_sectBBsize ==1 ); - hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; } -if (!hbp) -return -ENOMEM ; dbp =xlog_alloc_buffer (log,BTOBB (h_size));if (!dbp) {kvfree (hbp);Signature · 1 line
2.39.2
Re: [PATCH 3/3] xfs: clean up buffer allocation in xlog_do_recovery_pass
In reply to Christoph HellwigQuoted 5 lines · Christoph Hellwig
Merge the initial
xlog_alloc_buffercalls, and pass the variable designating the length that is initialized to 1 above instead of passing the open coded 1 directly.Signed-off-by Christoph Hellwig <[email protected]>
Nice cleanup,
--D
Quoted 53 lines
--- fs/xfs/xfs_log_recover.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)fs/xfs/xfs_log_recover.c +6
-7@@ -3010,6 +3010,10 @@ xlog_do_recovery_pass(for (i =0 ; i <XLOG_RHASH_SIZE ; i++)INIT_HLIST_HEAD (&rhash[i]); + hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; +/* * Read the header of the tail block and get the iclog buffer size from * h_size. Use this to tell how many sectors make up the log header. @@ -3020,10 +3024,6 @@ xlog_do_recovery_pass(* iclog header and extract the header size from it. Get a * new hbp that is the correct size. */ - hbp =xlog_alloc_buffer (log,1 ); -if (!hbp) -return -ENOMEM ; - error =xlog_bread (log, tail_blk,1 , hbp, &offset);if (error) goto bread_err1;@@ -3071,16 +3071,15 @@ xlog_do_recovery_pass(if (hblks >1 ) {kvfree (hbp); hbp =xlog_alloc_buffer (log, hblks); +if (!hbp) +return -ENOMEM ; } } }else {ASSERT (log->l_sectBBsize ==1 ); - hbp =xlog_alloc_buffer (log,1 ); h_size =XLOG_BIG_RECORD_BSIZE ; } -if (!hbp) -return -ENOMEM ; dbp =xlog_alloc_buffer (log,BTOBB (h_size));if (!dbp) {kvfree (hbp);Signature · 1 line
2.39.2