btrfs: Do mandatory tree block check before submitting bio
PATCH 1/1 btrfs: Do mandatory tree block check before submitting bio
There are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at btrfs_mark_buffer_dirty() time, as it's not mandatory, only for CONFIG_BTRFS_FS_CHECK_INTEGRITY enabled build.
This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The timing is set to csum_tree_block() where @verify == 0. At that timing, we're generation csum for tree blocks before submitting the metadata bio, so we could avoid all the unnecessary calls at btrfs_mark_buffer_dirty(), but still catch enough error.
--- fs/btrfs/disk-io.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/btrfs/disk-io.c +10 -0
@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +/* +* Here we're calculating csum before writing it to disk, +* do comprenhensive check here to catch memory corruption +*/ +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_full (fs_info, buf); +if (err <0 ) +return err;write_extent_buffer (buf, result,0 , csum_size); }
Signature · 1 line
2.20.1
Hi, Qu,
Quoted 7 lines · Qu Wenruo
There are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at
btrfs_mark_buffer_dirty()time, as it's not mandatory, only forCONFIG_BTRFS_FS_CHECK_INTEGRITYenabled build.
Quoted 7 lines
This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The timing is set to
csum_tree_block()where @verify == 0. At that timing, we're generation csum for tree blocks before submitting the metadata bio, so we could avoid all the unnecessary calls atbtrfs_mark_buffer_dirty(), but still catch enough error.
I agree wholeheartedly with the idea of this change. Just one
question:
How does this get reported to the user/administrator? As I
understand it, a detectably corrupt metadata page will generate an I/O error from the filesystem before it's written to disk? How will this show up in kernel logs? Is it distinguishable in any way from a similar error that was generated on reading such a corrupt metadata node from the disk?
Basically, I want to be able to distinguish this case (error
detected when writing) from the existing case (error detected when reading) when someone shows up on IRC with a "broken filesystem".
Hugo.
Quoted 27 lines
Reported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ 1 file changed, 10 insertions(+)fs/btrfs/disk-io.c +10
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +/* +* Here we're calculating csum before writing it to disk, +* do comprenhensive check here to catch memory corruption +*/ +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_full (fs_info, buf); +if (err <0 ) +return err;write_extent_buffer (buf, result,0 , csum_size); }
Signature · 4 lines
Hugo Mills | ... one ping(1) to rule them all, and in the hugo@... carfax.org.uk | darkness bind(2) them. http://carfax.org.uk/ | PGP: E2AB1DE4 | Illiad
Quoted 18 lines · Hugo Mills
Hi, Qu,Quoted 7 lines · Qu Wenruo
There are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at
btrfs_mark_buffer_dirty()time, as it's not mandatory, only forCONFIG_BTRFS_FS_CHECK_INTEGRITYenabled build.Quoted 7 lines
This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The timing is set to
csum_tree_block()where @verify == 0. At that timing, we're generation csum for tree blocks before submitting the metadata bio, so we could avoid all the unnecessary calls atbtrfs_mark_buffer_dirty(), but still catch enough error.
Quoted 7 lines
I agree wholeheartedly with the idea of this change. Just one
question:How does this get reported to the user/administrator? As I
understand it, a detectably corrupt metadata page will generate an I/O error from the filesystem before it's written to disk? How will this show up in kernel logs?
Well, you caught me.
I haven't try the error case, and in fact if it fails, it fails by triggering kernel BUG_ON(), thus you may not have a chance to see the error message from btrfs module.
Quoted 3 lines
Is it distinguishable in any way from a
similar error that was generated on reading such a corrupt metadata node from the disk?
Kind of distinguishable for this patch, when you hit kernel BUG_ON() at fs/btrfs/extent_io.c:4016 then it's definitely from this patch. :P
Quoted 4 lines
Basically, I want to be able to distinguish this case (error
detected when writing) from the existing case (error detected when reading) when someone shows up on IRC with a "broken filesystem".
Definitely, I'll address this and the BUG_ON() in next version.
Thanks,
Qu
Quoted 31 lines
Hugo.
Quoted 27 lines
Reported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ 1 file changed, 10 insertions(+)fs/btrfs/disk-io.c +10
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +/* +* Here we're calculating csum before writing it to disk, +* do comprenhensive check here to catch memory corruption +*/ +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_full (fs_info, buf); +if (err <0 ) +return err;write_extent_buffer (buf, result,0 , csum_size); }
Quoted 35 lines · Qu Wenruo
Quoted 18 lines · Hugo Mills
Hi, Qu,Quoted 7 lines · Qu Wenruo
There are at least 2 reports about memory bit flip sneaking into on-disk data.
Currently we only have a relaxed check triggered at
btrfs_mark_buffer_dirty()time, as it's not mandatory, only forCONFIG_BTRFS_FS_CHECK_INTEGRITYenabled build.Quoted 7 lines
This patch will address the hole by triggering comprehensive check on tree blocks before writing it back to disk.
The timing is set to
csum_tree_block()where @verify == 0. At that timing, we're generation csum for tree blocks before submitting the metadata bio, so we could avoid all the unnecessary calls atbtrfs_mark_buffer_dirty(), but still catch enough error.Quoted 7 lines
I agree wholeheartedly with the idea of this change. Just one
question:How does this get reported to the user/administrator? As I
understand it, a detectably corrupt metadata page will generate an I/O error from the filesystem before it's written to disk? How will this show up in kernel logs?Well, you caught me.
I haven't try the error case, and in fact if it fails, it fails by triggering kernel
BUG_ON(), thus you may not have a chance to see the error message from btrfs module.
Quoted 6 lines
Quoted 3 lines
Is it distinguishable in any way from a
similar error that was generated on reading such a corrupt metadata node from the disk?Kind of distinguishable for this patch, when you hit kernel
BUG_ON()atfs/btrfs/extent_io.c:4016then it's definitely from this patch. :P
Haha. :)
Quoted 5 lines
Quoted 3 lines
Basically, I want to be able to distinguish this case (error
detected when writing) from the existing case (error detected when reading) when someone shows up on IRC with a "broken filesystem".Definitely, I'll address this and the
BUG_ON()in next version.
The error-on-read case gives us a pretty good report on what's
wrong and where. Typically something like "bad key order" or "wrong item size", plus the logical address of the metadata chunk so we can dump it with debug-tree -b.
Having the same messages from this call-site to indicate the kind
of error found, and also something to indicate that it was detected before hitting disk (i.e. which call-site for the checks triggered the error) would be, I think, the minimal information needed. If we could also have the human-readable dump of the full metadata page logged as well, that would be ideal -- we won't be able to use debug-tree to diagnose the issue afterwards, as it won't have reached the disk.
Thanks,
Hugo.
Quoted 35 lines
Thanks,
QuQuoted 31 lines
Hugo.
Quoted 27 lines
Reported-by Leonard Lausen <[email protected]>Signed-off-by Qu Wenruo <[email protected]>--- fs/btrfs/disk-io.c | 10 ++++++++++ 1 file changed, 10 insertions(+)fs/btrfs/disk-io.c +10
-0@@ -313,6 +313,16 @@ static int csum_tree_block(struct btrfs_fs_info *fs_info,return -EUCLEAN ; } }else { +/* +* Here we're calculating csum before writing it to disk, +* do comprenhensive check here to catch memory corruption +*/ +if (btrfs_header_level (buf)) + err =btrfs_check_node (fs_info, buf); +else + err =btrfs_check_leaf_full (fs_info, buf); +if (err <0 ) +return err;write_extent_buffer (buf, result,0 , csum_size); }
Signature · 4 lines
Hugo Mills | Friends: people who know you well, but like you hugo@... carfax.org.uk | anyway. http://carfax.org.uk/ | PGP: E2AB1DE4 |