btrfs-progs: introduce "btrfs rescue fix-data-checksum"
Subsystems (3)
v2 cover btrfs-progs: introduce "btrfs rescue fix-data-checksum"
[CHANGELOG]
v2:
- Rename the subcommand to "fix-data-checksum" It's better to use full name in the command name
- Remove unused members inside corrupted_block
The old @extent_bytenr and @extent_len is no longer needed, even for the future file-deletion action.
- Fix the bitmap size off-by-1 bug
We must use the bit 0 to represent mirror 1, or the bitmap size will exceed num_mirrors.
- Introduce -i|--interactive mode
Will ask the user for the action on the corrupted block, including:
* Ignore
The default behavior if no command is provided
* Use specified mirror to update the data checksum item The user must input a number inside range [1, num_mirrors].
- Introduce -m|--mirror <num> mode
Use specified mirror for all corrupted blocks.
The value <num> must be >= 1. And if the value is larger than the actual max mirror number, the real mirror number will be num % (num_mirror + 1).
We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
Although the problem is worked around in v6.15 (and being backported), for the affected fs there is no good way to fix them, other than complex manually find out which files are affected and delete them.
This series introduce the initial implementation of "btrfs rescue fix-data-checksum", which is designed to fix such problem by either:
- Update the csum items using the data from specified copy
The subcommand has 3 modes so far:
- Readonly mode
Only report all corrupted blocks and affected files, no repair is done.
- Interactive mode
Ask for what to do, including
* Ignore (the default)
* Use certain mirror to update the checksum item
- Mirror mode
Use specified mirror to update the checksum item, the batch mode of the interactive one.
In the future, there will be one more mode:
- Delete mode
Delete all involved files.
There are still some points to address before implementing this mode.
Qu Wenruo (6):
btrfs-progs: introduce "btrfs rescue fix-data-checksum"
btrfs-progs: fix a bug in btrfs_find_item()
btrfs-progs: fix-data-checksum: show affected files
btrfs-progs: fix-data-checksum: introduce interactive mode
btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
btrfs-progs: fix-data-checksum: introduce -m|--mirror option
Documentation/btrfs-rescue.rst | 28 ++ Makefile | 2 +- cmds/rescue-fix-data-checksum.c | 511 ++++++++++++++++++++++++++++++++ cmds/rescue.c | 65 ++++ cmds/rescue.h | 10 + kernel-shared/ctree.c | 17 +- kernel-shared/file-item.c | 2 +- kernel-shared/file-item.h | 5 + 8 files changed, 635 insertions(+), 5 deletions(-) create mode 100644 cmds/rescue-fix-data-checksum.c
--
2.49.0
PATCH v2 1/6 btrfs-progs: introduce "btrfs rescue fix-data-checksum"
It's a long known problem that direct IO can lead to data checksum mismatches if the user space is also modifying its buffer during writeback.
Although it's fixed in the recent v6.15 release (and backported to older kernels), we still need a user friendly way to fix those problems.
This patch introduce the dryrun version of "btrfs rescue fix-data-checksum", which reports the logical bytenr and corrupted mirrors.
--- Documentation/btrfs-rescue.rst | 19 +++ Makefile | 2 +- cmds/rescue-fix-data-checksum.c | 288 ++++++++++++++++++++++++++++++++ cmds/rescue.c | 48 ++++++ cmds/rescue.h | 7 + 5 files changed, 363 insertions(+), 1 deletion(-) create mode 100644 cmds/rescue-fix-data-checksum.c
Documentation/btrfs-rescue.rst +19 -0
@@ -50,6 +50,25 @@ fix-device-size <device> WARNING: CPU: 3 PID: 439 at fs/btrfs/ctree.h:1559 btrfs_update_device+0x1c5/0x1d0 [btrfs] +fix-data-checksum <device> + fix data checksum mismatch + + There is a long existing problem that if a user space program is doing + direct IO and modifies the buffer before the write back finished, it + can lead to data checksum mismatches. + + This problem is known but not fixed until upstream release v6.15 + (backported to older kernels). So it's possible to hit false data + checksum mismatch for any long running btrfs. + + In that case this program can be utilized to repair such problem. + + ``Options`` + + -r|--readonly + readonly mode, only scan and report for data checksum mismatch, + do no repair + .. _man-rescue-clear-ino-cache: clear-ino-cache <device>
Makefile +1 -1
@@ -256,7 +256,7 @@ cmds_objects = cmds/subvolume.o cmds/subvolume-list.o \ cmds/inspect.o cmds/balance.o cmds/send.o cmds/receive.o \ cmds/quota.o cmds/qgroup.o cmds/replace.o check/main.o \ cmds/restore.o cmds/rescue.o cmds/rescue-chunk-recover.o \ - cmds/rescue-super-recover.o \ + cmds/rescue-super-recover.o cmds/rescue-fix-data-checksum.o \ cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o \ cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o \ cmds/reflink.o \
cmds/rescue-fix-data-checksum.c new file +288 -0
288 lines not on this page · raw message
cmds/rescue.c +48 -0
66 lines not on this page · raw message
cmds/rescue.h +7 -0
@@ -20,7 +20,14 @@#ifndef __BTRFS_RESCUE_H__#define __BTRFS_RESCUE_H__ +enum btrfs_fix_data_checksum_mode { +BTRFS_FIX_DATA_CSUMS_READONLY , +BTRFS_FIX_DATA_CSUMS_LAST , +}; +int btrfs_recover_superblocks (const char *path,int yes);int btrfs_recover_chunk_tree (const char *path,int yes); +int btrfs_recover_fix_data_checksum (const char *path, +enum btrfs_fix_data_checksum_mode mode);#endif
Signature · 1 line
2.49.0
PATCH v2 3/6 btrfs-progs: fix-data-checksum: show affected files
In reply to Qu WenruoPreviously "btrfs rescue fix-data-checksum" only show affected logical bytenr, which is not helpful to determine which files are affected.
Enhance the output by also outputting the affected subvolumes (in numeric form), and the file paths inside that subvolume.
The example looks like this:
logical=13631488 corrtuped mirrors=1 affected files:
(subvolume 5)/foo
(subvolume 5)/dir/bar
logical=13635584 corrtuped mirrors=1 affected files:
(subvolume 5)/foo
(subvolume 5)/dir/bar
Although the end result is still not perfect, it's still much easier to find out which files are affected.
--- cmds/rescue-fix-data-checksum.c | 59 +++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 3 deletions(-)
cmds/rescue-fix-data-checksum.c +56 -3
@@ -18,6 +18,7 @@#include "kernel-shared/disk-io.h" #include "kernel-shared/ctree.h" #include "kernel-shared/volumes.h" +#include "kernel-shared/backref.h" #include "common/messages.h" #include "common/open-utils.h" #include "cmds/rescue.h"
@@ -158,6 +159,49 @@ static int iterate_one_csum_item(struct btrfs_fs_info *fs_info, struct btrfs_patreturn ret; } +static int print_filenames (u64 ino,u64 offset,u64 rootid,void *ctx) +{ +struct btrfs_fs_info *fs_info = ctx; +struct btrfs_root *root; +struct btrfs_key key; +struct inode_fs_paths *ipath; +struct btrfs_path path = {0 }; +int ret; + + key.objectid = rootid; + key.type =BTRFS_ROOT_ITEM_KEY ; + key.offset = (u64)-1 ; + + root =btrfs_read_fs_root (fs_info, &key); +if (IS_ERR (root)) { + ret =PTR_ERR (root); + errno = -ret; +error ("failed to get subvolume %llu: %m" , rootid); +return ret; + } + ipath =init_ipath (128 *BTRFS_PATH_NAME_MAX , root, &path); +if (IS_ERR (ipath)) { + ret =PTR_ERR (ipath); + errno = -ret; +error ("failed to initialize ipath: %m" ); +return ret; + } + ret =paths_from_inode (ino, ipath); +if (ret <0 ) { + errno = -ret; +error ("failed to resolve root %llu ino %llu to paths: %m" , rootid, ino); + goto out; + } +for (int i =0 ; i < ipath->fspath->elem_cnt; i++) +printf (" (subvolume %llu)/%s\n" , rootid, (char *)ipath->fspath->val[i]); +if (ipath->fspath->elem_missed) +printf (" (subvolume %llu) %d files not printed\n" , rootid, + ipath->fspath->elem_missed); +out: +free_ipath (ipath); +return ret; +} +static int iterate_csum_root (struct btrfs_fs_info *fs_info,struct btrfs_root *csum_root) {struct btrfs_path path = {0 };
@@ -197,9 +241,10 @@ next:return ret; } -static void report_corrupted_blocks (void ) +static void report_corrupted_blocks (struct btrfs_fs_info *fs_info) {struct corrupted_block *entry; +struct btrfs_path path = {0 };if (list_empty (&corrupted_blocks)) {printf ("No data checksum mismatch found\n" );
@@ -208,6 +253,7 @@ static void report_corrupted_blocks(void)list_for_each_entry (entry, &corrupted_blocks, list) {bool has_printed = false; +int ret;printf ("logical=%llu corrtuped mirrors=" , entry->logical);/* Poor man's bitmap print. */
@@ -223,7 +269,14 @@ static void report_corrupted_blocks(void) has_printed=true; } } -printf ("\n" ); +printf (" affected files:\n" ); + ret =iterate_inodes_from_logical (entry->logical, fs_info, &path, + print_filenames, fs_info); +if (ret <0 ) { + errno = -ret; +error ("failed to iterate involved files: %m" ); +break ; + } } }
@@ -280,7 +333,7 @@ int btrfs_recover_fix_data_checksum(const char *path, errno = -ret;error ("failed to iterate csum tree: %m" ); } -report_corrupted_blocks (); +report_corrupted_blocks (fs_info); out_close:free_corrupted_blocks ();close_ctree_fs_info (fs_info);
Signature · 1 line
2.49.0
PATCH v2 2/6 btrfs-progs: fix a bug in btrfs_find_item()
In reply to Qu Wenruo[BUG]
There is a seldomly utilized function, btrfs_find_item(), which has no document and is not behaving correctly.
Inside backref.c, iterate_inode_refs() and btrfs_ref_to_path() both utilize this function, to find the parent inode.
However btrfs_find_item() will never return 0 if @ioff is passed as 0 for such usage, result early failure for all kinds of inode iteration functions.
[CAUSE]
Both functions pass 0 as the @ioff parameter initially, e.g:
We have the following fs tree root:
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
generation 3 transid 9 size 6 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x0(none) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 index 0 namelen 2 name: .. item 2 key (256 DIR_ITEM 2507850652) itemoff 16078 itemsize 33 location key (257 INODE_ITEM 0) type FILE transid 9 data_len 0 name_len 3 name: foo item 3 key (256 DIR_INDEX 2) itemoff 16045 itemsize 33 location key (257 INODE_ITEM 0) type FILE transid 9 data_len 0 name_len 3 name: foo item 4 key (257 INODE_ITEM 0) itemoff 15885 itemsize 160 generation 9 transid 9 size 16384 nbytes 16384 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x0(none) item 5 key (257 INODE_REF 256) itemoff 15872 itemsize 13 index 2 namelen 3 name: foo item 6 key (257 EXTENT_DATA 0) itemoff 15819 itemsize 53 generation 9 type 1 (regular) extent data disk byte 13631488 nr 16384 extent data offset 0 nr 16384 ram 16384 extent compression 0 (none)
Then we call paths_from_inode() with:
- @inum = 257
- ipath = {.fs_root = 5}
Then we got the following sequence:
iterate_irefs(257, fs_root, inode_to_path)
|- iterate_inode_refs()
|- inode_ref_info()
|- btrfs_find_item(257, 0, fs_root)
| Returned 1, with @found_key updated to
| (257, INODE_REF, 256). This makes iterate_irefs() exit immediately, but obviously that btrfs_find_item() call is to find any INODE_REF, not to find the exact match.
[FIX]
If btrfs_find_item() found an item matching the objectid and type, then it should return 0 other than 1.
Fix it and keep the behavior the same across btrfs-progs and the kernel.
Since we're here, also add some comments explaining the function.
--- kernel-shared/ctree.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
kernel-shared/ctree.c +14 -3
@@ -1246,6 +1246,17 @@ static void reada_for_search(struct btrfs_fs_info *fs_info, } } +/* +* Find the first key in @fs_root that matches all the following conditions: +* +* - key.obojectid == @iobjectid +* - key.type == @key_type +* - key.offset >= ioff +* +* Return 0 if such key can be found, and @found_key is updated. +* Return >0 if no such key can be found. +* Return <0 for critical errors. +*/ int btrfs_find_item (struct btrfs_root *fs_root,struct btrfs_path *found_path,u64 iobjectid,u64 ioff,u8 key_type,struct btrfs_key *found_key)
@@ -1280,10 +1291,10 @@ int btrfs_find_item(struct btrfs_root *fs_root, struct btrfs_path *found_path,btrfs_item_key_to_cpu (eb, found_key, path->slots[0 ]);if (found_key->type != key.type || - found_key->objectid != key.objectid) { + found_key->objectid != key.objectid) ret =1 ; - goto out; - } +else + ret =0 ; out:if (path != found_path)
Signature · 1 line
2.49.0
PATCH v2 5/6 btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
In reply to Qu WenruoThis adds a new group of action in the interactive mode to fix a csum mismatch.
The output looks like this:
logical=13631488 corrtuped mirrors=1 affected files:
(subvolume 5)/foo
(subvolume 5)/dir/bar
<<I>>gnore/<1>:1
Csum item for logical 13631488 updated using data from mirror 1
In the interactive mode, the update-csum-item behavior is outputted as all available mirror numbers.
Considering all the existing (and future) action should starts with an alphabet, it's pretty easy to distinguish mirror number from other actions.
The update-csum-item action itself is pretty straight-forward, just read out the data from specified mirror, then calculate a new checksum, and update the corresponding csum item in csum tree.
--- cmds/rescue-fix-data-checksum.c | 119 +++++++++++++++++++++++++++++--- kernel-shared/file-item.c | 2 +- kernel-shared/file-item.h | 5 ++ 3 files changed, 114 insertions(+), 12 deletions(-)
cmds/rescue-fix-data-checksum.c +108 -11
@@ -20,6 +20,8 @@#include "kernel-shared/ctree.h" #include "kernel-shared/volumes.h" #include "kernel-shared/backref.h" +#include "kernel-shared/transaction.h" +#include "kernel-shared/file-item.h" #include "common/messages.h" #include "common/open-utils.h" #include "cmds/rescue.h"
@@ -48,6 +50,7 @@ struct corrupted_block {enum fix_data_checksum_action_value {ACTION_IGNORE , +ACTION_UPDATE_CSUM ,ACTION_LAST , };
@@ -59,6 +62,10 @@ static const struct fix_data_checksum_action { .value = ACTION_IGNORE, .string = "ignore", }, + [ACTION_UPDATE_CSUM] = { + .value = ACTION_UPDATE_CSUM, + .string = "update-csum", + }, };static int global_repair_mode;
@@ -258,10 +265,13 @@ next: }#define ASK_ACTION_BUFSIZE (32) -static enum fix_data_checksum_action_value ask_action () +static enum fix_data_checksum_action_value ask_action (unsigned int num_mirrors, +unsigned int *mirror_ret) { +unsigned long ret;char buf[ASK_ACTION_BUFSIZE ] = {0 };bool printed; +char *endptr; again: printed = false;
@@ -269,12 +279,22 @@ again:if (printed)printf ("/" );/* Mark Ignore as default */ -if (i ==ACTION_IGNORE ) +if (i ==ACTION_IGNORE ) {printf ("<<%c>>%s" ,toupper (actions[i].string[0 ]), actions[i].string +1 ); -else + }else if (i ==ACTION_UPDATE_CSUM ) { +/* +* For update-csum action, we need a mirror number, +* so output all valid mirrors numbers instead. +*/ +for (int cur_mirror =1 ; cur_mirror <= num_mirrors; + cur_mirror++) +printf ("<%u>" , cur_mirror); + }else {printf ("<%c>%s" ,toupper (actions[i].string[0 ]), actions[i].string +1 ); + } + printed = true; }printf (":" );fflush (stdout);
@@ -285,13 +305,79 @@ again:return ACTION_IGNORE ;/* Check exact match or matching the initial letter. */ for (int i =0 ; i <ACTION_LAST ; i++) { -if (strncasecmp (buf, actions[i].string,1 ) ==0 || -strncasecmp (buf, actions[i].string,ASK_ACTION_BUFSIZE ) ==0 ) +if ((strncasecmp (buf, actions[i].string,1 ) ==0 || +strncasecmp (buf, actions[i].string,ASK_ACTION_BUFSIZE ) ==0 ) && + actions[i].value !=ACTION_UPDATE_CSUM )return actions[i].value; } -/* No valid action found, retry. */ -warning ("invalid action, please retry" ); - goto again; +/* No match, check if it's some numeric string. */ + ret =strtoul (buf, &endptr,10 ); +if (endptr == buf || ret ==ULONG_MAX ) { +/* No valid action found, retry. */ +warning ("invalid action, please retry" ); + goto again; + } +if (ret > num_mirrors || ret ==0 ) { +warning ("invalid mirror number %lu, must be in range [1, %d], please retry" , + ret, num_mirrors); + goto again; + } + *mirror_ret = ret; +return ACTION_UPDATE_CSUM ; +} + +static int update_csum_item (struct btrfs_fs_info *fs_info,u64 logical, +unsigned int mirror) +{ +struct btrfs_trans_handle *trans; +struct btrfs_root *csum_root =btrfs_csum_root (fs_info, logical); +struct btrfs_path path = {0 }; +struct btrfs_csum_item *citem; +u64 read_len = fs_info->sectorsize; +u8 csum[BTRFS_CSUM_SIZE ] = {0 }; +u8 *buf; +int ret; + + buf =malloc (fs_info->sectorsize); +if (!buf) +return -ENOMEM ; + ret =read_data_from_disk (fs_info, buf, logical, &read_len, mirror); +if (ret <0 ) { + errno = -ret; +error ("failed to read block at logical %llu mirror %u: %m" , + logical, mirror); + goto out; + } + trans =btrfs_start_transaction (csum_root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error_msg (ERROR_MSG_START_TRANS ,"%m" ); + goto out; + } + citem =btrfs_lookup_csum (trans, csum_root, &path, logical, +BTRFS_EXTENT_CSUM_OBJECTID , fs_info->csum_type,1 ); +if (IS_ERR (citem)) { + ret =PTR_ERR (citem); + errno = -ret; +error ("failed to find csum item for logical %llu: $m" , logical); +btrfs_abort_transaction (trans, ret); + goto out; + } +btrfs_csum_data (fs_info, fs_info->csum_type, buf, csum, fs_info->sectorsize); +write_extent_buffer (path.nodes[0 ], csum, (unsigned long )citem, fs_info->csum_size); +btrfs_release_path (&path); + ret =btrfs_commit_transaction (trans, csum_root); +if (ret <0 ) { + errno = -ret; +error_msg (ERROR_MSG_COMMIT_TRANS ,"%m" ); + } +printf ("Csum item for logical %llu updated using data from mirror %u\n" , + logical, mirror); +out: +free (buf); +btrfs_release_path (&path); +return ret; } static void report_corrupted_blocks(struct btrfs_fs_info *fs_info,
@@ -307,6 +393,7 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info, }list_for_each_entry (entry, &corrupted_blocks, list) { +unsigned int mirror;bool has_printed = false;int ret;
@@ -334,10 +421,20 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info, }switch (mode) {case BTRFS_FIX_DATA_CSUMS_INTERACTIVE : - action =ask_action (); -UASSERT (action ==ACTION_IGNORE ); - fallthrough; + action =ask_action (entry->num_mirrors, &mirror); +break ;case BTRFS_FIX_DATA_CSUMS_READONLY : + action =ACTION_IGNORE ; +break ; +default : +UASSERT (0 ); + } + +switch (action) { +case ACTION_IGNORE : +break ; +case ACTION_UPDATE_CSUM : + ret =update_csum_item (fs_info, entry->logical, mirror);break ;default :UASSERT (0 );
kernel-shared/file-item.c +1 -1
@@ -112,7 +112,7 @@ fail:return err; } -static struct btrfs_csum_item * +struct btrfs_csum_item * btrfs_lookup_csum(struct btrfs_trans_handle *trans, struct btrfs_root *root, struct btrfs_path *path,
kernel-shared/file-item.h +5 -0
@@ -89,6 +89,11 @@ int btrfs_insert_file_extent(struct btrfs_trans_handle *trans, struct btrfs_file_extent_item *stack_fi);int btrfs_csum_file_block (struct btrfs_trans_handle *trans,u64 logical,u64 csum_objectid,u32 csum_type,const char *data); +struct btrfs_csum_item * +btrfs_lookup_csum (struct btrfs_trans_handle *trans, +struct btrfs_root *root, +struct btrfs_path *path, +u64 bytenr,u64 csum_objectid,u16 csum_type,int cow); int btrfs_insert_inline_extent(struct btrfs_trans_handle *trans, struct btrfs_root *root, u64 objectid, u64 offset, const char *buffer, size_t size,
Signature · 1 line
2.49.0
PATCH v2 4/6 btrfs-progs: fix-data-checksum: introduce interactive mode
In reply to Qu WenruoThis mode will ask user for how to fix each block.
User input can match the first letter or the whole action name to specify given action, the input is verified case insensitive.
If no user input is provided, the default action is to ignore the corrupted block.
If the input matches no action, a warning is outputted and user must retry until a valid input is provided.
--- Documentation/btrfs-rescue.rst | 3 ++ cmds/rescue-fix-data-checksum.c | 69 ++++++++++++++++++++++++++++++++- cmds/rescue.c | 9 ++++- cmds/rescue.h | 1 + 4 files changed, 78 insertions(+), 4 deletions(-)
Documentation/btrfs-rescue.rst +3 -0
@@ -69,6 +69,9 @@ fix-data-checksum <device> readonly mode, only scan and report for data checksum mismatch, do no repair + -i|--interactive + interactive mode, ask for how to repair, ignore the error by default + .. _man-rescue-clear-ino-cache: clear-ino-cache <device>
cmds/rescue-fix-data-checksum.c +67 -2
102 lines not on this page · raw message
cmds/rescue.c +7 -2
@@ -280,7 +280,8 @@ static const char * const cmd_rescue_fix_data_checksum_usage[] = {"btrfs rescue fix-data-checksum <device>" ,"Fix data checksum mismatches." ,"" , -OPTLINE ("-r" ,"readonly mode, only report errors without repair" ), +OPTLINE ("-r|--readonly" ,"readonly mode, only report errors without repair" ), +OPTLINE ("-i|--interactive" ,"interactive mode, ignore the error by default." ),HELPINFO_INSERT_GLOBALS ,HELPINFO_INSERT_VERBOSE ,NULL
@@ -298,15 +299,19 @@ static int cmd_rescue_fix_data_checksum(const struct cmd_struct *cmd,enum {GETOPT_VAL_DRYRUN =GETOPT_VAL_FIRST , };static const struct option long_options [] = { {"readonly" , no_argument,NULL ,'r' }, + {"interactive" , no_argument,NULL ,'i' }, {"NULL" ,0 ,NULL ,0 }, }; - c =getopt_long (argc, argv,"r" , long_options,NULL ); + c =getopt_long (argc, argv,"ri" , long_options,NULL );if (c <0 )break ;switch (c) {case 'r' : mode =BTRFS_FIX_DATA_CSUMS_READONLY ;break ; +case 'i' : + mode =BTRFS_FIX_DATA_CSUMS_INTERACTIVE ; +break ;default :usage_unknown_option (cmd, argv); }
cmds/rescue.h +1 -0
@@ -22,6 +22,7 @@enum btrfs_fix_data_checksum_mode {BTRFS_FIX_DATA_CSUMS_READONLY , +BTRFS_FIX_DATA_CSUMS_INTERACTIVE ,BTRFS_FIX_DATA_CSUMS_LAST , };
Signature · 1 line
2.49.0
PATCH v2 6/6 btrfs-progs: fix-data-checksum: introduce -m|--mirror option
In reply to Qu WenruoThis option allows "btrfs rescue fix-data-checksum" to use the specified mirror number to update checksum item for all corrupted mirrors.
If the specified number is larger than the max number of mirrors, the real mirror number will be num % (num_mirrors + 1).
--- Documentation/btrfs-rescue.rst | 6 ++++++ cmds/rescue-fix-data-checksum.c | 16 ++++++++++++---- cmds/rescue.c | 16 ++++++++++++++-- cmds/rescue.h | 4 +++- 4 files changed, 35 insertions(+), 7 deletions(-)
Documentation/btrfs-rescue.rst +6 -0
@@ -72,6 +72,12 @@ fix-data-checksum <device> -i|--interactive interactive mode, ask for how to repair, ignore the error by default + -m|--mirror <num> + use specified mirror to update the checksum item for all corrupted blocks. + + The value must be >= 1, and if the corrupted block has less mirrors than + the value, the mirror number will be `num % (num_mirrors + 1)`. + .. _man-rescue-clear-ino-cache: clear-ino-cache <device>
cmds/rescue-fix-data-checksum.c +12 -4
@@ -381,7 +381,8 @@ out: }static void report_corrupted_blocks (struct btrfs_fs_info *fs_info, -enum btrfs_fix_data_checksum_mode mode) +enum btrfs_fix_data_checksum_mode mode, +unsigned int mirror) {struct corrupted_block *entry;struct btrfs_path path = {0 };
@@ -393,7 +394,6 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info, }list_for_each_entry (entry, &corrupted_blocks, list) { -unsigned int mirror;bool has_printed = false;int ret;
@@ -426,6 +426,10 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info,case BTRFS_FIX_DATA_CSUMS_READONLY : action =ACTION_IGNORE ;break ; +case BTRFS_FIX_DATA_CSUMS_UPDATE_CSUM_ITEM : + action =ACTION_UPDATE_CSUM ; + mirror = mirror % (entry->num_mirrors +1 ); +break ;default :UASSERT (0 ); }
@@ -434,6 +438,7 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info,case ACTION_IGNORE :break ;case ACTION_UPDATE_CSUM : +UASSERT (mirror >0 && mirror <= entry->num_mirrors); ret =update_csum_item (fs_info, entry->logical, mirror);break ;default :
@@ -455,7 +460,8 @@ static void free_corrupted_blocks(void) }int btrfs_recover_fix_data_checksum (const char *path, -enum btrfs_fix_data_checksum_mode mode) +enum btrfs_fix_data_checksum_mode mode, +unsigned int mirror) {struct btrfs_fs_info *fs_info;struct btrfs_root *csum_root;
@@ -465,6 +471,8 @@ int btrfs_recover_fix_data_checksum(const char *path,if (mode >=BTRFS_FIX_DATA_CSUMS_LAST )return -EINVAL ; +if (mode ==BTRFS_FIX_DATA_CSUMS_UPDATE_CSUM_ITEM ) +UASSERT (mirror >0 ); ret =check_mounted (path);if (ret <0 ) { errno = -ret;
@@ -495,7 +503,7 @@ int btrfs_recover_fix_data_checksum(const char *path, errno = -ret;error ("failed to iterate csum tree: %m" ); } -report_corrupted_blocks (fs_info, mode); +report_corrupted_blocks (fs_info, mode, mirror); out_close:free_corrupted_blocks ();close_ctree_fs_info (fs_info);
cmds/rescue.c +14 -2
@@ -31,6 +31,7 @@#include "kernel-shared/extent_io.h" #include "kernel-shared/accessors.h" #include "kernel-shared/uapi/btrfs_tree.h" +#include "common/string-utils.h" #include "common/messages.h" #include "common/utils.h" #include "common/help.h"
@@ -282,6 +283,7 @@ static const char * const cmd_rescue_fix_data_checksum_usage[] = {"" ,OPTLINE ("-r|--readonly" ,"readonly mode, only report errors without repair" ),OPTLINE ("-i|--interactive" ,"interactive mode, ignore the error by default." ), +OPTLINE ("-m|--mirror <mirror>" ,"update csum item using specified mirror" ),HELPINFO_INSERT_GLOBALS ,HELPINFO_INSERT_VERBOSE ,NULL
@@ -291,6 +293,7 @@ static int cmd_rescue_fix_data_checksum(const struct cmd_struct *cmd, int argc, char **argv) { enum btrfs_fix_data_checksum_mode mode = BTRFS_FIX_DATA_CSUMS_READONLY; +unsigned int mirror =0 ;int ret; optind =0 ;
@@ -300,9 +303,10 @@ static int cmd_rescue_fix_data_checksum(const struct cmd_struct *cmd,static const struct option long_options [] = { {"readonly" , no_argument,NULL ,'r' }, {"interactive" , no_argument,NULL ,'i' }, + {"mirror" , required_argument,NULL ,'m' }, {"NULL" ,0 ,NULL ,0 }, }; - c =getopt_long (argc, argv,"ri" , long_options,NULL ); + c =getopt_long (argc, argv,"rim:" , long_options,NULL );if (c <0 )break ;switch (c) {
@@ -312,13 +316,21 @@ static int cmd_rescue_fix_data_checksum(const struct cmd_struct *cmd,case 'i' : mode =BTRFS_FIX_DATA_CSUMS_INTERACTIVE ;break ; +case 'm' : + mode =BTRFS_FIX_DATA_CSUMS_UPDATE_CSUM_ITEM ; + mirror =arg_strtou64 (optarg); +if (mirror ==0 ) { +error ("invalid mirror number %u, must be >= 1" , mirror); +return 1 ; + } +break ;default :usage_unknown_option (cmd, argv); } }if (check_argc_min (argc - optind,1 ))return 1 ; - ret =btrfs_recover_fix_data_checksum (argv[optind], mode); + ret =btrfs_recover_fix_data_checksum (argv[optind], mode, mirror);if (ret <0 ) { errno = -ret;error ("failed to fix data checksums: %m" );
cmds/rescue.h +3 -1
@@ -23,12 +23,14 @@enum btrfs_fix_data_checksum_mode {BTRFS_FIX_DATA_CSUMS_READONLY ,BTRFS_FIX_DATA_CSUMS_INTERACTIVE , +BTRFS_FIX_DATA_CSUMS_UPDATE_CSUM_ITEM ,BTRFS_FIX_DATA_CSUMS_LAST , };int btrfs_recover_superblocks (const char *path,int yes);int btrfs_recover_chunk_tree (const char *path,int yes);int btrfs_recover_fix_data_checksum (const char *path, -enum btrfs_fix_data_checksum_mode mode); +enum btrfs_fix_data_checksum_mode mode, +unsigned int mirror);#endif
Signature · 1 line
2.49.0
Quoted 63 lines · Qu Wenruo
[CHANGELOG]
v2:
- Rename the subcommand to "fix-data-checksum" It's better to use full name in the command name- Remove unused members inside
corrupted_block
The old @extent_bytenr and @extent_len is no longer needed, even for the future file-deletion action.- Fix the bitmap size off-by-1 bug
We must use the bit 0 to represent mirror 1, or the bitmap size will exceednum_mirrors.- Introduce -i|--interactive mode
Will ask the user for the action on the corrupted block, including:* Ignore
The default behavior if no command is provided* Use specified mirror to update the data checksum item The user must input a number inside range [1,
num_mirrors].- Introduce -m|--mirror <num> mode
Use specified mirror for all corrupted blocks.
The value <num> must be >= 1. And if the value is larger than the actual max mirror number, the real mirror number will benum % (num_mirror + 1).We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
Although the problem is worked around in v6.15 (and being backported), for the affected fs there is no good way to fix them, other than complex manually find out which files are affected and delete them.
This series introduce the initial implementation of "btrfs rescue fix-data-checksum", which is designed to fix such problem by either:
- Update the csum items using the data from specified copy
The subcommand has 3 modes so far:
- Readonly mode
Only report all corrupted blocks and affected files, no repair is done.- Interactive mode
Ask for what to do, including* Ignore (the default)
* Use certain mirror to update the checksum item- Mirror mode
Use specified mirror to update the checksum item, the batch mode of the interactive one.In the future, there will be one more mode:
- Delete mode
Delete all involved files.There are still some points to address before implementing this mode.
Quoted 7 lines
Qu Wenruo (6):
btrfs-progs: introduce "btrfs rescue fix-data-checksum"
btrfs-progs: fix a bug inbtrfs_find_item()
btrfs-progs: fix-data-checksum: show affected files
btrfs-progs: fix-data-checksum: introduce interactive mode
btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
btrfs-progs: fix-data-checksum: introduce -m|--mirror option
Thanks, this is certainly useful and is a reasonable compromise to have a way to fix the checksums. I did some conding style fixups in the devel branch, I'll comment under the patches.
cmds/rescue-fix-data-checksum.c line 205
Quoted 6 lines · Qu Wenruo
cmds/rescue-fix-data-checksum.c new file +6
-0+static void report_corrupted_blocks (void ) +{ +struct corrupted_block *entry; + +if (list_empty (&corrupted_blocks)) { +printf ("No data checksum mismatch found\n" );
printf(...) -> pr_verbose(LOG_DEFAULT, ...) so the verbosity options are respected. The semantics is the same as printf, so there's no implicit "\n" and you can glue the lines as you need.
Quoted 24 lines
cmds/rescue-fix-data-checksum.c new file +23
-0+return ; + } + +list_for_each_entry (entry, &corrupted_blocks, list) { +bool has_printed = false; + +printf ("logical=%llu corrtuped mirrors=" , entry->logical); +/* Poor man's bitmap print. */ +for (int i =0 ; i < entry->num_mirrors; i++) { +if (test_bit (i, entry->error_mirror_bitmap)) { +if (has_printed) +printf ("," ); +/* +* Bit 0 means mirror 1, thus we need to increase +* the value by 1. +*/ +printf ("%d" , i +1 ); + has_printed=true; + } + } +printf ("\n" ); + } +}
cmds/rescue-fix-data-checksum.c
Quoted 10 lines
cmds/rescue-fix-data-checksum.c new file +10
-0+static int cmd_rescue_fix_data_checksum (const struct cmd_struct *cmd, +int argc,char **argv) +{ +enum btrfs_fix_data_checksum_mode mode =BTRFS_FIX_DATA_CSUMS_READONLY ; +int ret; + optind =0 ; + +while (1 ) { +int c; +enum {GETOPT_VAL_DRYRUN =GETOPT_VAL_FIRST , };
The ending "," does not need to be there
cmds/rescue.c out: · line 302
Quoted 4 lines
cmds/rescue.c +4
-0+static const struct option long_options [] = { + {"readonly" , no_argument,NULL ,'r' }, + {"NULL" ,0 ,NULL ,0 }, + };
Missing newline
Quoted 21 lines
cmds/rescue.c +21
-0+ c =getopt_long (argc, argv,"r" , long_options,NULL ); +if (c <0 ) +break ; +switch (c) { +case 'r' : + mode =BTRFS_FIX_DATA_CSUMS_READONLY ; +break ; +default : +usage_unknown_option (cmd, argv); + } + } +if (check_argc_min (argc - optind,1 )) +return 1 ; + ret =btrfs_recover_fix_data_checksum (argv[optind], mode); +if (ret <0 ) { + errno = -ret; +error ("failed to fix data checksums: %m" ); + } +return !!ret; +} +static DEFINE_SIMPLE_COMMAND(rescue_fix_data_checksum, "fix-data-checksum");
Re: [PATCH v2 2/6] btrfs-progs: fix a bug in btrfs_find_item()
In reply to Qu WenruoQuoted 95 lines · Qu Wenruo
[BUG]
There is a seldomly utilized function,btrfs_find_item(), which has no document and is not behaving correctly.Inside
backref.c,iterate_inode_refs()andbtrfs_ref_to_path()both utilize this function, to find the parent inode.However
btrfs_find_item()will never return 0 if @ioff is passed as 0 for such usage, result early failure for all kinds of inode iteration functions.[CAUSE]
Both functions pass 0 as the @ioff parameter initially, e.g:We have the following fs tree root:
item 0 key (256
INODE_ITEM0) itemoff 16123 itemsize 160generation 3 transid 9 size 6 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 1 flags 0x0(none) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 index 0 namelen 2 name: .. item 2 key (256 DIR_ITEM 2507850652) itemoff 16078 itemsize 33 location key (257 INODE_ITEM 0) type FILE transid 9 data_len 0 name_len 3 name: foo item 3 key (256 DIR_INDEX 2) itemoff 16045 itemsize 33 location key (257 INODE_ITEM 0) type FILE transid 9 data_len 0 name_len 3 name: foo item 4 key (257 INODE_ITEM 0) itemoff 15885 itemsize 160 generation 9 transid 9 size 16384 nbytes 16384 block group 0 mode 100600 links 1 uid 0 gid 0 rdev 0 sequence 4 flags 0x0(none) item 5 key (257 INODE_REF 256) itemoff 15872 itemsize 13 index 2 namelen 3 name: foo item 6 key (257 EXTENT_DATA 0) itemoff 15819 itemsize 53 generation 9 type 1 (regular) extent data disk byte 13631488 nr 16384 extent data offset 0 nr 16384 ram 16384 extent compression 0 (none)Then we call
paths_from_inode()with:
- @inum = 257
- ipath = {.fs_root = 5}Then we got the following sequence:
iterate_irefs(257, fs_root, inode_to_path)
|-iterate_inode_refs()|- inode_ref_info() |- btrfs_find_item(257, 0, fs_root) | Returned 1, with @found_key updated to | (257, INODE_REF, 256).This makes
iterate_irefs()exit immediately, but obviously thatbtrfs_find_item()call is to find anyINODE_REF, not to find the exact match.[FIX]
Ifbtrfs_find_item()found an item matching the objectid and type, then it should return 0 other than 1.Fix it and keep the behavior the same across btrfs-progs and the kernel.
Since we're here, also add some comments explaining the function.
Signed-off-by Qu Wenruo <[email protected]>--- kernel-shared/ctree.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)kernel-shared/ctree.c +11
-0@@ -1246,6 +1246,17 @@ static void reada_for_search(struct btrfs_fs_info *fs_info, } } +/* +* Find the first key in @fs_root that matches all the following conditions: +* +* - key.obojectid == @iobjectid +* - key.type == @key_type +* - key.offset >= ioff +* +* Return 0 if such key can be found, and @found_key is updated. +* Return >0 if no such key can be found. +* Return <0 for critical errors. +*/ int btrfs_find_item (struct btrfs_root *fs_root,struct btrfs_path *found_path,u64 iobjectid,u64 ioff,u8 key_type,struct btrfs_key *found_key)
kernel-shared/ctree.c int btrfs_find_item(struct btrfs_root *fs_root, struct btrfs_path *found_path, · line 1294
Quoted 5 lines
kernel-shared/ctree.c +1
-1@@ -1280,10 +1291,10 @@ int btrfs_find_item(struct btrfs_root *fs_root, struct btrfs_path *found_path,btrfs_item_key_to_cpu (eb, found_key, path->slots[0 ]);if (found_key->type != key.type || - found_key->objectid != key.objectid) { + found_key->objectid != key.objectid)
This fits one line. We don't have perfect 1:1 line matching with kernel so the style can be fixed.
Quoted 11 lines
kernel-shared/ctree.c +2
-2ret =1 ; - goto out; - } + else + ret =0 ; out:if (path != found_path)Signature · 1 line
2.49.0
Re: [PATCH v2 3/6] btrfs-progs: fix-data-checksum: show affected files
In reply to Qu WenruoQuoted 50 lines · Qu Wenruo
Previously "btrfs rescue fix-data-checksum" only show affected logical bytenr, which is not helpful to determine which files are affected.
Enhance the output by also outputting the affected subvolumes (in numeric form), and the file paths inside that subvolume.
The example looks like this:
logical=13631488 corrtuped mirrors=1 affected files:
(subvolume 5)/foo
(subvolume 5)/dir/bar
logical=13635584 corrtuped mirrors=1 affected files:
(subvolume 5)/foo
(subvolume 5)/dir/barAlthough the end result is still not perfect, it's still much easier to find out which files are affected.
Signed-off-by Qu Wenruo <[email protected]>--- cmds/rescue-fix-data-checksum.c | 59 +++++++++++++++++++++++++++++++-- 1 file changed, 56 insertions(+), 3 deletions(-)cmds/rescue-fix-data-checksum.c +12
-0@@ -18,6 +18,7 @@#include "kernel-shared/disk-io.h" #include "kernel-shared/ctree.h" #include "kernel-shared/volumes.h" +#include "kernel-shared/backref.h" #include "common/messages.h" #include "common/open-utils.h" #include "cmds/rescue.h" @@ -158,6 +159,49 @@ static int iterate_one_csum_item(struct btrfs_fs_info *fs_info, struct btrfs_patreturn ret; } +static int print_filenames (u64 ino,u64 offset,u64 rootid,void *ctx) +{ +struct btrfs_fs_info *fs_info = ctx; +struct btrfs_root *root; +struct btrfs_key key; +struct inode_fs_paths *ipath; +struct btrfs_path path = {0 }; +int ret; + + key.objectid = rootid; + key.type =BTRFS_ROOT_ITEM_KEY ;
cmds/rescue-fix-data-checksum.c static int iterate_one_csum_item(struct btrfs_fs_info *fs_info, struct btrfs_pat · line 182
Quoted 10 lines
cmds/rescue-fix-data-checksum.c +10
-0+ key.offset = (u64)-1 ; + + root =btrfs_read_fs_root (fs_info, &key); +if (IS_ERR (root)) { + ret =PTR_ERR (root); + errno = -ret; +error ("failed to get subvolume %llu: %m" , rootid); +return ret; + } + ipath =init_ipath (128 *BTRFS_PATH_NAME_MAX , root, &path);
128 is a magic constant, I've left it like that but we can use the PATH_MAX and some named definition for the count.
Quoted 22 lines
cmds/rescue-fix-data-checksum.c +22
-0+if (IS_ERR (ipath)) { + ret =PTR_ERR (ipath); + errno = -ret; +error ("failed to initialize ipath: %m" ); +return ret; + } + ret =paths_from_inode (ino, ipath); +if (ret <0 ) { + errno = -ret; +error ("failed to resolve root %llu ino %llu to paths: %m" , rootid, ino); + goto out; + } +for (int i =0 ; i < ipath->fspath->elem_cnt; i++) +printf (" (subvolume %llu)/%s\n" , rootid, (char *)ipath->fspath->val[i]); +if (ipath->fspath->elem_missed) +printf (" (subvolume %llu) %d files not printed\n" , rootid, + ipath->fspath->elem_missed); +out: +free_ipath (ipath); +return ret; +} +
Re: [PATCH v2 4/6] btrfs-progs: fix-data-checksum: introduce interactive mode
In reply to Qu Wenruocmds/rescue-fix-data-checksum.c line 18
Quoted 8 lines · Qu Wenruo
cmds/rescue-fix-data-checksum.c +1
-0@@ -14,6 +14,7 @@* Boston, MA 021110-1307, USA. */ +#include <ctype.h> #include "kerncompat.h"
The kerncompat.h needs to be first as there are workarounds for some system includes. This has been unified everywhere else so please stick to the ordering.
Quoted 31 lines
cmds/rescue-fix-data-checksum.c +15
-0#include "kernel-shared/disk-io.h" #include "kernel-shared/ctree.h" @@ -45,6 +46,21 @@ struct corrupted_block {unsigned long *error_mirror_bitmap; }; +enum fix_data_checksum_action_value { +ACTION_IGNORE , +ACTION_LAST , +}; + +static const struct fix_data_checksum_action { +enum fix_data_checksum_action_value value; +const char *string; +} actions[] = { + [ACTION_IGNORE ] = { + .value =ACTION_IGNORE , + .string ="ignore" , + }, +}; +static int global_repair_mode;LIST_HEAD (corrupted_blocks);@@ -241,10 +257,49 @@ next:return ret; }@@ -277,6 +332,16 @@ static void report_corrupted_blocks(struct btrfs_fs_info *fs_info)error ("failed to iterate involved files: %m" );break ;
cmds/rescue-fix-data-checksum.c static void report_corrupted_blocks(struct btrfs_fs_info *fs_info) · line 343
Quoted 10 lines
cmds/rescue-fix-data-checksum.c +9
-0} +switch (mode) { +case BTRFS_FIX_DATA_CSUMS_INTERACTIVE : + action =ask_action (); +UASSERT (action ==ACTION_IGNORE ); + fallthrough; +case BTRFS_FIX_DATA_CSUMS_READONLY : +break ; +default : +UASSERT (0 );
I haven't fixed that but please don't do the ASSERT(0), we have INTERNAL_ERROR for runtime errors or handle it and return with a hard exit.
Quoted 3 lines
cmds/rescue-fix-data-checksum.c +1
-0+ } } }
Re: [PATCH v2 5/6] btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
In reply to Qu WenruoQuoted 5 lines · Qu Wenruo
cmds/rescue-fix-data-checksum.c +5
-0+static int update_csum_item (struct btrfs_fs_info *fs_info,u64 logical, +unsigned int mirror) +{ +struct btrfs_trans_handle *trans; +struct btrfs_root *csum_root =btrfs_csum_root (fs_info, logical);
cmds/rescue-fix-data-checksum.c again: · line 343
Quoted 10 lines
cmds/rescue-fix-data-checksum.c +10
-0+struct btrfs_path path = {0 }; +struct btrfs_csum_item *citem; +u64 read_len = fs_info->sectorsize; +u8 csum[BTRFS_CSUM_SIZE ] = {0 }; +u8 *buf; +int ret; + + buf =malloc (fs_info->sectorsize); +if (!buf) +return -ENOMEM ;
Not fixed, but the block buffers can be on stack as well if they're just for the single function, the size is bounded.
Quoted 40 lines
cmds/rescue-fix-data-checksum.c +37
-0+ ret =read_data_from_disk (fs_info, buf, logical, &read_len, mirror); +if (ret <0 ) { + errno = -ret; +error ("failed to read block at logical %llu mirror %u: %m" , + logical, mirror); + goto out; + } + trans =btrfs_start_transaction (csum_root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error_msg (ERROR_MSG_START_TRANS ,"%m" ); + goto out; + } + citem =btrfs_lookup_csum (trans, csum_root, &path, logical, +BTRFS_EXTENT_CSUM_OBJECTID , fs_info->csum_type,1 ); +if (IS_ERR (citem)) { + ret =PTR_ERR (citem); + errno = -ret; +error ("failed to find csum item for logical %llu: $m" , logical); +btrfs_abort_transaction (trans, ret); + goto out; + } +btrfs_csum_data (fs_info, fs_info->csum_type, buf, csum, fs_info->sectorsize); +write_extent_buffer (path.nodes[0 ], csum, (unsigned long )citem, fs_info->csum_size); +btrfs_release_path (&path); + ret =btrfs_commit_transaction (trans, csum_root); +if (ret <0 ) { + errno = -ret; +error_msg (ERROR_MSG_COMMIT_TRANS ,"%m" ); + } +printf ("Csum item for logical %llu updated using data from mirror %u\n" , + logical, mirror); +out: +free (buf); +btrfs_release_path (&path); +return ret; } static void report_corrupted_blocks(struct btrfs_fs_info *fs_info,
Re: [PATCH v2 5/6] btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
Quoted 11 lines · David Sterba
Quoted 10 lines · Qu Wenruo
cmds/rescue-fix-data-checksum.c +10
-0+static int update_csum_item (struct btrfs_fs_info *fs_info,u64 logical, +unsigned int mirror) +{ +struct btrfs_trans_handle *trans; +struct btrfs_root *csum_root =btrfs_csum_root (fs_info, logical); +struct btrfs_path path = {0 }; +struct btrfs_csum_item *citem; +u64 read_len = fs_info->sectorsize; +u8 csum[BTRFS_CSUM_SIZE ] = {0 }; +u8 *buf;
Quoted 8 lines
cmds/rescue-fix-data-checksum.c again: · line 343
Quoted 5 lines
cmds/rescue-fix-data-checksum.c +5
-0+int ret; + + buf =malloc (fs_info->sectorsize); +if (!buf) +return -ENOMEM ;Not fixed, but the block buffers can be on stack as well if they're just for the single function, the size is bounded.
Unfortunately the sectorsize is still runtime determined, thus it can not be on-stack.
Thanks,
Qu>
Quoted 40 lines
Quoted 40 lines
cmds/rescue-fix-data-checksum.c +37
-0+ ret =read_data_from_disk (fs_info, buf, logical, &read_len, mirror); +if (ret <0 ) { + errno = -ret; +error ("failed to read block at logical %llu mirror %u: %m" , + logical, mirror); + goto out; + } + trans =btrfs_start_transaction (csum_root,1 ); +if (IS_ERR (trans)) { + ret =PTR_ERR (trans); + errno = -ret; +error_msg (ERROR_MSG_START_TRANS ,"%m" ); + goto out; + } + citem =btrfs_lookup_csum (trans, csum_root, &path, logical, +BTRFS_EXTENT_CSUM_OBJECTID , fs_info->csum_type,1 ); +if (IS_ERR (citem)) { + ret =PTR_ERR (citem); + errno = -ret; +error ("failed to find csum item for logical %llu: $m" , logical); +btrfs_abort_transaction (trans, ret); + goto out; + } +btrfs_csum_data (fs_info, fs_info->csum_type, buf, csum, fs_info->sectorsize); +write_extent_buffer (path.nodes[0 ], csum, (unsigned long )citem, fs_info->csum_size); +btrfs_release_path (&path); + ret =btrfs_commit_transaction (trans, csum_root); +if (ret <0 ) { + errno = -ret; +error_msg (ERROR_MSG_COMMIT_TRANS ,"%m" ); + } +printf ("Csum item for logical %llu updated using data from mirror %u\n" , + logical, mirror); +out: +free (buf); +btrfs_release_path (&path); +return ret; } static void report_corrupted_blocks(struct btrfs_fs_info *fs_info,
Quoted 22 lines · Qu Wenruo
[CHANGELOG]
v2:
- Rename the subcommand to "fix-data-checksum" It's better to use full name in the command name- Remove unused members inside
corrupted_block
The old @extent_bytenr and @extent_len is no longer needed, even for the future file-deletion action.- Fix the bitmap size off-by-1 bug
We must use the bit 0 to represent mirror 1, or the bitmap size will exceednum_mirrors.- Introduce -i|--interactive mode
Will ask the user for the action on the corrupted block, including:* Ignore
The default behavior if no command is provided* Use specified mirror to update the data checksum item The user must input a number inside range [1,
num_mirrors].
Quoted 9 lines
- Introduce -m|--mirror <num> mode
Use specified mirror for all corrupted blocks.
The value <num> must be >= 1. And if the value is larger than the actual max mirror number, the real mirror number will benum % (num_mirror + 1).We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
What about having an ioctl (on a mounted fs) which allow us to read data from a file even if the csum doesn't match ? I asked that because the problem usually happen on a specific file and not to an entire filesystem. In this case I think that it would be more practical to read the block using the IOCTL, and then rewrite the block, at the specific offset (to update the checksum).
Of course there are several tradeoff: the "unmounted" version, doesn't duplicate a shared block, where my approach (read the data using an ioctl to avoid the CSUM mismatch and rewrite it) make a fork if the block is shared. However, as told before, the problem is related to specifics file and it seems a waste of resource reading an entire filesystem to correct few files. No to mention that the IOCTL can be done on a "live" filesystem.
Said that, of course "btrfs rescue fix-data-checksum" is better than nothing.
BR
G.Baroncelli
Quoted 55 lines
Although the problem is worked around in v6.15 (and being backported), for the affected fs there is no good way to fix them, other than complex manually find out which files are affected and delete them.
This series introduce the initial implementation of "btrfs rescue fix-data-checksum", which is designed to fix such problem by either:
- Update the csum items using the data from specified copy
The subcommand has 3 modes so far:
- Readonly mode
Only report all corrupted blocks and affected files, no repair is done.- Interactive mode
Ask for what to do, including* Ignore (the default)
* Use certain mirror to update the checksum item- Mirror mode
Use specified mirror to update the checksum item, the batch mode of the interactive one.In the future, there will be one more mode:
- Delete mode
Delete all involved files.There are still some points to address before implementing this mode.
Qu Wenruo (6):
btrfs-progs: introduce "btrfs rescue fix-data-checksum"
btrfs-progs: fix a bug inbtrfs_find_item()
btrfs-progs: fix-data-checksum: show affected files
btrfs-progs: fix-data-checksum: introduce interactive mode
btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
btrfs-progs: fix-data-checksum: introduce -m|--mirror optionDocumentation/btrfs-rescue.rst | 28 ++ Makefile | 2 +- cmds/rescue-fix-data-checksum.c | 511 ++++++++++++++++++++++++++++++++ cmds/rescue.c | 65 ++++ cmds/rescue.h | 10 + kernel-shared/ctree.c | 17 +- kernel-shared/file-item.c | 2 +- kernel-shared/file-item.h | 5 + 8 files changed, 635 insertions(+), 5 deletions(-)create mode 100644
cmds/rescue-fix-data-checksum.c--
2.49.0
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 29 lines · Goffredo Baroncelli
Quoted 28 lines · Qu Wenruo
[CHANGELOG]
v2:
- Rename the subcommand to "fix-data-checksum" It's better to use full name in the command name- Remove unused members inside
corrupted_block
The old @extent_bytenr and @extent_len is no longer needed, even for the future file-deletion action.- Fix the bitmap size off-by-1 bug
We must use the bit 0 to represent mirror 1, or the bitmap size will exceednum_mirrors.- Introduce -i|--interactive mode
Will ask the user for the action on the corrupted block, including:* Ignore
The default behavior if no command is provided* Use specified mirror to update the data checksum item The user must input a number inside range [1,
num_mirrors].- Introduce -m|--mirror <num> mode
Use specified mirror for all corrupted blocks.
The value <num> must be >= 1. And if the value is larger than the actual max mirror number, the real mirror number will benum % (num_mirror + 1).
Quoted 5 lines
Quoted 3 lines
We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
What about having an ioctl (on a mounted fs) which allow us to read data from a file even if the csum doesn't match ?
The problem is again with mirrors.
Unless you ask the end user to provide a mirror number, for a with multiple mirrors, and the mirrors doesn't match each other, the behavior will be a mess.
That's why I'm planning to add something like a mirror priority, with a new mirror "best" to try use the mirror with the most matches.
Despite the mirror number problem (we need to ask the end user for mirror number), I think it's possible to implement the feature (mostly) in user space.
E.g. combine fiemap result with btrfs-map-logical, then read from the disk directly.
Quoted 3 lines
I asked that because the problem usually happen on a specific file
and not to an entire filesystem. In this case I think that it would be more practical to read the block using the IOCTL, and then rewrite the block, at the specific offset (to update the checksum).
I'm fine with the idea of reading from raw data idea (although prefer to implement it in user space), but not a huge fan to simply re-write with COW.
E.g. the bad csum is still there, can still be exposed by scrub, even it's not referred by any file anymore.
Quoted 6 lines
Of course there are several tradeoff: the "unmounted" version, doesn't duplicate a shared block, where my approach (read the data using an ioctl to avoid the CSUM mismatch and rewrite it) make a fork if the block is shared. However, as told before, the problem is related to specifics file and it seems a waste of resource reading an entire filesystem to correct few files. No to mention that the IOCTL can be done on a "live" filesystem.
I do not think we should treat the csum error so easily, it's still a big problem.
Even it's known that direct IO with buffer modified halfway can lead to corruption, the csum mismatch is still a huge problem. The end user should check if their problem is properly written, or use the latest kernel with proper backport.
If it's really caused by hardware (memory or disk or whatever), it's a huge deal and definitely needs proper inspection and verification.
So overall, if one hits a csum mismatch, especially after the direct IO fix, then it should not be treated as "something that can be easily fixed online", it should be something huge, at least needs some tool to handle it offline.
Thanks,
Qu
Quoted 63 lines
Said that, of course "btrfs rescue fix-data-checksum" is better than nothing.
BR
G.BaroncelliQuoted 55 lines
Although the problem is worked around in v6.15 (and being backported), for the affected fs there is no good way to fix them, other than complex manually find out which files are affected and delete them.
This series introduce the initial implementation of "btrfs rescue fix-data-checksum", which is designed to fix such problem by either:
- Update the csum items using the data from specified copy
The subcommand has 3 modes so far:
- Readonly mode
Only report all corrupted blocks and affected files, no repair is done.- Interactive mode
Ask for what to do, including* Ignore (the default)
* Use certain mirror to update the checksum item- Mirror mode
Use specified mirror to update the checksum item, the batch mode of the interactive one.In the future, there will be one more mode:
- Delete mode
Delete all involved files.There are still some points to address before implementing this mode.
Qu Wenruo (6):
btrfs-progs: introduce "btrfs rescue fix-data-checksum"
btrfs-progs: fix a bug inbtrfs_find_item()
btrfs-progs: fix-data-checksum: show affected files
btrfs-progs: fix-data-checksum: introduce interactive mode
btrfs-progs: fix-data-checksum: update csum items to fix csum mismatch
btrfs-progs: fix-data-checksum: introduce -m|--mirror optionDocumentation/btrfs-rescue.rst | 28 ++
Makefile | 2 +-cmds/rescue-fix-data-checksum.c| 511 ++++++++++++++++++++++++++++++++ cmds/rescue.c | 65 ++++
cmds/rescue.h | 10 + kernel-shared/ctree.c | 17 +- kernel-shared/file-item.c | 2 +- kernel-shared/file-item.h | 5 +8 files changed, 635 insertions(+), 5 deletions(-)create mode 100644
cmds/rescue-fix-data-checksum.cSignature · 1 line
2.49.0
Quoted 4 lines · Qu Wenruo
Quoted 1 line · Goffredo Baroncelli
On 15/05/2025 10.00, Qu Wenruo wrote:
[...]
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
What about having an ioctl (on a mounted fs) which allow us to read data from a file even if the csum doesn't match ?
The problem is again with mirrors.
Quoted 9 lines
Unless you ask the end user to provide a mirror number, for a with multiple mirrors, and the mirrors doesn't match each other, the behavior will be a mess.
That's why I'm planning to add something like a mirror priority, with a new mirror "best" to try use the mirror with the most matches.
Despite the mirror number problem (we need to ask the end user for mirror number), I think it's possible to implement the feature (mostly) in user space.
E.g. combine fiemap result with btrfs-map-logical, then read from the disk directly.
This would expose to the user space the complexity of the filesystem; this means having two code path doing the same thing in two subtle different ways. Pay attention to consider also a raid5/6 case with a missing disks.
I don't know how important is select a valid mirror. Or at least, I don't see a realistic case where a mirror may be better than another one.
After your consideration, I would like to suggest the following:
1) read the data from a mirror, the csum matches, return data
2) read the data from a mirror, the csum DOES NOT match:
2.1) read the data from another mirror(s), the csum matches,
- rebuild the csum
- return the data
2.2) read the data from another mirror(s), the csum still doesn't match, return error 2.3) (NEW, alternatively to 2.2) read the data from another mirror(s), the csum still doesn't match:
- take the latest read attempt as valid
- rebuild the csum
- return the last read attempt
2.3) mode should be a enabled by an IOCTL on a specific fd. The assumption here is that if the csum doesn't match any mirror may be equally bad/good. But in the IOCTL we could add a parameter to specify the order of the mirrors (still we need a way to specify the order of the mirror in the raid5 case).
Quoted 8 lines
Quoted 3 lines
I asked that because the problem usually happen on a specific file
and not to an entire filesystem. In this case I think that it would be more practical to read the block using the IOCTL, and then rewrite the block, at the specific offset (to update the checksum).I'm fine with the idea of reading from raw data idea (although prefer to implement it in user space), but not a huge fan to simply re-write with COW.
E.g. the bad csum is still there, can still be exposed by scrub, even it's not referred by any file anymore.
If I understood correctly, you are saying that due to the fact that even if an extent is partially referred, all data in the extent is tracked and has his checksum. A rewriting in the middle would generate a new extent with a new checksum, but the old data still exists in the extent with his dedicated (and un-matching ) csum. (I repeated this because it seems to me a technical detail not so obvious but very important)
Quoted 7 lines
Quoted 6 lines
Of course there are several tradeoff: the "unmounted" version, doesn't duplicate a shared block, where my approach (read the data using an ioctl to avoid the CSUM mismatch and rewrite it) make a fork if the block is shared. However, as told before, the problem is related to specifics file and it seems a waste of resource reading an entire filesystem to correct few files. No to mention that the IOCTL can be done on a "live" filesystem.
Quoted 9 lines
I do not think we should treat the csum error so easily, it's still a big problem.
Even it's known that direct IO with buffer modified halfway can lead to corruption, the csum mismatch is still a huge problem. The end user should check if their problem is properly written, or use the latest kernel with proper backport.
If it's really caused by hardware (memory or disk or whatever), it's a huge deal and definitely needs proper inspection and verification.
So overall, if one hits a csum mismatch, especially after the direct IO fix, then it should not be treated as "something that can be easily fixed online", it should be something huge, at least needs some tool to handle it offline.
Even tough I agree about the severity (e.g. the corruption happened due to an hw error), I less agree about the fact that this is a justification to not have a simpler interface (form an user interface POV) that don't requires an un-mount/mount of the filesystem.
Un-mounting a root filesystem is a not so easy operation, so we should provide a different way to access the data.
However I agree that we should allow the user to select a different "mirror", considering that a "mirror" may be any valid combination of disk to rebuild the data (e.g. when we will define the interface, we should consider raid 6).
Quoted 4 lines
Thanks,
QuQuoted 1 line
BR
Goffredo
Signature · 2 lines
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it> Key fingerprint BBF5 1610 0B64 DAC6 5F7D 17B2 0EDA 9B37 8B82 E0B5
Quoted 16 lines · Goffredo Baroncelli
Quoted 4 lines · Qu Wenruo
Quoted 1 line · Goffredo Baroncelli
On 15/05/2025 10.00, Qu Wenruo wrote:
[...]
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
We have a long history of data csum mismatch, caused by direct IO and buffered being modified during writeback.
What about having an ioctl (on a mounted fs) which allow us to read data from a file even if the csum doesn't match ?
The problem is again with mirrors.
Unless you ask the end user to provide a mirror number, for a with multiple mirrors, and the mirrors doesn't match each other, the behavior will be a mess.
Quoted 8 lines
Quoted 7 lines
That's why I'm planning to add something like a mirror priority, with a new mirror "best" to try use the mirror with the most matches.
Despite the mirror number problem (we need to ask the end user for mirror number), I think it's possible to implement the feature (mostly) in user space.
E.g. combine fiemap result with btrfs-map-logical, then read from the disk directly.
This would expose to the user space the complexity of the filesystem;
Let me be clear, the filesystem is complex.
If there is a csum mismatch, the end user should dig into the rabbit hole to find out why, fixing it with some tool should not be the first thing they want to do.
Quoted 1 line
this means having two code path doing the same thing in two subtle different ways.
It's fine to have two code paths doing the same thing.
One for offline and one for online.
Quoted 3 lines
Pay attention to consider also a raid5/6 case with a missing disks.
I don't know how important is select a valid mirror. Or at least, I don't see a realistic case where a mirror may be better than another one.
RAID1C3, RAID1C4, where one can have 2 or more mirrors matches each other but not matching the csum.
Quoted 15 lines
After your consideration, I would like to suggest the following:
1) read the data from a mirror, the csum matches, return data
2) read the data from a mirror, the csum DOES NOT match:
2.1) read the data from another mirror(s), the csum matches,
- rebuild the csum
- return the data
2.2) read the data from another mirror(s), the csum still doesn't
match, return error
2.3) (NEW, alternatively to 2.2) read the data from another
mirror(s), the csum still doesn't match:
- take the latest read attempt as valid
- rebuild the csum
- return the last read attempt
Quoted 1 line
2.3) mode should be a enabled by an IOCTL on a specific fd.
Again, we should not even put a complex csum repair into kernel mode and allow it to be done online.
Back to the first point, csum mismatch should be investigated, it's not something one should hit commonly, and we should not make fixing it easier either.
Csum mismatch should be treated as tree-checker errors.
I have not yet seen anyone suggesting there should be some fancy ioctl to fix a tree-checker error.
Then there should not be any easy tool to fix csum error online.
Quoted 6 lines
The assumption here is that if the csum doesn't match any mirror may be equally bad/good. But in the IOCTL we could add a parameter to specify the order of the mirrors (still we need a way to specify the order of the mirror in the raid5 case).
Quoted 4 lines
Quoted 3 lines
I asked that because the problem usually happen on a specific file
and not to an entire filesystem. In this case I think that it would be more practical to read the block using the IOCTL, and then rewrite the block, at the specific offset (to update the checksum).
Quoted 6 lines
Quoted 4 lines
I'm fine with the idea of reading from raw data idea (although prefer to implement it in user space), but not a huge fan to simply re-write with COW.
E.g. the bad csum is still there, can still be exposed by scrub, even it's not referred by any file anymore.
If I understood correctly, you are saying that due to the fact that even if an extent is partially referred, all data in the extent is tracked and has his checksum. A rewriting in the middle would generate a new extent with a new checksum, but the old data still exists in the extent with his dedicated (and un-matching ) csum. (I repeated this because it seems to me a technical detail not so obvious but very important)
Yes.
Quoted 6 lines
Quoted 2 lines
So overall, if one hits a csum mismatch, especially after the direct IO fix, then it should not be treated as "something that can be easily fixed online", it should be something huge, at least needs some tool to handle it offline.
Even tough I agree about the severity (e.g. the corruption happened due to an hw error), I less agree about the fact that this is a justification to not have a simpler interface (form an user interface POV) that don't requires an un-mount/mount of the filesystem.
Un-mounting a root filesystem is a not so easy operation, so we should provide a different way to access the data.
If you can not trust your hardware, online repair is just going to cause more problems.
Again, replace csum error with tree-checker.
Thanks,
Qu
Quoted 11 lines
However I agree that we should allow the user to select a different "mirror", considering that a "mirror" may be any valid combination of disk to rebuild the data (e.g. when we will define the interface, we should consider raid 6).
Quoted 4 lines
Thanks,
QuQuoted 1 line
BR
Goffredo