btrfs-progs: check and tune: add device and noscan options
Subsystems (2)
cover btrfs-progs: check and tune: add device and noscan options
By default, btrfstune and btrfs check scans all and only the block devices in the system.
To scan regular files without mapping them to a loop device, adds the --device option.
To indicate not to scan the system for other devices, adds the --noscan option.
For example:
The command below will scan both regular files and the devices provided in the --device option, along with the system block devices.
btrfstune -m --device /tdev/td1,/tdev/td2 /tdev/td3
or
btrfs check --device /tdev/td1 --device /tdev/td2 /tdev/td3In some cases, if you need to avoid the default system scan for the block device, you can use the --noscan option.
btrfstune -m --noscan --device /tdev/td1,/tdev/td2 /tdev/td3
btrfs check --noscan --device /tdev/td1,/tdev/td2 /tdev/td3 This patch bundle depends on the preparatory patch bundle sent before:
[PATCH 0/6 v3] btrfs-progs: cleanup and preparatory around device scan
And, replaces [1] in the mailing list, as the --device option helper function is peeled and transformed into a common helper function in a separate patch.
[1] [PATCH 0/4] btrfs-progs: tune: add --device and --noscan option
Anand Jain (10):
btrfs-progs: common: add --device option helpers
btrfs-progs: tune: consolidate return goto free-out
btrfs-progs: tune: introduce --device option
btrfs-progs: docs: update btrfstune --device option
btrfs-progs: tune: introduce --noscan option
btrfs-progs: docs: update btrfstune --noscan option
btrfs-progs: check: introduce --device option
btrfs-progs: docs: update btrfs check --device option
btrfs-progs: check: introduce --noscan option
btrfs-progs: docs: update btrfs check --noscan option
Documentation/btrfs-check.rst | 6 +++ Documentation/btrfstune.rst | 7 ++++ check/main.c | 45 +++++++++++++++++++++- common/device-scan.c | 32 ++++++++++++++++ common/device-scan.h | 2 + tune/main.c | 71 ++++++++++++++++++++++++++++------- 6 files changed, 148 insertions(+), 15 deletions(-)
Signature · 1 line
2.31.1
PATCH 1/10 btrfs-progs: common: add --device option helpers
Preparatory patch adds two helper functions: array_append() and free_array(), which facilitate reading the device list provided at the --device option.
--- common/device-scan.c | 32 ++++++++++++++++++++++++++++++++ common/device-scan.h | 2 ++ 2 files changed, 34 insertions(+)
common/device-scan.c +32 -0
@@ -31,6 +31,7 @@#include <dirent.h> #include <limits.h> #include <stdbool.h> +#include <ctype.h> #include <blkid/blkid.h> #include <uuid/uuid.h> #ifdef HAVE_LIBUDEV
@@ -540,3 +541,34 @@ int btrfs_scan_argv_devices(int dev_optind, int dev_argc, char **dev_argv)return 0 ; } + +bool array_append (char **dest,char *src,int *cnt) +{ +char *this_tok =strtok (src,"," ); +int ret_cnt = *cnt; + +while (this_tok !=NULL ) { + ret_cnt++; + dest =realloc (dest,sizeof (char *) * ret_cnt); +if (!dest) +return false; + + dest[ret_cnt -1 ] =strdup (this_tok); + *cnt = ret_cnt; + + this_tok =strtok (NULL ,"," ); + } + +return true; +} + +void free_array (char **prt,int cnt) +{ +if (!prt) +return ; + +for (int i =0 ; i < cnt; i++) +free (prt[i]); + +free (prt); +}
common/device-scan.h +2 -0
@@ -59,5 +59,7 @@ int add_seen_fsid(u8 *fsid, struct seen_fsid *seen_fsid_hash[],void free_seen_fsid (struct seen_fsid *seen_fsid_hash[]);int test_uuid_unique (const char *uuid_str);int btrfs_scan_argv_devices (int dev_optind,int argc,char **argv); +bool array_append (char **dest,char *src,int *cnt); +void free_array (char **prt,int cnt);#endif
Signature · 1 line
2.31.1
PATCH 2/10 btrfs-progs: tune: consolidate return goto free-out
In reply to Anand JainThe upcoming "--device" option requires memory to parse devices, which should be freed before returning from the main() function. As a preparation for adding the "--device" option to the "btrfstune" command, provide a consolidated error return exit from the main function with a "goto" labeled "free_out". The label "free_out" may not make sense currently, but it will when the "--device" option is added.
There are several return statements within the main function, and changing all of them in the main "--device" feature patch would deviate from the actual for the feature changes. Hence, it made sense to create a preparatory patch.
The return value for any failure remains the same as in the original code.
--- tune/main.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
tune/main.c +17 -10
@@ -145,7 +145,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])bool to_fst = false;int csum_type =-1 ;char *new_fsid_str =NULL ; -int ret; +int ret =1 ;u64 super_flags =0 ;int fd =-1 ;
@@ -233,18 +233,18 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])set_argv0 (argv); device = argv[optind];if (check_argc_exact (argc - optind,1 )) -return 1 ; + goto free_out;if (random_fsid && new_fsid_str) {error ("random fsid can't be used with specified fsid" ); -return 1 ; + goto free_out; }if (!super_flags && !seeding_flag && !(random_fsid || new_fsid_str) && !change_metadata_uuid && csum_type ==-1 && !to_bg_tree && !to_extent_tree && !to_fst) {error ("at least one option should be specified" );usage (&tune_cmd,1 ); -return 1 ; + goto free_out; }if (new_fsid_str) {
@@ -253,18 +253,21 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) ret =uuid_parse (new_fsid_str, tmp);if (ret <0 ) {error ("could not parse UUID: %s" , new_fsid_str); -return 1 ; + ret =1 ; + goto free_out; }if (!test_uuid_unique (new_fsid_str)) {error ("fsid %s is not unique" , new_fsid_str); -return 1 ; + ret =1 ; + goto free_out; } } fd =open (device,O_RDWR );if (fd <0 ) {error ("mount check: cannot open %s: %m" , device); -return 1 ; + ret =1 ; + goto free_out; } ret = check_mounted_where(fd, device, NULL, 0, NULL,
@@ -273,18 +276,21 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) errno = -ret;error ("could not check mount status of %s: %m" , device);close (fd); -return 1 ; + ret =1 ; + goto free_out; }else if (ret) {error ("%s is mounted" , device);close (fd); -return 1 ; + ret =1 ; + goto free_out; } root =open_ctree_fd (fd, device,0 , ctree_flags);if (!root) {error ("open ctree failed" ); -return 1 ; + ret =1 ; + goto free_out; }if (to_bg_tree) {
@@ -431,5 +437,6 @@ out:close_ctree (root);btrfs_close_all_devices (); +free_out:return ret; }
Signature · 1 line
2.31.1
PATCH 3/10 btrfs-progs: tune: introduce --device option
In reply to Anand JainAs of now, btrfstune only accepts one device from the command line and then scans the system to find other parter devices if any.
However, this method mandates always accessing the file raw image as a loop device.
This patch modifies btrfstune to accept other devices or reg-files from the command line using an option --device and scans/registers them.
For example:
btrfstune -m --device /tdev/td1,/tdev/td2 /tdev/td3 or btrfstune -m --device /tdev/td1 --device /tdev/td2 /tdev/td3
--- tune/main.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-)
tune/main.c +33 -1
@@ -23,6 +23,7 @@#include <getopt.h> #include <errno.h> #include <stdbool.h> +#include <ctype.h> #include <uuid/uuid.h> #include "kernel-shared/ctree.h" #include "kernel-shared/disk-io.h"
@@ -117,6 +118,7 @@ static const char * const tune_usage[] = {"" ,"General:" ,OPTLINE ("-f" ,"allow dangerous operations, make sure that you are aware of the dangers" ), +OPTLINE ("--device" ,"devices or regular-files of the filesystem to be scanned" ),OPTLINE ("--help" ,"print this help" ), #if EXPERIMENTAL "",
@@ -144,6 +146,8 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])bool to_bg_tree = false;bool to_fst = false;int csum_type =-1 ; +int argc_devices =0 ; +char **argv_devices =NULL ;char *new_fsid_str =NULL ;int ret =1 ;u64 super_flags =0 ;
@@ -155,7 +159,8 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])enum {GETOPT_VAL_CSUM =GETOPT_VAL_FIRST ,GETOPT_VAL_ENABLE_BLOCK_GROUP_TREE ,GETOPT_VAL_DISABLE_BLOCK_GROUP_TREE , -GETOPT_VAL_ENABLE_FREE_SPACE_TREE }; +GETOPT_VAL_ENABLE_FREE_SPACE_TREE , +GETOPT_VAL_DEVICE };static const struct option long_options[] = { {"help" , no_argument,NULL ,GETOPT_VAL_HELP }, {"convert-to-block-group-tree" , no_argument,NULL ,
@@ -167,6 +172,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) #if EXPERIMENTAL { "csum", required_argument, NULL, GETOPT_VAL_CSUM }, #endif + { "device", required_argument, NULL, GETOPT_VAL_DEVICE }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "S:rxfuU:nmM:", long_options, NULL);
@@ -210,6 +216,21 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])case GETOPT_VAL_ENABLE_BLOCK_GROUP_TREE : to_bg_tree = true;break ; +case GETOPT_VAL_DEVICE : +if (!argv_devices) { + argv_devices =malloc (sizeof (char *)); +if (!argv_devices) { +error ("memory alloc failed" ); +return 1 ; + } + } + +if (!array_append (argv_devices, optarg, + &argc_devices)) { +error ("memory alloc failed" ); + goto free_out; + } +break ;case GETOPT_VAL_DISABLE_BLOCK_GROUP_TREE : to_extent_tree = true;break ;
@@ -285,6 +306,16 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) goto free_out; } +/* +* check_mounted_where() with noscan == true frees the scanned devices +* scan the command line provided device list now. +*/ +if (argv_devices) { + ret =btrfs_scan_argv_devices (0 , argc_devices, argv_devices); +if (ret) + goto free_out; + } + root =open_ctree_fd (fd, device,0 , ctree_flags);if (!root) {
@@ -438,5 +469,6 @@ out:btrfs_close_all_devices (); free_out: +free_array (argv_devices, argc_devices);return ret; }
Signature · 1 line
2.31.1
PATCH 4/10 btrfs-progs: docs: update btrfstune --device option
In reply to Anand JainUpdate the Documentation/btrfstune.rst to carry the new --device option.
--- Documentation/btrfstune.rst | 3 +++ 1 file changed, 3 insertions(+)
Documentation/btrfstune.rst +3 -0
@@ -46,6 +46,9 @@ OPTIONS Allow dangerous changes, e.g. clear the seeding flag or change fsid. Make sure that you are aware of the dangers. +--device + List of block devices or regular files that are part of the filesystem. + -m (since kernel: 5.0)
Signature · 1 line
2.31.1
PATCH 5/10 btrfs-progs: tune: introduce --noscan option
In reply to Anand JainThe function check_where_mounted() scans the system for other btrfs devices, but in certain cases, we may need a way to instruct btrfstune not to perform the system scan and instead only work on the devices provided through the command line. And so, add an option --noscan.
For example:
$ mkfs.btrfs -fq -draid0 -mraid0 ./td1 ./td2
$ btrfstune -m ./td1 warning, device 2 is missing ERROR: could not setup extent tree ERROR: open ctree failed
$ losetup --find --show ./td2 /dev/loop4 $ btrfstune -m ./td1
Or just
$ btrfstune -m --device ./td1 ./td2
The 'noscan' option is optional because there may be scenarios where we have a copy that we don't want to modify the fsid. In the following scenario, we keep 'td2' out of the metadata_uuid changes, even though its loop device is created.
$ cp td2 td3 $ btrsftune --noscan --device ./td3 -m ./td1
Thanks
--- tune/main.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
tune/main.c +9 -3
@@ -119,6 +119,7 @@ static const char * const tune_usage[] = {"General:" ,OPTLINE ("-f" ,"allow dangerous operations, make sure that you are aware of the dangers" ),OPTLINE ("--device" ,"devices or regular-files of the filesystem to be scanned" ), +OPTLINE ("--noscan" ,"do not scan the devices from the system, use only the listed ones" ),OPTLINE ("--help" ,"print this help" ), #if EXPERIMENTAL "",
@@ -145,6 +146,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])bool to_extent_tree = false;bool to_bg_tree = false;bool to_fst = false; +bool noscan = false;int csum_type =-1 ;int argc_devices =0 ;char **argv_devices =NULL ;
@@ -160,7 +162,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[])GETOPT_VAL_ENABLE_BLOCK_GROUP_TREE ,GETOPT_VAL_DISABLE_BLOCK_GROUP_TREE ,GETOPT_VAL_ENABLE_FREE_SPACE_TREE , - GETOPT_VAL_DEVICE }; + GETOPT_VAL_DEVICE, GETOPT_VAL_NOSCAN };static const struct option long_options[] = { {"help" , no_argument,NULL ,GETOPT_VAL_HELP }, {"convert-to-block-group-tree" , no_argument,NULL ,
@@ -173,6 +175,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) { "csum", required_argument, NULL, GETOPT_VAL_CSUM }, #endif { "device", required_argument, NULL, GETOPT_VAL_DEVICE }, + { "noscan", no_argument, NULL, GETOPT_VAL_NOSCAN }, { NULL, 0, NULL, 0 } }; int c = getopt_long(argc, argv, "S:rxfuU:nmM:", long_options, NULL);
@@ -245,6 +248,10 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) csum_type =parse_csum_type (optarg);break ;#endif +case GETOPT_VAL_NOSCAN : + ctree_flags |=OPEN_CTREE_NO_DEVICES ; + noscan = true; +break ;case GETOPT_VAL_HELP :default :usage (&tune_cmd, c !=GETOPT_VAL_HELP );
@@ -292,7 +299,7 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) } ret =check_mounted_where (fd, device,NULL ,0 ,NULL , -SBREAD_IGNORE_FSID_MISMATCH , false); +SBREAD_IGNORE_FSID_MISMATCH , noscan);if (ret <0 ) { errno = -ret;error ("could not check mount status of %s: %m" , device);
@@ -317,7 +324,6 @@ int BOX_MAIN(btrfstune)(int argc, char *argv[]) } root =open_ctree_fd (fd, device,0 , ctree_flags); -if (!root) {error ("open ctree failed" ); ret =1 ;
Signature · 1 line
2.31.1
PATCH 6/10 btrfs-progs: docs: update btrfstune --noscan option
In reply to Anand JainUpdate the Documentation/btrfstune.rst to carry the new --noscan option.
--- Documentation/btrfstune.rst | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/btrfstune.rst +4 -0
@@ -49,6 +49,10 @@ OPTIONS --device List of block devices or regular files that are part of the filesystem. +--noscan + Do not automatically scan the system for other devices of the same + filesystem, only use the devices provided as the arguments. + -m (since kernel: 5.0)
Signature · 1 line
2.31.1
PATCH 7/10 btrfs-progs: check: introduce --device option
In reply to Anand JainAs of now, btrfs check only accepts one device from the command line and then scans the system to find other parter devices if any.
However, this method mandates always accessing the file raw image as a loop device.
This patch modifies btrfs check to accept other devices or reg-files from the command line using an option --device and scans/registers them.
For example:
btrfs check --device /tdev/td1,/tdev/td2 /tdev/td3
or
btrfs check --device /tdev/td1 --device /tdev/td2 /tdev/td3--- check/main.c | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)
check/main.c +39 -2
@@ -51,6 +51,7 @@#include "common/messages.h" #include "common/task-utils.h" #include "common/device-utils.h" +#include "common/device-scan.h" #include "common/utils.h" #include "common/rbtree-utils.h" #include "common/help.h"
@@ -9953,6 +9954,7 @@ static const char * const cmd_check_usage[] = {OPTLINE ("-b|--backup" ,"use the first valid backup root copy" ),OPTLINE ("-r|--tree-root <bytenr>" ,"use the given bytenr for the tree root" ),OPTLINE ("--chunk-root <bytenr>" ,"use the given bytenr for the chunk tree root" ), +OPTLINE ("--device <device>" ,"use the given device or regular-file" ),"" ,"Operation modes:" , OPTLINE("--readonly", "run in read-only mode (default)"),
@@ -9989,6 +9991,8 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)u64 tree_root_bytenr =0 ;u64 chunk_root_bytenr =0 ;char uuidbuf[BTRFS_UUID_UNPARSED_SIZE ]; +char **argv_devices =NULL ; +int argc_devices =0 ;int ret =0 ;int err =0 ;u64 num;
@@ -10010,7 +10014,8 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)GETOPT_VAL_INIT_EXTENT ,GETOPT_VAL_CHECK_CSUM ,GETOPT_VAL_READONLY ,GETOPT_VAL_CHUNK_TREE ,GETOPT_VAL_MODE ,GETOPT_VAL_CLEAR_SPACE_CACHE , - GETOPT_VAL_CLEAR_INO_CACHE, GETOPT_VAL_FORCE }; +GETOPT_VAL_CLEAR_INO_CACHE ,GETOPT_VAL_FORCE , + GETOPT_VAL_DEVICE };static const struct option long_options[] = { {"super" , required_argument,NULL ,'s' }, { "repair", no_argument, NULL, GETOPT_VAL_REPAIR },
@@ -10035,6 +10040,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv) {"clear-ino-cache" , no_argument ,NULL , GETOPT_VAL_CLEAR_INO_CACHE}, { "force", no_argument, NULL, GETOPT_VAL_FORCE }, + { "device", required_argument, NULL, GETOPT_VAL_DEVICE }, { NULL, 0, NULL, 0} };
@@ -10126,6 +10132,22 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)case GETOPT_VAL_FORCE : force =1 ;break ; +case GETOPT_VAL_DEVICE : +if (!argv_devices) { + argv_devices =malloc (sizeof (char *)); +if (!argv_devices) { +error ("memory alloc failed" ); +exit (1 ); + } + } + +if (!array_append (argv_devices, optarg, + &argc_devices)) { +error ("memory alloc failed" ); +free (argv_devices); +exit (1 ); + } +break ; } }
@@ -10200,6 +10222,19 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv) ctree_flags &= ~OPEN_CTREE_EXCLUSIVE ; } +/* +* check_mounted_where() with noscan == true frees the scanned devices +* scan the command line provided device list now. +*/ +if (argv_devices) { + ret =btrfs_scan_argv_devices (0 , argc_devices, argv_devices); +if (ret) { + ret = -EIO ; + err |= !!ret; + goto free_out; + } + } +/* only allow partial opening under repair mode */ if (opt_check_repair) ctree_flags |=OPEN_CTREE_PARTIAL ;
@@ -10214,7 +10249,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)error ("cannot open file system" ); ret = -EIO ; err |= !!ret; - goto err_out; + goto free_out; } root = gfs_info->fs_root;
@@ -10560,6 +10595,8 @@ out:free_root_recs_tree (&root_cache); close_out:close_ctree (root); +free_out: +free_array (argv_devices, argc_devices); err_out:if (g_task_ctx.progress_enabled)task_deinit (g_task_ctx.info);
Signature · 1 line
2.31.1
PATCH 8/10 btrfs-progs: docs: update btrfs check --device option
In reply to Anand JainUpdate the Documentation/btrfs-check.rst to carry the new --device option.
--- Documentation/btrfs-check.rst | 2 ++ 1 file changed, 2 insertions(+)
Documentation/btrfs-check.rst +2 -0
@@ -86,6 +86,8 @@ SAFE OR ADVISORY OPTIONS --clear-ino-cache remove leftover items pertaining to the deprecated inode map feature +--device + List of block devices or regular files that are part of the filesystem. DANGEROUS OPTIONS -----------------
Signature · 1 line
2.31.1
PATCH 9/10 btrfs-progs: check: introduce --noscan option
In reply to Anand JainThe function check_where_mounted() scans the system for other btrfs devices, but in certain cases, we may need a way to instruct btrfs check not to perform the system scan and instead only work on the devices provided through the command line. And so, add an option --noscan.
--- check/main.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
check/main.c +5 -1
@@ -10015,7 +10015,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)GETOPT_VAL_READONLY ,GETOPT_VAL_CHUNK_TREE ,GETOPT_VAL_MODE ,GETOPT_VAL_CLEAR_SPACE_CACHE ,GETOPT_VAL_CLEAR_INO_CACHE ,GETOPT_VAL_FORCE , - GETOPT_VAL_DEVICE }; + GETOPT_VAL_DEVICE, GETOPT_VAL_NOSCAN };static const struct option long_options[] = { {"super" , required_argument,NULL ,'s' }, { "repair", no_argument, NULL, GETOPT_VAL_REPAIR },
@@ -10041,6 +10041,7 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv) GETOPT_VAL_CLEAR_INO_CACHE}, { "force", no_argument, NULL, GETOPT_VAL_FORCE }, { "device", required_argument, NULL, GETOPT_VAL_DEVICE }, + { "noscan", no_argument, NULL, GETOPT_VAL_NOSCAN }, { NULL, 0, NULL, 0} };
@@ -10148,6 +10149,9 @@ static int cmd_check(const struct cmd_struct *cmd, int argc, char **argv)exit (1 ); }break ; +case GETOPT_VAL_NOSCAN : + ctree_flags |=OPEN_CTREE_NO_DEVICES ; +break ; } }
Signature · 1 line
2.31.1
PATCH 10/10 btrfs-progs: docs: update btrfs check --noscan option
In reply to Anand JainUpdate the Documentation/btrsf-check.rst to carry the new --noscan option.
--- Documentation/btrfs-check.rst | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/btrfs-check.rst +4 -0
@@ -89,6 +89,10 @@ SAFE OR ADVISORY OPTIONS --device List of block devices or regular files that are part of the filesystem. +--noscan + Do not automatically scan the system for other devices for the same + filesystem, only use the devices provided as the arguments. + DANGEROUS OPTIONS -----------------
Signature · 1 line
2.31.1
Quoted 18 lines · Anand Jain
By default, btrfstune and btrfs check scans all and only the block devices in the system.
To scan regular files without mapping them to a loop device, adds the --device option.
To indicate not to scan the system for other devices, adds the --noscan option.
For example:
The command below will scan both regular files and the devices provided in the --device option, along with the system block devices.
btrfstune -m --device /tdev/td1,/tdev/td2 /tdev/td3 or btrfs check --device /tdev/td1 --device /tdev/td2 /tdev/td3
Quoted 6 lines
In some cases, if you need to avoid the default system scan for the block device, you can use the --noscan option.
btrfstune -m --noscan --device /tdev/td1,/tdev/td2 /tdev/td3 btrfs check --noscan --device /tdev/td1,/tdev/td2 /tdev/td3
From the examples above I don't understand which devices get scanned or not, there are the --device ones and then the agtument. Also for examples please use something recognizable like /dev/sdx, /dev/sdy, otherwise it looks like it requires some special type of device.
I'd expect that --noscan will not scan any device that is part of the filesystem that is pointed to by the agrument (/tdev/td3 in this case).
If the option --noscan + --device are meant to work together that only the given devices are either scanned or not (this is the part I don't see clearly), then I'd suggest to let --noscan take a value of device that won't be scanned.
Eventually there could be a special value --noscan=all to not scan all devices. Also please drop the syntax where the devices are separated by ",", one option per device should work for everybody and you can avoid parsting the option value.
Re: [PATCH 01/10] btrfs-progs: common: add --device option helpers
In reply to Anand JainQuoted 2 lines · Anand Jain
Preparatory patch adds two helper functions:
array_append()andfree_array(), which facilitate reading the device list provided at the --device option.
That it's for --device is for later and not that interesting when adding some API.
Quoted 43 lines
Signed-off-by Anand Jain <[email protected]>--- common/device-scan.c | 32 ++++++++++++++++++++++++++++++++ common/device-scan.h | 2 ++ 2 files changed, 34 insertions(+)common/device-scan.c +22
-0@@ -31,6 +31,7 @@#include <dirent.h> #include <limits.h> #include <stdbool.h> +#include <ctype.h> #include <blkid/blkid.h> #include <uuid/uuid.h> #ifdef HAVE_LIBUDEV @@ -540,3 +541,34 @@ int btrfs_scan_argv_devices(int dev_optind, int dev_argc, char **dev_argv)return 0 ; } + +bool array_append (char **dest,char *src,int *cnt) +{ +char *this_tok =strtok (src,"," ); +int ret_cnt = *cnt; + +while (this_tok !=NULL ) { + ret_cnt++; + dest =realloc (dest,sizeof (char *) * ret_cnt); +if (!dest) +return false; + + dest[ret_cnt -1 ] =strdup (this_tok); + *cnt = ret_cnt; + + this_tok =strtok (NULL ,"," ); + } + +return true; +} +
common/device-scan.c int btrfs_scan_argv_devices(int dev_optind, int dev_argc, char **dev_argv) · line 574
Quoted 10 lines
common/device-scan.c +10
-0+void free_array (char **prt,int cnt) +{ +if (!prt) +return ; + +for (int i =0 ; i < cnt; i++) +free (prt[i]); + +free (prt); +}
Looks like this is an extensible pointer array, we could use that in more places where there are repeated parameters and we need to track all the values (not just the last one).
Then this should be in a structure and the usage side will do only something like ptr_array_append(&array, newvalue), and not that all places will have to track the base double pointer, count and has to handle allocation failures. This should be wrapped into an API.
Re: [PATCH 04/10] btrfs-progs: docs: update btrfstune --device option
In reply to Anand JainQuoted 13 lines · Anand Jain
Update the
Documentation/btrfstune.rstto carry the new --device option.Signed-off-by Anand Jain <[email protected]>--- Documentation/btrfstune.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/btrfstune.rst +0
-0
Documentation/btrfstune.rst OPTIONS · line 49
Quoted 4 lines
Documentation/btrfstune.rst +1
-0@@ -46,6 +46,9 @@ OPTIONS Allow dangerous changes, e.g. clear the seeding flag or change fsid. Make sure that you are aware of the dangers. +--device
Options that take a value should also document it.
Quoted 7 lines
Documentation/btrfstune.rst +2
-0+ List of block devices or regular files that are part of the filesystem. + -m (since kernel: 5.0)Signature · 1 line
2.31.1
Re: [PATCH 01/10] btrfs-progs: common: add --device option helpers
In reply to David SterbaQuoted 36 lines · David Sterba
Quoted 20 lines · Anand Jain
common/device-scan.c +20
-0+bool array_append (char **dest,char *src,int *cnt) +{ +char *this_tok =strtok (src,"," ); +int ret_cnt = *cnt; + +while (this_tok !=NULL ) { + ret_cnt++; + dest =realloc (dest,sizeof (char *) * ret_cnt); +if (!dest) +return false; + + dest[ret_cnt -1 ] =strdup (this_tok); + *cnt = ret_cnt; + + this_tok =strtok (NULL ,"," ); + } + +return true; +} +common/device-scan.c int btrfs_scan_argv_devices(int dev_optind, int dev_argc, char **dev_argv) · line 574
Quoted 10 lines
common/device-scan.c +10
-0+void free_array (char **prt,int cnt) +{ +if (!prt) +return ; + +for (int i =0 ; i < cnt; i++) +free (prt[i]); + +free (prt); +}Looks like this is an extensible pointer array, we could use that in more places where there are repeated parameters and we need to track all the values (not just the last one).
Quoted 4 lines
Then this should be in a structure and the usage side will do only something like
ptr_array_append(&array, newvalue), and not that all places will have to track the base double pointer, count and has to handle allocation failures. This should be wrapped into an API.
I did a quick search for reallocs, that usually mean there's an extensible array and there are too many, so I get you wanted add one more and be done. But now it looks like a proper structure should be used. I'll try to implement something.
Re: [PATCH 01/10] btrfs-progs: common: add --device option helpers
Quoted 42 lines · David Sterba
Quoted 36 lines · David Sterba
Quoted 20 lines · Anand Jain
common/device-scan.c +20
-0+bool array_append (char **dest,char *src,int *cnt) +{ +char *this_tok =strtok (src,"," ); +int ret_cnt = *cnt; + +while (this_tok !=NULL ) { + ret_cnt++; + dest =realloc (dest,sizeof (char *) * ret_cnt); +if (!dest) +return false; + + dest[ret_cnt -1 ] =strdup (this_tok); + *cnt = ret_cnt; + + this_tok =strtok (NULL ,"," ); + } + +return true; +} +common/device-scan.c int btrfs_scan_argv_devices(int dev_optind, int dev_argc, char **dev_argv) · line 574
Quoted 10 lines
common/device-scan.c +10
-0+void free_array (char **prt,int cnt) +{ +if (!prt) +return ; + +for (int i =0 ; i < cnt; i++) +free (prt[i]); + +free (prt); +}Looks like this is an extensible pointer array, we could use that in more places where there are repeated parameters and we need to track all the values (not just the last one).
Quoted 4 lines
Then this should be in a structure and the usage side will do only something like
ptr_array_append(&array, newvalue), and not that all places will have to track the base double pointer, count and has to handle allocation failures. This should be wrapped into an API.
Quoted 4 lines
I did a quick search for reallocs, that usually mean there's an extensible array and there are too many, so I get you wanted add one more and be done. But now it looks like a proper structure should be used. I'll try to implement something.
Yes, indeed, this has to be an API. I thought it could be done separately.
Thanks.
Re: [PATCH 04/10] btrfs-progs: docs: update btrfstune --device option
In reply to David SterbaQuoted 17 lines · David Sterba
Quoted 16 lines · Anand Jain
Update the
Documentation/btrfstune.rstto carry the new --device option.Signed-off-by Anand Jain <[email protected]>--- Documentation/btrfstune.rst | 3 +++ 1 file changed, 3 insertions(+)Documentation/btrfstune.rst +0
-0@@ -46,6 +46,9 @@ OPTIONS Allow dangerous changes, e.g. clear the seeding flag or change fsid. Make sure that you are aware of the dangers.
Quoted 3 lines
Documentation/btrfstune.rst OPTIONS · line 49
Quoted 1 line
Documentation/btrfstune.rst +1
-0+--deviceOptions that take a value should also document it.
Oh. yes. Now added for both tune and check.
Thanks.
Quoted 8 lines
Quoted 8 lines
Documentation/btrfstune.rst +2
-0+ List of block devices or regular files that are part of the filesystem. + -m (since kernel: 5.0)Signature · 1 line
2.31.1
Quoted 22 lines · David Sterba
Quoted 21 lines · Anand Jain
By default, btrfstune and btrfs check scans all and only the block devices in the system.
To scan regular files without mapping them to a loop device, adds the --device option.
To indicate not to scan the system for other devices, adds the --noscan option.
For example:
The command below will scan both regular files and the devices provided in the --device option, along with the system block devices.
btrfstune -m --device /tdev/td1,/tdev/td2 /tdev/td3 or btrfs check --device /tdev/td1 --device /tdev/td2 /tdev/td3In some cases, if you need to avoid the default system scan for the block device, you can use the --noscan option.
Quoted 8 lines
Quoted 3 lines
btrfstune -m --noscan --device /tdev/td1,/tdev/td2 /tdev/td3 btrfs check --noscan --device /tdev/td1,/tdev/td2 /tdev/td3From the examples above I don't understand which devices get scanned or
not, there are the --device ones and then the agtument. Also for examples please use something recognizable like/dev/sdx,/dev/sdy, otherwise it looks like it requires some special type of device.
Let me try again..
The --noscan option in btrfstune is designed to disable automatic system-wide scanning + detection and assembly of Btrfs devices. Instead, it utilizes the devices provided in the command line argument,
at the option --device and the last argument.
I used devices like "/tdev/td1," and so on to indicate that it's not just the block device as an argument but also the regular file images. Seems like I didn't explain it clearly. I'll try update this.
Quoted 2 lines
I'd expect that --noscan will not scan any device that is part of the filesystem that is pointed to by the agrument (/tdev/td3 in this case).
Indeed true, except for the devices specified in the --device option and the last argument of the 'btrfstune' command. I propose this approach due to the split brain (CHANGING_FSID) scenario, where devices to be assembled may have different fsids but matching metadata_uuid, making automatic assembly unsafe.
In my upcoming kernel patch, I have a test case that demonstrates bug in the kernel's automatic device assembly during the split brain scenarios and as below:
------------
Consider two filesystems with two unique fsids.
267baf3a-d949-4e8d-bb09-79edab766d39 (/dev/loop0 /dev/loop1) b698d8f4-cc56-4480-b202-c29882cfa13e (/dev/loop2 /dev/loop3)
The metadata_uuid (271ca08e-ed4c-4c80-8919-fe59533a635b) is the same for both fsids.
Suppose 267baf3a-d949-4e8d-bb09-79edab766d39 attempts to change its fsid using 'btrfstune -m' option, but encounters a write failure on devid 1.
$ /incomplete-fsid-change/btrfstune -m /dev/loop0 dbg: Wrote devid 2 skip_write_sb_cnt 0 dbg: Wrote devid 1 skip_write_sb_cnt 1 dbg: Wrote devid 2 skip_write_sb_cnt 1 dbg: SKIP devid 1 skip_write_sb_cnt 2 <-- Last sb write failed
Check if one of the device has CHANGING_FSID set. Also, note that
All devices /dev/loop[0-3] have same metadata_uuid. /dev/loop[0-1] have different fsids. /dev/loop[2-3] have greater generation number.
$ cat sb btrfs inspect dump-super $1 | \ egrep "device|devid|^metadata_uuid|^fsid|^incom|^generation|^flags"
$ sb /dev/loo0 superblock: bytenr=65536, device=/dev/loop0 flags 0x1000000001 <-- CHANGEING_FSID_V2 fsid 267baf3a-d949-4e8d-bb09-79edab766d39 <-- metadata_uuid 271ca08e-ed4c-4c80-8919-fe59533a635b generation 13 num_devices 2 incompat_flags 0x761 dev_item.devid 1
$ sb /dev/loop1 superblock: bytenr=65536, device=/dev/loop1 flags 0x1 fsid 54ba1e89-b754-4a18-b464-0cc5ec47187a <-- metadata_uuid 271ca08e-ed4c-4c80-8919-fe59533a635b generation 14 <-- num_devices 2 incompat_flags 0x761 dev_item.devid 2
$ sb /dev/loop2 superblock: bytenr=65536, device=/dev/loop2 flags 0x1 fsid b698d8f4-cc56-4480-b202-c29882cfa13e metadata_uuid 271ca08e-ed4c-4c80-8919-fe59533a635b generation 15 <-- num_devices 2 incompat_flags 0x761 dev_item.devid 1
$ sb /dev/loop3 superblock: bytenr=65536, device=/dev/loop3 flags 0x1 fsid b698d8f4-cc56-4480-b202-c29882cfa13e metadata_uuid 271ca08e-ed4c-4c80-8919-fe59533a635b generation 15 num_devices 2 incompat_flags 0x761 dev_item.devid 2
Despite /dev/loop0 and /dev/loop3 not belonging to the same fsid, in the context of recovering from the CHANGING_FSID scenario, kernel can assemble them together.
$ btrfs dev scan --forget $ btrfs dev scan /dev/loop3 Scanning for btrfs filesystems on '/dev/loop3'
$ mount /dev/loop0 /btrfs $ btrfs fi show -m /btrfs Label: none uuid: b698d8f4-cc56-4480-b202-c29882cfa13e Total devices 2 FS bytes used 144.00KiB devid 1 size 300.00MiB used 48.00MiB path /dev/loop0 <-- devid 2 size 300.00MiB used 40.00MiB path /dev/loop3 <--
[ 984.959694] BTRFS error (device loop3): parent transid verify failed on 30507008 wanted 15 found 14
In my view, automatic device assembly is unsafe. Devices with the CHANGING_FSID flag should be treated similar to a filesystem needing fsck and fix or assemble them back in the userland.
----------------
Quoted 3 lines
If the option --noscan + --device are meant to work together that only the given devices are either scanned or not (this is the part I don't see clearly),
Only the specified devices are scanned and used for assembling the volume if their metadata_uuids match and devids line up.
Quoted 2 lines
then I'd suggest to let --noscan take a value of device that won't be scanned.
Oh. Do you mean something like, for example:
(not related to any of the test case):
btrfstune -m --noscan=/dev/sda --noscan=/dev/sdb /dev/sdc
I'm okay, but I think we need to rename 'noscan' to something else?
Quoted 3 lines
Eventually there could be a special value --noscan=all to not scan all devices.
--noscan and --device options are also present in the "btrfs inspect dump-tree" sub-command.
I hope its different usage won't confuse the users.
Quoted 3 lines
Also please drop the syntax where the devices are separated by ",", one option per device should work for everybody and you can avoid parsting the option value.
I kept it the same as in the mount -o device option; I'm ok dropping support for ','.
Thanks, Anand