btrfs-progs: enhance --subvol/--inode-flags options
Subsystems (2)
cover btrfs-progs: enhance --subvol/--inode-flags options
Currently both --subvol and --inode-flags save the full path into their structures, and check each inode against those full path.
For long paths it can be time consuming, and this introduces extra memory for each structure.
This series enhance the handling of those options by:
- Extract the validation part into a dedicated helper inside rootdir.[ch]
- Use st_dev/st_ino to replace full_path
This reduces runtime and memory usage for the involved structures.
- Remove the memory usage warning note
Even with the old 8K per structure memory usage, 1024 options will only 8M memory, that's accetable even for a lot of micro-controllers, not to mention modern desktop/servers.
I'm a little paranoid at that time, with the memory usage almost halved, we can safely remove that warning note.
Qu Wenruo (5):
btrfs-progs: mkfs/rootdir: extract subvol validation code into a
helper
btrfs-progs: mkfs/rootdir: extract inode flags validation code into a
helper
btrfs-progs: mkfs/rootdir: enhance subvols detection
btrfs-progs: mkfs/rootdir: enhance inode flags detection
btrfs-progs: doc/mkfs: remove the note about memory usage
Documentation/mkfs.btrfs.rst | 5 -- mkfs/main.c | 84 ++------------------ mkfs/rootdir.c | 147 +++++++++++++++++++++++++++++++---- mkfs/rootdir.h | 14 +++- 4 files changed, 148 insertions(+), 102 deletions(-)
--
2.50.1
PATCH 1/5 btrfs-progs: mkfs/rootdir: extract subvol validation code into a helper
Currently we do the validation of subvolume parameters inside mkfs/main.c, but considering all things like structure rootdir_subvol are all inside rootdir.[ch], it's better to move the validation part into rootdir.[ch].
Extract the validation part into a helper,btrfs_mkfs_validate_subvols(), into rootdir.[ch].
Furthermore since we're here, also slightly enhance the duplicated subvolume check, so that the runtime is halved.
--- mkfs/main.c | 47 +++-------------------------------------------- mkfs/rootdir.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ mkfs/rootdir.h | 2 ++ 3 files changed, 51 insertions(+), 44 deletions(-)
mkfs/main.c +3 -44
@@ -1527,50 +1527,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv) } } -list_for_each_entry (rds, &subvols, list) { -char path[PATH_MAX ]; -struct rootdir_subvol *rds2; - -if (path_cat_out (path, source_dir, rds->dir)) { -error ("path invalid: %s" , path); - ret =1 ; - goto error; - } - -if (!realpath (path, rds->full_path)) { -error ("could not get canonical path: %s" , rds->dir); - ret =1 ; - goto error; - } - -if (!path_exists (rds->full_path)) { -error ("subvolume path does not exist: %s" , rds->dir); - ret =1 ; - goto error; - } - -if (!path_is_dir (rds->full_path)) { -error ("subvolume is not a directory: %s" , rds->dir); - ret =1 ; - goto error; - } - -if (!path_is_in_dir (source_dir, rds->full_path)) { -error ("subvolume %s is not a child of %s" , rds->dir, source_dir); - ret =1 ; - goto error; - } - - for (rds2 = list_first_entry(&subvols, struct rootdir_subvol, list); - rds2 != rds; - rds2 =list_next_entry (rds2, list)) { -if (strcmp (rds2->full_path, rds->full_path) ==0 ) { -error ("subvolume specified more than once: %s" , rds->dir); - ret =1 ; - goto error; - } - } - } + ret =btrfs_mkfs_validate_subvols (source_dir, &subvols); +if (ret <0 ) + goto error;list_for_each_entry (rif, &inode_flags_list, list) {char path[PATH_MAX ];
mkfs/rootdir.c +46 -0
52 lines not on this page · raw message
mkfs/rootdir.h +2 -0
@@ -39,6 +39,7 @@ struct btrfs_root;struct rootdir_subvol {struct list_head list; +/* The path inside the source_dir. */ char dir[PATH_MAX ];char full_path[PATH_MAX ];bool is_default;
@@ -59,6 +60,7 @@ struct rootdir_inode_flags_entry {bool nodatasum; }; +int btrfs_mkfs_validate_subvols (const char *source_dir,struct list_head *subvols); int btrfs_mkfs_fill_dir(struct btrfs_trans_handle *trans, const char *source_dir, struct btrfs_root *root, struct list_head *subvols, struct list_head *inode_flags_list,
Signature · 1 line
2.50.1
PATCH 2/5 btrfs-progs: mkfs/rootdir: extract inode flags validation code into a helper
In reply to Qu WenruoCurrently we do the validation of inode flag parameters inside mkfs/main.c, but considering all things like structure rootdir_inode_flags_entry are all inside rootdir.[ch], it's better to move the validation part into rootdir.[ch].
Extract the validation part into a helper, btrfs_mkfs_validate_inode_flags(), into rootdir.[ch].
--- mkfs/main.c | 37 +++---------------------------------- mkfs/rootdir.c | 39 +++++++++++++++++++++++++++++++++++++++ mkfs/rootdir.h | 1 + 3 files changed, 43 insertions(+), 34 deletions(-)
mkfs/main.c +3 -34
@@ -1531,40 +1531,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)if (ret <0 ) goto error; -list_for_each_entry (rif, &inode_flags_list, list) { -char path[PATH_MAX ]; -struct rootdir_inode_flags_entry *rif2; - -if (path_cat_out (path, source_dir, rif->inode_path)) { - ret = -EINVAL ; -error ("path invalid: %s" , path); - goto error; - } -if (!realpath (path, rif->full_path)) { - ret = -errno; -error ("could not get canonical path: %s: %m" , path); - goto error; - } -if (!path_exists (rif->full_path)) { - ret = -ENOENT ; -error ("inode path does not exist: %s" , rif->full_path); - goto error; - } -list_for_each_entry (rif2, &inode_flags_list, list) { -/* -* Only compare entries before us. So we won't compare -* the same pair twice. -*/ -if (rif2 == rif) -break ; -if (strcmp (rif2->full_path, rif->full_path) ==0 ) { -error ("duplicated inode flag entries for %s" , - rif->full_path); - ret = -EEXIST ; - goto error; - } - } - } + ret =btrfs_mkfs_validate_inode_flags (source_dir, &inode_flags_list); +if (ret <0 ) + goto error;if (*fs_uuid) {uuid_t dummy_uuid;
mkfs/rootdir.c +39 -0
@@ -1123,6 +1123,45 @@ int btrfs_mkfs_validate_subvols(const char *source_dir, struct list_head *subvolreturn 0 ; } +int btrfs_mkfs_validate_inode_flags (const char *source_dir,struct list_head *inode_flags) +{ +struct rootdir_inode_flags_entry *rif; + +list_for_each_entry (rif, inode_flags, list) { +char path[PATH_MAX ]; +struct rootdir_inode_flags_entry *rif2; +int ret; + +if (path_cat_out (path, source_dir, rif->inode_path)) { +error ("path invalid: %s" , path); +return -EINTR ; + } +if (!realpath (path, rif->full_path)) { + ret = -errno; +error ("could not get canonical path: %s: %m" , path); +return ret; + } +if (!path_exists (rif->full_path)) { +error ("inode path does not exist: %s" , rif->full_path); +return -ENOENT ; + } +list_for_each_entry (rif2, inode_flags, list) { +/* +* Only compare entries before us. So we won't compare +* the same pair twice. +*/ +if (rif2 == rif) +break ; +if (strcmp (rif2->full_path, rif->full_path) ==0 ) { +error ("duplicated inode flag entries for %s" , + rif->full_path); +return -EEXIST ; + } + } + } +return 0 ; +} + static int add_file_items(struct btrfs_trans_handle *trans, struct btrfs_root *root, struct btrfs_inode_item *btrfs_inode, u64 objectid,
mkfs/rootdir.h +1 -0
@@ -61,6 +61,7 @@ struct rootdir_inode_flags_entry { };int btrfs_mkfs_validate_subvols (const char *source_dir,struct list_head *subvols); +int btrfs_mkfs_validate_inode_flags (const char *source_dir,struct list_head *inode_flags); int btrfs_mkfs_fill_dir(struct btrfs_trans_handle *trans, const char *source_dir, struct btrfs_root *root, struct list_head *subvols, struct list_head *inode_flags_list,
Signature · 1 line
2.50.1
PATCH 3/5 btrfs-progs: mkfs/rootdir: enhance subvols detection
In reply to Qu WenruoCurrently for --subvol parameter, we save the full path of the host fs into rootdir_subvol sturcture, then compare each inode we hit with that saved full path.
This string comparison can be time consuming (up to PATH_MAX characters), and we're doing it for every directory inode.
On the other hand, nftw() also provides a stat structure of the current inode, stat::st_dev and stat::st_ino pair can always uniquely locate an inode in the host filesystem.
With that said, we can just save the st_dev/st_ino pair when validating the subvol parameters, and use st_dev/st_ino to check if we hit the target subvolume.
This has two benefits:
- Reduce the memory usage of each rootdir_subvol Now we need no full_path member, replacing it with two u64. This saves (4K - 16) bytes per rootdir_subvol.
- Reduce the runtime of direct inode comparison
Instead of doing strcmp() for up to 4K bytes, it's just two u64 comparison.
--- mkfs/rootdir.c | 52 +++++++++++++++++++++++++++++++++++--------------- mkfs/rootdir.h | 4 +++- 2 files changed, 40 insertions(+), 16 deletions(-)
mkfs/rootdir.c +37 -15
@@ -1083,6 +1083,8 @@ int btrfs_mkfs_validate_subvols(const char *source_dir, struct list_head *subvollist_for_each_entry (rds, subvols, list) {char path[PATH_MAX ]; +char full_path[PATH_MAX ]; +struct stat stbuf;struct rootdir_subvol *rds2;int ret;
@@ -1092,21 +1094,29 @@ int btrfs_mkfs_validate_subvols(const char *source_dir, struct list_head *subvolerror ("path invalid '%s': %m" , path);return ret; } -if (!realpath (path, rds->full_path)) { +if (!realpath (path, full_path)) { ret = -errno;error ("could not get canonical path of '%s': %m" , rds->dir);return ret; } - ret =path_exists (rds->full_path); + ret =path_exists (full_path);if (ret <0 ) {error ("subvolume path does not exist: %s" , rds->dir);return ret; } - ret =path_is_dir (rds->full_path); + ret =path_is_dir (full_path);if (ret <0 ) {error ("subvolume is not a directory: %s" , rds->dir);return ret; } + ret =lstat (full_path, &stbuf); +if (ret <0 ) { + ret = -errno; +error ("failed to get stat of '%s': %m" , full_path); +return ret; + } + rds->st_dev = stbuf.st_dev; + rds->st_ino = stbuf.st_ino;list_for_each_entry (rds2, subvols, list) {/* * Only compare entries before us, So we won't compare
@@ -1114,7 +1124,7 @@ int btrfs_mkfs_validate_subvols(const char *source_dir, struct list_head *subvol*/ if (rds2 == rds)break ; -if (strcmp (rds2->full_path, rds->full_path) ==0 ) { +if (rds2->st_dev == rds->st_dev && rds2->st_ino == rds->st_ino) {error ("subvolume specified more than once: %s" , rds->dir);return -EINVAL ; }
@@ -1424,15 +1434,26 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st,struct btrfs_root *new_root;struct inode_entry *parent;struct btrfs_inode_item inode_item = {0 }; +char *path_dump; +char *base_path;u64 subvol_id, ino; subvol_id = next_subvol_id++; + path_dump =strdup (full_path); +if (!path_dump) +return -ENOMEM ; + base_path =path_basename (path_dump); +if (!base_path) { + ret = -errno; +error ("failed to get basename of '%s': %m" , path_dump); + goto out; + } ret =btrfs_make_subvolume (g_trans, subvol_id, subvol->readonly);if (ret <0 ) { errno = -ret;error ("failed to create subvolume: %m" ); -return ret; + goto out; }if (subvol->is_default)
@@ -1447,19 +1468,18 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st, ret =PTR_ERR (new_root); errno = -ret;error ("unable to read fs root id %llu: %m" , subvol_id); -return ret; + goto out; } parent =rootdir_path_last (¤t_path); ret =btrfs_link_subvolume (g_trans, parent->root, parent->ino, -path_basename (subvol->full_path), -strlen (path_basename (subvol->full_path)), + base_path,strlen (base_path), new_root);if (ret) { errno = -ret; -error ("unable to link subvolume %s: %m" ,path_basename (subvol->full_path)); -return ret; +error ("unable to link subvolume %s: %m" , base_path); + goto out; } ino =btrfs_root_dirid (&new_root->root_item);
@@ -1469,7 +1489,7 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st, errno = -ret;error ("failed to add xattr item for the top level inode in subvol %llu: %m" , subvol_id); -return ret; + goto out; }stat_to_inode_item (&inode_item, st);
@@ -1479,7 +1499,7 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st,if (ret <0 ) { errno = -ret;error ("failed to update root dir for root %llu: %m" , subvol_id); -return ret; + goto out; } ret =rootdir_path_push (¤t_path, new_root, ino);
@@ -1487,10 +1507,12 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st, errno = -ret;error ("failed to allocate new entry for subvolume %llu ('%s'): %m" , subvol_id, full_path); -return ret; + goto out; } -return 0 ; +out: +free (path_dump); +return ret; } static int read_inode_item(struct btrfs_root *root, struct btrfs_inode_item *inode_item,
@@ -1611,7 +1633,7 @@ static int ftw_add_inode(const char *full_path, const struct stat *st,if (S_ISDIR (st->st_mode)) {list_for_each_entry (rds, g_subvols, list) { -if (!strcmp (full_path, rds->full_path)) { +if (st->st_dev == rds->st_dev && st->st_ino == rds->st_ino) { ret =ftw_add_subvol (full_path, st, typeflag, ftwbuf, rds);
mkfs/rootdir.h +3 -1
@@ -41,7 +41,9 @@ struct rootdir_subvol {struct list_head list;/* The path inside the source_dir. */ char dir[PATH_MAX ]; -char full_path[PATH_MAX ]; +/* st_dev and st_ino is going to uniquely determine an inode inside the host fs. */ +dev_t st_dev; +ino_t st_ino;bool is_default;bool readonly; };
Signature · 1 line
2.50.1
PATCH 4/5 btrfs-progs: mkfs/rootdir: enhance inode flags detection
In reply to Qu WenruoCurrently for --inode-flags parameter, we save the full path of the host fs into rootdir_inode_flags_entry sturcture, then compare each inode we hit with that saved full path.
This string comparison can be time consuming (up to PATH_MAX characters), and we're doing it for every inode.
On the other hand, nftw() also provides a stat structure of the current inode, stat::st_dev and stat::st_ino pair can always uniquely locate an inode in the host filesystem.
With that said, we can just save the st_dev/st_ino pair when validating the inode flag parameters, and use st_dev/st_ino to check if we hit the target inode.
This has two benefits:
- Reduce the memory usage of each rootdir_inode_flags_entry Now we need no full_path member, replacing it with two u64. This saves (4K - 16) bytes per rootdir_inode_flags_entry.
- Reduce the runtime of inode comparison
Instead of doing strcmp() for up to 4K bytes, it's just two u64 comparison.
--- mkfs/rootdir.c | 28 +++++++++++++++++++--------- mkfs/rootdir.h | 7 ++++--- 2 files changed, 23 insertions(+), 12 deletions(-)
mkfs/rootdir.c +19 -9
@@ -1139,22 +1139,32 @@ int btrfs_mkfs_validate_inode_flags(const char *source_dir, struct list_head *inlist_for_each_entry (rif, inode_flags, list) {char path[PATH_MAX ]; +char full_path[PATH_MAX ];struct rootdir_inode_flags_entry *rif2; +struct stat stbuf;int ret;if (path_cat_out (path, source_dir, rif->inode_path)) {error ("path invalid: %s" , path);return -EINTR ; } -if (!realpath (path, rif->full_path)) { +if (!realpath (path, full_path)) { ret = -errno;error ("could not get canonical path: %s: %m" , path);return ret; } -if (!path_exists (rif->full_path)) { -error ("inode path does not exist: %s" , rif->full_path); +if (!path_exists (full_path)) { +error ("inode path does not exist: %s" , full_path);return -ENOENT ; } + ret =lstat (full_path, &stbuf); +if (ret <0 ) { + ret = -errno; +error ("failed to get stat of '%s': %m" , full_path); +return ret; + } + rif->st_dev = stbuf.st_dev; + rif->st_ino = stbuf.st_ino;list_for_each_entry (rif2, inode_flags, list) {/* * Only compare entries before us. So we won't compare
@@ -1162,9 +1172,9 @@ int btrfs_mkfs_validate_inode_flags(const char *source_dir, struct list_head *in*/ if (rif2 == rif)break ; -if (strcmp (rif2->full_path, rif->full_path) ==0 ) { +if (rif2->st_dev == rif->st_dev && rif2->st_ino == rif->st_ino) {error ("duplicated inode flag entries for %s" , - rif->full_path); + full_path);return -EEXIST ; } }
@@ -1410,12 +1420,12 @@ static void update_inode_flags(const struct rootdir_inode_flags_entry *rif, }static void search_and_update_inode_flags (struct btrfs_inode_item *stack_inode, -const char *full_path) +const struct stat *st) {struct rootdir_inode_flags_entry *rif;list_for_each_entry (rif, g_inode_flags_list, list) { -if (strcmp (rif->full_path, full_path) ==0 ) { +if (rif->st_dev == st->st_dev && rif->st_ino == st->st_ino) {update_inode_flags (rif, stack_inode);list_del (&rif->list);
@@ -1493,7 +1503,7 @@ static int ftw_add_subvol(const char *full_path, const struct stat *st, }stat_to_inode_item (&inode_item, st); -search_and_update_inode_flags (&inode_item, full_path); +search_and_update_inode_flags (&inode_item, st);btrfs_set_stack_inode_nlink (&inode_item,1 ); ret =update_inode_item (g_trans, new_root, &inode_item, ino);if (ret <0 ) {
@@ -1687,7 +1697,7 @@ static int ftw_add_inode(const char *full_path, const struct stat *st,return ret; }stat_to_inode_item (&inode_item, st); -search_and_update_inode_flags (&inode_item, full_path); +search_and_update_inode_flags (&inode_item, st); ret =btrfs_insert_inode (g_trans, root, ino, &inode_item);if (ret <0 ) {
mkfs/rootdir.h +4 -3
@@ -49,14 +49,15 @@ struct rootdir_subvol { };/* -* Represent a flag for specified inode at @full_path. +* Represent a flag for specified inode at "$source_dir/$inode_path". */ struct rootdir_inode_flags_entry {struct list_head list; -/* Fully canonicalized path to the source file. */ -char full_path[PATH_MAX ];/* Path inside the source directory. */ char inode_path[PATH_MAX ]; +/* st_dev and st_ino is going to uniquely determine an inode inside the host fs. */ +dev_t st_dev; +ino_t st_ino;bool nodatacow;bool nodatasum;
Signature · 1 line
2.50.1
PATCH 5/5 btrfs-progs: doc/mkfs: remove the note about memory usage
In reply to Qu WenruoWith the recent enhancements to --subvol and --inode-flags options, the memory usage for them are just 4K+, and even for the older 8K+ memory usage, it's really hard to consider them as heavy memory usage.
E.g. even if the end user specified over 1000 entries of --subvol/--inode-flags, it only takes a little over 8MiB for the older code or over 4MiB for the newer code.
Since we're in the user space and the year is 2025 not 1995, such memory usage is far from heavy.
Just remove the paranoid note from the man page.
--- Documentation/mkfs.btrfs.rst | 5 ----- 1 file changed, 5 deletions(-)
Documentation/mkfs.btrfs.rst +0 -5
@@ -243,11 +243,6 @@ OPTIONS each subvolume is independent and will not inherit from the parent directory. (The same as the kernel behavior.) - .. note:: - Both *--inode-flags* and *--subvol* options are memory hungry, - will consume at least 8KiB for each option. Please keep the - usage of both options to minimum. - --shrink Shrink the filesystem to its minimal size, only works with *--rootdir* option.
Signature · 1 line
2.50.1
Quoted 22 lines · Qu Wenruo
Currently both --subvol and --inode-flags save the full path into their structures, and check each inode against those full path.
For long paths it can be time consuming, and this introduces extra memory for each structure.
This series enhance the handling of those options by:
- Extract the validation part into a dedicated helper inside rootdir.[ch]
- Use st_dev/st_ino to replace
full_path
This reduces runtime and memory usage for the involved structures.- Remove the memory usage warning note
Even with the old 8K per structure memory usage, 1024 options will only 8M memory, that's accetable even for a lot of micro-controllers, not to mention modern desktop/servers.I'm a little paranoid at that time, with the memory usage almost halved, we can safely remove that warning note.
Quoted 8 lines
Qu Wenruo (5):
btrfs-progs: mkfs/rootdir: extract subvol validation code into a
helper
btrfs-progs: mkfs/rootdir: extract inode flags validation code into a
helper
btrfs-progs: mkfs/rootdir: enhance subvols detection
btrfs-progs: mkfs/rootdir: enhance inode flags detection
btrfs-progs: doc/mkfs: remove the note about memory usage
Looks good thanks, please add it do devel.