btrfs-progs: new --inode-flags option
Subsystems (2)
cover btrfs-progs: new --inode-flags option
The new --inode-flags option allows us to specify certain btrfs specific flags to each inode.
Currently we only support *nodatacow* and *nodatasum*.
But in the future compression flag can also be added, allowing more accurate per-file compression.
Furthermore child inodes will inherit the flag from their parents, meaning one only needs to specify the flag to the parent directory, then all children files/directories will have the flag.
This new option also works well with --subvol, although one has to note that, the inode flag inheritance does not cross subvolume boundary (the same as the kernel).
Finally, nodatacow and nodatasum will disable compression, just like the kernel.
Qu Wenruo (4):
btrfs-progs: allow new inodes to inherit flags from their parents
btrfs-progs: do not generate checksum nor compress if the inode has
NODATACOW or NODATASUM
btrfs-progs: mkfs: add --inode-flags option
btrfs-progs: mkfs-tests: a new test case for --inode-flags
Documentation/mkfs.btrfs.rst | 35 +++++++ kernel-shared/inode.c | 64 ++++++++++++ mkfs/main.c | 120 ++++++++++++++++++++++- mkfs/rootdir.c | 95 ++++++++++++++++-- mkfs/rootdir.h | 15 +++ tests/mkfs-tests/038-inode-flags/test.sh | 55 +++++++++++ 6 files changed, 375 insertions(+), 9 deletions(-) create mode 100755 tests/mkfs-tests/038-inode-flags/test.sh
--
2.49.0
PATCH 4/4 btrfs-progs: mkfs-tests: a new test case for --inode-flags
The simple test will create a layout like the following:
rootdir
|- file1 |- file2 |- subv/ << Regular subvolume | |- file3 |- nocow_subv/ << NODATACOW subvolume | |- file4 |- nocow_dir/ << NODATACOW directory | |- dir2 | | |- file5 | |- file6 |- nocow_file1 << NODATACOW file
Any files under NODATACOW subvolume/directory should also be NODATACOW. The explicitly specified single file should also be NODATACOW.
--- tests/mkfs-tests/038-inode-flags/test.sh | 55 ++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100755 tests/mkfs-tests/038-inode-flags/test.sh
tests/mkfs-tests/038-inode-flags/test.sh new file +55 -0
@@ -0,0 +1,55 @@ +#!/bin/bash +# Basic test for mkfs.btrfs --inode-flags --rootdir. Create a dataset and use it as +# rootdir, then various inode-flags and verify the flag is properly set. + +source "$TEST_TOP/common" || exit + +check_prereq mkfs.btrfs +check_prereq btrfs +check_global_prereq lsattr + +setup_root_helper +prepare_test_dev + +tmp=$(_mktemp_dir mkfs-rootdir) + +write_file() +{ + local path="$1" + local size="$2" + + run_check dd if=/dev/zero of="$path" bs="$size" count=1 status=noxfer > /dev/null 2>&1 +} + +check_nodatacow() +{ + local path="$1" + + lsattr "$TEST_MNT"/"$path" | grep -q C || _fail "missing NODATACOW flag for $path" +} + +write_file "$tmp/file1" 64K +write_file "$tmp/file2" 64K +run_check mkdir -p "$tmp/subv" "$tmp/nocow_subv" "$tmp/nocow_dir/dir2" +write_file "$tmp/subv/file3" 64K +write_file "$tmp/nocow_subv/file4" 64K +write_file "$tmp/nocow_dir/dir2/file5" 64K +write_file "$tmp/nocow_dir/file6" 64K +write_file "$tmp/nocow_file1" 64K + +run_check_mkfs_test_dev --rootdir "$tmp" --inode-flags nodatacow:nocow_subv \ + --subvol nocow_subv --inode-flags nodatacow:nocow_dir \ + --inode-flags nodatacow:nocow_file1 + +run_check $SUDO_HELPER "$TOP/btrfs" check "$TEST_DEV" + +run_check_mount_test_dev +check_nodatacow "nocow_subv" +check_nodatacow "nocow_subv/file4" +check_nodatacow "nocow_dir" +check_nodatacow "nocow_dir/file6" +check_nodatacow "nocow_dir/dir2/file5" +check_nodatacow "nocow_file1" +run_check lsattr -R "$TEST_MNT" +run_check_umount_test_dev +run_check rm -rf -- "$tmp"
Signature · 1 line
2.49.0
PATCH 1/4 btrfs-progs: allow new inodes to inherit flags from their parents
In reply to Qu WenruoInside btrfs-progs (mostly 'mkfs/rootdir.c') new inodes are created in a different way than the kernel:
- A new orphan inode item is created
Without knowing the parent inode, thus it will always have the default flags (0).
- Link the new inode to a parent
Meanwhile inside the kernel, we know exactly the parent inode at new inode creation time, and can inherit the inode flags (NODATACOW/NODATASUM/COMPRESS/etc).
Address the missing ability by:
- Inherit the parent inode flags when linking an orphan inode The function btrfs_add_link() is called when linking an inode.
It can be called to creating the initial link if it's a new and orphan inode.
It can also be called to creating extra hard links.
If the inode is already orphan, we know it's newly created and should inherit the inode flag from the parent.
With this new ability, it will be much easier to implement new per-inode flags (like NODATACOW/NODATASUM) and get them properly passed down.
--- kernel-shared/inode.c | 64 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+)
kernel-shared/inode.c +64 -0
@@ -158,6 +158,59 @@ out:return ret; } +/* Similiar to btrfs_inherit_iflags(), but different interfaces. */ +static int inherit_inode_flags (struct btrfs_trans_handle *trans, +struct btrfs_root *root,u64 ino,u64 parent_ino) +{ +struct btrfs_path path = {0 }; +struct btrfs_key key; +struct btrfs_inode_item *iitem; +u64 parent_inode_flags; +u64 child_inode_flags; +int ret; + + key.objectid = parent_ino; + key.type =BTRFS_INODE_ITEM_KEY ; + key.offset =0 ; + + ret =btrfs_search_slot (NULL , root, &key, &path,0 ,0 ); +if (ret >0 ) + ret = -ENOENT ; +if (ret <0 ) + goto out; + + iitem = btrfs_item_ptr(path.nodes[0], path.slots[0], struct btrfs_inode_item); + parent_inode_flags =btrfs_inode_flags (path.nodes[0 ], iitem); +btrfs_release_path (&path); + + key.objectid = ino; + + ret =btrfs_search_slot (trans, root, &key, &path,0 ,1 ); +if (ret >0 ) + ret = -ENOENT ; +if (ret <0 ) + goto out; + iitem = btrfs_item_ptr(path.nodes[0], path.slots[0], struct btrfs_inode_item); + child_inode_flags =btrfs_inode_flags (path.nodes[0 ], iitem); + +if (parent_inode_flags &BTRFS_INODE_NOCOMPRESS ) { + child_inode_flags &= ~BTRFS_INODE_COMPRESS ; + child_inode_flags |=BTRFS_INODE_NOCOMPRESS ; + }else if (parent_inode_flags &BTRFS_INODE_COMPRESS ){ + child_inode_flags &= ~BTRFS_INODE_NOCOMPRESS ; + child_inode_flags |=BTRFS_INODE_COMPRESS ; + } +if (parent_inode_flags &BTRFS_INODE_NODATACOW ) { + child_inode_flags |=BTRFS_INODE_NODATACOW ; +if (S_ISREG (btrfs_inode_mode (path.nodes[0 ], iitem))) + child_inode_flags |=BTRFS_INODE_NODATASUM ; + } +btrfs_set_inode_flags (path.nodes[0 ], iitem, child_inode_flags); +out: +btrfs_release_path (&path); +return ret; +} +/* * Add dir_item/index for 'parent_ino' if add_backref is true, also insert a * backref from the ino to parent dir and update the nlink(Kernel version does
@@ -220,6 +273,17 @@ int btrfs_add_link(struct btrfs_trans_handle *trans, struct btrfs_root *root, nlink);btrfs_mark_buffer_dirty (path->nodes[0 ]);btrfs_release_path (path); +/* +* If this is the first nlink of the inode, meaning the +* inode is newly created under the parent inode, this +* new child inode should inherit the inode flags from +* the parent. +*/ +if (nlink ==1 ) { + ret =inherit_inode_flags (trans, root, ino, parent_ino); +if (ret <0 ) + goto out; + } } }
Signature · 1 line
2.49.0
PATCH 2/4 btrfs-progs: do not generate checksum nor compress if the inode has NODATACOW or NODATASUM
In reply to Qu WenruoCurrently mkfs.btrfs --rootdir is implying data checksum, but soon we will support per-inode NODATACOW|NODATASUM flags.
To support per-inode NODATACOW|NODATASUM flags:
- Avoid compression if the inode has either NODATACOW|NODATASUM flag
- Do not generate data checksum if the inode has either NODATACOW|NODATASUM flag.
Both behaviors are the following the kernel ones.
--- mkfs/rootdir.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
mkfs/rootdir.c +13 -4
@@ -716,12 +716,19 @@ static int add_file_item_extent(struct btrfs_trans_handle *trans,u64 buf_size;char *write_buf;bool do_comp = g_compression !=BTRFS_COMPRESS_NONE ; +bool datasum = true;ssize_t comp_ret;u64 flags =btrfs_stack_inode_flags (btrfs_inode);if (flags &BTRFS_INODE_NOCOMPRESS ) do_comp = false; +if (flags &BTRFS_INODE_NODATACOW || + flags &BTRFS_INODE_NODATASUM ) { + datasum = false; + do_comp = false; + } + buf_size = do_comp ?BTRFS_MAX_COMPRESSED :MAX_EXTENT_SIZE ; to_read =min (file_pos + buf_size, source->size) - file_pos;
@@ -852,13 +859,15 @@ static int add_file_item_extent(struct btrfs_trans_handle *trans,return ret; } -for (unsigned int i =0 ; i < to_write / sectorsize; i++) { - ret =btrfs_csum_file_block (trans, first_block + (i * sectorsize), +if (datasum) { +for (unsigned int i =0 ; i < to_write / sectorsize; i++) { + ret =btrfs_csum_file_block (trans, first_block + (i * sectorsize),BTRFS_EXTENT_CSUM_OBJECTID , root->fs_info->csum_type, write_buf + (i * sectorsize)); -if (ret) -return ret; +if (ret) +return ret; + } }btrfs_set_stack_file_extent_type (&stack_fi,BTRFS_FILE_EXTENT_REG );
Signature · 1 line
2.49.0
PATCH 3/4 btrfs-progs: mkfs: add --inode-flags option
In reply to Qu WenruoThis new option allows end users to specify certain per-inode flags for specified file/directory inside rootdir.
And mkfs will follow the kernel behavior by inheriting the inode flag from the parent.
For example:
rootdir
|- file1
|- file2
|- dir1/
| |- file3
|- subv/ << will be created as a subvolume using --subvol option
|- dir2/
| |- file4
|- file5When `mkfs.btrfs --rootdir rootdir --subvol subv --inode-flags nodatacow:dir1 --inode-flags nodatacow:subv", then the following files and directory will have *nodatacow* flag set:
- dir1
- file3
- subv
- dir2
- file4
- file5
For now only two flags are supported:
- nodatacow
Disable data COW, implies *nodatasum* for regular files
- nodatasum
Disable data checksum only.
This also works with --compress option, and files with nodatasum or nodatacow flag will skip compression.
--- Documentation/mkfs.btrfs.rst | 35 ++++++++++ mkfs/main.c | 120 ++++++++++++++++++++++++++++++++++- mkfs/rootdir.c | 78 +++++++++++++++++++++-- mkfs/rootdir.h | 15 +++++ 4 files changed, 243 insertions(+), 5 deletions(-)
Documentation/mkfs.btrfs.rst +35 -0
@@ -213,6 +213,41 @@ OPTIONS :file:`hardlink1` and :file:`hardlink2` because :file:`hardlink3` will be inside a new subvolume. +--inode-flags <flags>:<path> + Specify that *path* to have inode *flags*, other than the default one (which + implies data CoW and data checksum). The option *--rootdir* must also be + specified. This option can be speicified multiple times. + + The supported flag(s) are: + + * *nodatacow*: disable data CoW, implies *nodatasum* for regular files. + * *nodatasum*: disable data checksum only. + + *flags* can be separated by comma (','). + + Children inodes will inherit the flags from their parent inodes, like the + following case: + + .. code-block:: none + + rootdir/ + |- file1 + |- file2 + |- dir/ + |- file3 + + In that case, if *--inode-flags nodatacow:dir* is specified, both + :file:`dir` and :file:`file3` will have the *nodatacow* flag. + + And this option also works with *--subvol* option, but the inode flag of + each subvolume is independent and will not inherit from the parent directory. + (The same as the kernel behavior) + + .. note:: + Both *--inode-flags* and *--subvol* options are memory hungry, + will consume at least 8KiB for each option. + Please keep the usage of both options to minimal. + --shrink Shrink the filesystem to its minimal size, only works with *--rootdir* option.
mkfs/main.c +119 -1
180 lines not on this page · raw message
mkfs/rootdir.c +74 -4
127 lines not on this page · raw message
mkfs/rootdir.h +15 -0
@@ -45,8 +45,23 @@ struct rootdir_subvol {bool readonly; }; +/* +* Represent a flag for specified inode at @full_path. +*/ +struct rootdir_inode_flags_entry { +struct list_head list; +/* Fully canonicalized path to the source file. */ +char full_path[PATH_MAX ]; +/* Path inside the source directory. */ +char inode_path[PATH_MAX ]; + +bool nodatacow; +bool nodatasum; +}; +int btrfs_mkfs_fill_dir (struct btrfs_trans_handle *trans,const char *source_dir,struct btrfs_root *root,struct list_head *subvols, +struct list_head *inode_flags_list,enum btrfs_compression_type compression,unsigned int compression_level); u64 btrfs_mkfs_size_dir(const char *dir_name, u32 sectorsize, u64 min_dev_size,
Signature · 1 line
2.49.0