btrfs: add per-inode compression levels in xattrs
v4 cover btrfs: add per-inode compression levels in xattrs
Add per-inode compression levels using
btrfs property set /path/to/file compression "algo:level", using the same syntax as the compress mount option.
If set on folders, all new children will inherit the setting, while already existing children will be unaffected.
This patch series also fixes a small bug: before, when setting btrfs.compression, it would keep the mount option level, even if the algo was different (!!), so with e.g. compress=zstd:15 and btrfs.compression=zlib the data gets compressed at zlib:9 (because it would still get clamped at the right range).
After this patch series, if the user doesn't specify a level (e.g. "zstd") btrfs inherits the level from the mount option, while if instead the user specifies a level it will use that (e.g. "zstd:0" would use the default for the algorithm, and "zstd:10" would of course use level 10).
I also want to thank Zygo for helping me by explaining stuff and for noticing this bug.
Now, there is the question of having a fix for the bug as a separate patch to be backported or not, and there are 3 possibilities for that (original email listing them[1]):
- Option 1:
don't have a specific patch to be backported, keep the current behaviour on older kernels.
This is the option I personally prefer[2].
Pros:
* documentation[3] can be made clear about this behaviour, telling users that kernels < 7.X will "support" cross-algo level specification, while >= 7.X won't and will only inherit the level if the algorithm is the same
* won't break any existing use-case
Cons:
* new users on older kernels will encounter this bug
- Option 2:
have a specific patch to be backported that keeps the current behaviour, but only if the compress= algo is the same as the btrfs.compression one, if they aren't, use the default for the btrfs.compression algo (e.g. compress=zstd:15 and btrfs.compression=zlib would compress at zlib:3 instead of clamp(zlib, 15) = 9).
Pros:
* fixes the cross-algo bug without breaking setups that rely on level inheritance in a way that makes sense
Cons:
* documentation has to mention that the behaviour might be the bugged one or this one, depending on if the kernel has this bugfix patch
* may break some weird use-cases, but no one should really expect the level to be inherited if the algorithm is different
- Option 3:
have a specific patch to be backported that fixes the behaviour entirely, using the default level in all btrfs.compression cases, since levels wouldn't be supported.
Equivalent of setting algo:0 after this patch series gets applied.
Pros:
* very simple code fix, 2 lines of compress_level = 0, while option 2 would need an additional if to check the compress type against the fs_info one
Cons:
* documentation has to mention that the behaviour might be the bugged one or this one, depending on if the kernel has this bugfix patch
* breaks all use-cases relying on level inheritance from the mount option to the xattr
Changes in this v4:
* remove the single patch for the bug, as per [2]
* reduce code duplication by putting the level parsing logic after the algorithm type logic, using a level_pos variable to indicate the position of the ':' instead of having the same exact code for each algo.
* add btrfs_compress_typelevel2str() helper
* fix btrfs.compression not being preserved when *any* chattr operation gets executed changed
[1]: https://lore.kernel.org/linux-btrfs/[email protected]
[2]: https://lore.kernel.org/linux-btrfs/[email protected]
[3]: https://github.com/kdave/btrfs-progs/pull/1152
koraynilay (6):
btrfs: export btrfs_match_compress_type(), move it to compression.h
btrfs: also validate compression levels in
btrfs_compress_is_valid_type()
btrfs: add per-inode compression levels in xattrs
btrfs: support inheritance for per-inode compression levels
btrfs: add btrfs_compress_typelevel2str() helper
btrfs: preserve btrfs.compression when setting inode flags
fs/btrfs/btrfs_inode.h | 1 + fs/btrfs/compression.c | 55 +++++++++++++++++++++++++++++-- fs/btrfs/compression.h | 5 ++- fs/btrfs/inode.c | 10 ++++++ fs/btrfs/ioctl.c | 13 +++++++- fs/btrfs/props.c | 73 +++++++++++++++++++++++++++++------------- fs/btrfs/props.h | 9 ++++++ fs/btrfs/super.c | 8 ----- 8 files changed, 139 insertions(+), 35 deletions(-)
base-commit: 818bebeb63dd6bf5f4e07e145f6cdbace520a34c
Signature · 1 line
2.55.0
PATCH v4 1/6 btrfs: export btrfs_match_compress_type(), move it to compression.h
Export btrfs_match_compress_type() from being a static and locally used function in super.c and move it to compression.c, exporting it in compression.h. This allows it to be used in the next patches as validation for compression algorithm names.
--- fs/btrfs/compression.c | 17 +++++++++++++++++ fs/btrfs/compression.h | 1 + fs/btrfs/super.c | 8 -------- 3 files changed, 18 insertions(+), 8 deletions(-)
fs/btrfs/compression.c +17 -0
@@ -1650,3 +1650,20 @@ int btrfs_compress_str2level(unsigned int type, const char *str, int *level_ret) *level_ret =btrfs_compress_set_level (type, level);return 0 ; } + +/** +* btrfs_match_compress_type - Check if the string matches the compression type +* @string: The string to check +* @type: The compression type string (name) to match against (e.g. "zstd") +* @may_have_level: If true, the string may have a level suffix (e.g., ":1") +* +* Return: %true if the string matches the type and, if %may_have_level is %true, +* has a level suffix, %false otherwise. +*/ +bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level) +{ +const int len =strlen (type); + +return (strncmp (string, type, len) ==0 ) && + ((may_have_level && string[len] ==':' ) || string[len] =='\0' ); +}
fs/btrfs/compression.h +1 -0
@@ -97,6 +97,7 @@ void btrfs_submit_compressed_write(struct btrfs_ordered_extent *ordered,void btrfs_submit_compressed_read (struct btrfs_bio *bbio);int btrfs_compress_str2level (unsigned int type,const char *str,int *level_ret); +bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level);struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info,gfp_t gfp);void btrfs_free_compr_folio (struct folio *folio);
fs/btrfs/super.c +0 -8
@@ -261,14 +261,6 @@ static const struct fs_parameter_spec btrfs_fs_parameters[] = { {} }; -static bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level) -{ -const int len =strlen (type); - -return (strncmp (string, type, len) ==0 ) && - ((may_have_level && string[len] ==':' ) || string[len] =='\0' ); -} -static int btrfs_parse_compress (struct btrfs_fs_context *ctx,const struct fs_parameter *param,int opt) {
Signature · 1 line
2.55.0
PATCH v4 2/6 btrfs: also validate compression levels in btrfs_compress_is_valid_type()
In reply to koraynilayChange btrfs_compress_is_valid_type() to use btrfs_match_compress_type() and btrfs_compress_str2level() instead of a simple strncmp(), which would validate even incorrect strings (e.g. "zstd:invalid" or "zstdinvalid").
This also makes the function validate levels in the same way the compress= option gets parsed, allowing bigger or smaller values, but still clamping them to the min or max supported.
Furthermore, remove the len parameter, since now it requires a NUL-terminated string because of btrfs_compress_str2level(); this change is fine because btrfs_compress_is_valid_type() is used only once in props.c by prop_compression_validate(), which now uses kmemdup_nul() to convert the user-supplied xattr value to a NUL-terminated string.
--- fs/btrfs/compression.c | 18 +++++++++++++++--- fs/btrfs/compression.h | 2 +- fs/btrfs/props.c | 16 ++++++++++++---- 3 files changed, 28 insertions(+), 8 deletions(-)
fs/btrfs/compression.c +15 -3
@@ -70,18 +70,30 @@ static struct compressed_bio *alloc_compressed_bio(struct btrfs_inode *inode,return to_compressed_bio (bbio); } -bool btrfs_compress_is_valid_type (const char *str,size_t len) +/** +* btrfs_compress_is_valid_type - Check if a string is a valid compression type. +* @str: The compression string to check. Must be null-terminated. +* +* Return: %true if the string is a valid compression algorithm (optionally with +* a level suffix, e.g. "zstd" or "zstd:9"), %false otherwise. +*/ +bool btrfs_compress_is_valid_type (const char *str) { +size_t len =strlen (str);int i;for (i =1 ; i <ARRAY_SIZE (btrfs_compress_types); i++) {size_t comp_len =strlen (btrfs_compress_types[i]); +const char *comp_type = btrfs_compress_types[i]; +int tmp_level;if (len < comp_len)continue ; -if (!strncmp (btrfs_compress_types[i], str, comp_len)) -return true; +if (btrfs_match_compress_type (str, comp_type, true)) { +if (btrfs_compress_str2level (i, str + comp_len, &tmp_level) ==0 ) +return true; + } }return false; }
fs/btrfs/compression.h +1 -1
@@ -132,7 +132,7 @@ extern const struct btrfs_compress_levels btrfs_lzo_compress;extern const struct btrfs_compress_levels btrfs_zstd_compress;const char *btrfs_compress_type2str (enum btrfs_compression_type type); -bool btrfs_compress_is_valid_type (const char *str,size_t len); +bool btrfs_compress_is_valid_type (const char *str);int btrfs_compress_heuristic (struct btrfs_inode *inode,u64 start,u64 end);
fs/btrfs/props.c +12 -4
@@ -295,20 +295,28 @@ int btrfs_load_inode_props(struct btrfs_inode *inode, struct btrfs_path *path)static int prop_compression_validate (const struct btrfs_inode *inode,const char *value,size_t len) { +int ret = -EINVAL ; +const char *value_str; +if (!btrfs_inode_can_compress (inode))return -EINVAL ;if (!value)return 0 ; -if (btrfs_compress_is_valid_type (value, len)) -return 0 ; -if ((len ==2 &&strncmp ("no" , value,2 ) ==0 ) || (len ==4 &&strncmp ("none" , value,4 ) ==0 ))return 0 ; -return -EINVAL ; + value_str =kmemdup_nul (value, len,GFP_KERNEL ); +if (!value_str) +return -ENOMEM ; + +if (btrfs_compress_is_valid_type (value_str)) + ret =0 ; + +kfree (value_str); +return ret; } static int prop_compression_apply(struct btrfs_inode *inode, const char *value,
Signature · 1 line
2.55.0
PATCH v4 3/6 btrfs: add per-inode compression levels in xattrs
In reply to koraynilayAdd support for specifying algo:level using
btrfs property set /path/to/file compression "algo:level".
Add a signed 8 bit prop_compress_level property to btrfs_inode, which can support from level -128 to 127, plenty for the currently supported algo:level(s).
Change prop_compression_apply() to use the already present btrfs_match_compress_type() and btrfs_compress_str2level() to parse the xattr, using kmemdup_nul() to convert the user-supplied xattr value to a NUL-terminated string to be used by btrfs_compress_str2level().
--- fs/btrfs/btrfs_inode.h | 1 + fs/btrfs/inode.c | 10 ++++++++++ fs/btrfs/props.c | 36 +++++++++++++++++++++++++++++++----- 3 files changed, 42 insertions(+), 5 deletions(-)
fs/btrfs/btrfs_inode.h +1 -0
@@ -130,6 +130,7 @@ struct btrfs_inode {/* Cached value of inode property 'compression'. */ u8 prop_compress; +s8 prop_compress_level;/* * Force compression on the file using the defrag ioctl, could be
fs/btrfs/inode.c +10 -0
@@ -942,6 +942,11 @@ static void compress_file_range(struct btrfs_work *work) compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; +if (inode->prop_compress_level) + compress_level = inode->prop_compress_level; +else if (compress_type != fs_info->compress_type) + compress_level =0 ; +/* else use fs_info->compress_level */ }/* Compression level is applied here. */
@@ -2338,6 +2343,11 @@ static int run_delalloc_inline(struct btrfs_inode *inode, struct folio *locked_f compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; +if (inode->prop_compress_level) + compress_level = inode->prop_compress_level; +else if (compress_type != fs_info->compress_type) + compress_level =0 ; +/* else use fs_info->compress_level */ } cb =btrfs_compress_bio (inode,0 , blocksize, compress_type, compress_level,0 );if (IS_ERR (cb)) {
fs/btrfs/props.c +31 -5
@@ -324,6 +324,10 @@ static int prop_compression_apply(struct btrfs_inode *inode, const char *value, {struct btrfs_fs_info *fs_info = inode->root->fs_info;int type; +int level =0 ; +int ret =0 ; +const char *value_str; +int level_pos =0 ;/* Reset to defaults */ if (len ==0 ) {
@@ -343,23 +347,45 @@ static int prop_compression_apply(struct btrfs_inode *inode, const char *value,return 0 ; } -if (!strncmp ("lzo" , value,3 )) { + value_str =kmemdup_nul (value, len,GFP_KERNEL ); +if (!value_str) +return -ENOMEM ; + +if (btrfs_match_compress_type (value_str,"lzo" , true)) { type =BTRFS_COMPRESS_LZO ;btrfs_set_fs_incompat (fs_info,COMPRESS_LZO ); - }else if (!strncmp ("zlib" , value,4 )) { + }else if (btrfs_match_compress_type (value_str,"zlib" , true)) { type =BTRFS_COMPRESS_ZLIB ; - }else if (!strncmp ("zstd" , value,4 )) { + level_pos =4 ; + }else if (btrfs_match_compress_type (value_str,"zstd" , true)) { type =BTRFS_COMPRESS_ZSTD ; + level_pos =4 ;btrfs_set_fs_incompat (fs_info,COMPRESS_ZSTD ); }else { -return -EINVAL ; + ret = -EINVAL ; + goto out; + } + +/* Parse level if the algorithm supports it */ +if (level_pos >0 ) { +const char *level_str = value_str + level_pos; + +/* and if user specified it */ +if (level_str[0 ] ==':' ) { + ret =btrfs_compress_str2level (type, level_str, &level); +if (ret <0 ) + goto out; + } } inode->flags &= ~BTRFS_INODE_NOCOMPRESS ; inode->flags |=BTRFS_INODE_COMPRESS ; inode->prop_compress = type; + inode->prop_compress_level = level; -return 0 ; +out: +kfree (value_str); +return ret; }static bool prop_compression_ignore (const struct btrfs_inode *inode)
Signature · 1 line
2.55.0
PATCH v4 4/6 btrfs: support inheritance for per-inode compression levels
In reply to koraynilayChange prop_handler's extract() signature to take an output buffer and its length. This allows for the algo:level string to be generated dynamically, but now the caller is in charge of managing that buffer memory. Before this patch prop_compression_extract() would return only the address of statically compiled string without being able to add the level, now it concatenates the level to that statically compiled string.
This signature change is fine because the only currently supported prop is compression, so there is only 1 extract() function.
--- fs/btrfs/props.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
fs/btrfs/props.c +20 -3
@@ -18,6 +18,15 @@#include "super.h" #include "dir-item.h" +/* +* Max length of compression algorithm:level string. +* +* For now the longest possible string is "zstd:-15", which is +* 8 characters + 1 terminating null byte. +* Rounding it up to the closest power of 2 gives 16. +*/ +#define BTRFS_COMPRESS_PROP_MAX_LEN 16 +#define BTRFS_PROP_HANDLERS_HT_BITS 8 static DEFINE_HASHTABLE(prop_handlers_ht, BTRFS_PROP_HANDLERS_HT_BITS);
@@ -27,7 +36,7 @@ struct prop_handler {int (*validate)(const struct btrfs_inode *inode,const char *value,size_t len);int (*apply)(struct btrfs_inode *inode,const char *value,size_t len); -const char *(*extract)(const struct btrfs_inode *inode); +const char *(*extract)(const struct btrfs_inode *inode,char *buf,size_t len);bool (*ignore)(const struct btrfs_inode *inode);int inheritable; };
@@ -404,12 +413,19 @@ static bool prop_compression_ignore(const struct btrfs_inode *inode)return false; } -static const char *prop_compression_extract (const struct btrfs_inode *inode) +static const char *prop_compression_extract (const struct btrfs_inode *inode, +char *buf,size_t len) {switch (inode->prop_compress) {case BTRFS_COMPRESS_ZLIB :case BTRFS_COMPRESS_LZO :case BTRFS_COMPRESS_ZSTD : +if (inode->prop_compress_level) { +snprintf (buf, len,"%s:%d" , +btrfs_compress_type2str (inode->prop_compress), + inode->prop_compress_level); +return buf; + }return btrfs_compress_type2str (inode->prop_compress);default :break ;
@@ -446,6 +462,7 @@ int btrfs_inode_inherit_props(struct btrfs_trans_handle *trans,const struct prop_handler *h = &prop_handlers[i];const char *value;u64 num_bytes =0 ; +char buf[BTRFS_COMPRESS_PROP_MAX_LEN ];if (!h->inheritable)continue ;
@@ -453,7 +470,7 @@ int btrfs_inode_inherit_props(struct btrfs_trans_handle *trans,if (h->ignore (inode))continue ; - value = h->extract (parent); + value = h->extract (parent, buf,sizeof (buf));if (!value)continue ;
Signature · 1 line
2.55.0
PATCH v4 5/6 btrfs: add btrfs_compress_typelevel2str() helper
In reply to koraynilayAdd the type:level equivalent of btrfs_compress_type2str() and use it in prop_compression_extract() .
--- fs/btrfs/compression.c | 20 ++++++++++++++++++++ fs/btrfs/compression.h | 2 ++ fs/btrfs/props.c | 19 +++---------------- 3 files changed, 25 insertions(+), 16 deletions(-)
fs/btrfs/compression.c +20 -0
@@ -53,6 +53,26 @@ const char* btrfs_compress_type2str(enum btrfs_compression_type type)return NULL ; } +const char *btrfs_compress_typelevel2str (enum btrfs_compression_type type, +int level,char *buf,size_t len) +{ +switch (type) { +case BTRFS_COMPRESS_ZLIB : +case BTRFS_COMPRESS_LZO : +case BTRFS_COMPRESS_ZSTD : +if (level) { +snprintf (buf, len,"%s:%d" , +btrfs_compress_type2str (type), level); +return buf; + } +return btrfs_compress_type2str (type); +default : +break ; + } + +return NULL ; +} +static inline struct compressed_bio *to_compressed_bio (struct btrfs_bio *bbio) { return container_of(bbio, struct compressed_bio, bbio);
fs/btrfs/compression.h +2 -0
@@ -132,6 +132,8 @@ extern const struct btrfs_compress_levels btrfs_lzo_compress;extern const struct btrfs_compress_levels btrfs_zstd_compress;const char *btrfs_compress_type2str (enum btrfs_compression_type type); +const char *btrfs_compress_typelevel2str (enum btrfs_compression_type type, +int level,char *buf,size_t len);bool btrfs_compress_is_valid_type (const char *str);int btrfs_compress_heuristic (struct btrfs_inode *inode,u64 start,u64 end);
fs/btrfs/props.c +3 -16
@@ -416,22 +416,9 @@ static bool prop_compression_ignore(const struct btrfs_inode *inode)static const char *prop_compression_extract (const struct btrfs_inode *inode,char *buf,size_t len) { -switch (inode->prop_compress) { -case BTRFS_COMPRESS_ZLIB : -case BTRFS_COMPRESS_LZO : -case BTRFS_COMPRESS_ZSTD : -if (inode->prop_compress_level) { -snprintf (buf, len,"%s:%d" , -btrfs_compress_type2str (inode->prop_compress), - inode->prop_compress_level); -return buf; - } -return btrfs_compress_type2str (inode->prop_compress); -default : -break ; - } - -return NULL ; +return btrfs_compress_typelevel2str (inode->prop_compress, + inode->prop_compress_level, + buf, len); }static struct prop_handler prop_handlers[] = {
Signature · 1 line
2.55.0
PATCH v4 6/6 btrfs: preserve btrfs.compression when setting inode flags
In reply to koraynilayAny call to FS_IOC_SETFLAGS (e.g. via chattr), even when no flag bits change, and even for flags unrelated to compression, overwrites the btrfs.compression property with the mount default compression type.
Example:
# mount ... -o compress=zstd ...
$ touch zero
$ setfattr -n btrfs.compression -v zlib zero
$ getfattr -n btrfs.compression zero | grep =
btrfs.compression="zlib"
$ lsattr zero
--------c------------- zero
$ chattr +A zero
$ lsattr zero
-------Ac------------- zero
$ getfattr -n btrfs.compression zero | grep =
btrfs.compression="zstd"Here, +A modifies only the atime flag, but the compression property was silently replaced. The same happens even if the ioctl writes back the same flags value that was already set.
The problem is that btrfs_fileattr_set unconditionally regenerates the compression string from fs_info->compress_type (or falls back to "zlib") and overwrites any existing property.
Fix this by first checking for an existing per-inode compression property and using it if present. Only fall back to fs_info->compress_type or zlib when no property has been set. This ensures that inode-flag updates no longer clobber user-configured compression settings.
Also move BTRFS_COMPRESS_PROP_MAX_LEN from props.c to props.h so it can be used by ioctl.c, since it now requires a temporary buffer to write the property string to.
[ Commit message and code slightly adapted from Zygo's patch for this same
issue ]
--- fs/btrfs/ioctl.c | 13 ++++++++++++- fs/btrfs/props.c | 9 --------- fs/btrfs/props.h | 9 +++++++++ 3 files changed, 21 insertions(+), 10 deletions(-)
fs/btrfs/ioctl.c +12 -1
@@ -288,6 +288,7 @@ int btrfs_fileattr_set(struct mnt_idmap *idmap,unsigned int fsflags, old_fsflags;int ret;const char *comp =NULL ; +char comp_buf[BTRFS_COMPRESS_PROP_MAX_LEN ];u32 inode_flags;bool prop_set = false;
@@ -391,7 +392,17 @@ int btrfs_fileattr_set(struct mnt_idmap *idmap, inode_flags |=BTRFS_INODE_COMPRESS ; inode_flags &= ~BTRFS_INODE_NOCOMPRESS ; - comp =btrfs_compress_type2str (fs_info->compress_type); +/* +* If compression is already enabled, we must reconstruct the +* full "algo:level" property string and write it back, +* otherwise any chattr + operation would reset the compression +* algorithm to the fs_info one, also removing the level. +*/ + comp =btrfs_compress_typelevel2str (inode->prop_compress, + inode->prop_compress_level, + comp_buf,sizeof (comp_buf)); +if (!comp || comp[0 ] ==0 ) + comp =btrfs_compress_type2str (fs_info->compress_type);if (!comp || comp[0 ] ==0 ) comp =btrfs_compress_type2str (BTRFS_COMPRESS_ZLIB ); }else {
fs/btrfs/props.c +0 -9
@@ -18,15 +18,6 @@#include "super.h" #include "dir-item.h" -/* -* Max length of compression algorithm:level string. -* -* For now the longest possible string is "zstd:-15", which is -* 8 characters + 1 terminating null byte. -* Rounding it up to the closest power of 2 gives 16. -*/ -#define BTRFS_COMPRESS_PROP_MAX_LEN 16 -#define BTRFS_PROP_HANDLERS_HT_BITS 8 static DEFINE_HASHTABLE(prop_handlers_ht, BTRFS_PROP_HANDLERS_HT_BITS);
fs/btrfs/props.h +9 -0
@@ -9,6 +9,15 @@#include <linux/types.h> #include <linux/compiler_types.h> +/* +* Max length of compression algorithm:level string. +* +* For now the longest possible string is "zstd:-15", which is +* 8 characters + 1 terminating null byte. +* Rounding it up to the closest power of 2 gives 16. +*/ +#define BTRFS_COMPRESS_PROP_MAX_LEN 16 +struct btrfs_inode ;struct btrfs_path ;struct btrfs_trans_handle ;
Signature · 1 line
2.55.0