btrfs: add per-inode compression levels in xattrs
v3 cover btrfs: add per-inode compression levels in xattrs
Add per-inode compression levels using
btrfs property set /path/to/file compression "algo:level", using the same syntax as the compress mount option.
If set on folders, all new children will inherit the setting, while already existing children will be unaffected.
This patch series also fixes a small bug: before, when setting btrfs.compression, it would keep the mount option level, even if the algo was different (!!), so with e.g. compress=zstd:15 and btrfs.compression=zlib the data gets compressed at zlib:9 (because it would still get clamped at the right range); now if the level isn't explicitly set it will use the default level for the algo.
I also wanna thank Zygo for helping me by explaining stuff and for noticing this bug.
Changes in this v3:
* the first patch contains only the bugfix for the mentioned bug, so it can be backported to older kernels if needed;
* fix a bisection and compilation error.
koraynilay (5):
btrfs: fix -o compress= level getting inherited by compression xattr
property
btrfs: export btrfs_match_compress_type(), move it to compression.h
btrfs: also validate compression levels in
btrfs_compress_is_valid_type()
btrfs: add per-inode compression levels in xattrs
btrfs: support inheritance for per-inode compression levels
fs/btrfs/btrfs_inode.h | 1 + fs/btrfs/compression.c | 35 ++++++++++++++++++++-- fs/btrfs/compression.h | 3 +- fs/btrfs/inode.c | 2 ++ fs/btrfs/props.c | 66 ++++++++++++++++++++++++++++++++++-------- fs/btrfs/super.c | 8 ----- 6 files changed, 91 insertions(+), 24 deletions(-)
base-commit: a13307e97d5c54b65720bb71fa379960ded1e51a
Signature · 1 line
2.55.0
PATCH v3 1/5 btrfs: fix -o compress= level getting inherited by compression xattr property
When setting the btrfs.compression xattr, btrfs uses the mount option level (since levels in the xattr aren't supported yet) to compress the data, even if the compression algorithm between the property and the compress= option are different (!!).
For example, when mounting with compress=zstd:15 and setting btrfs.compression=zlib, the data would get compressed at zlib:9, because btrfs would take "zlib" from the xattr but "15" from the mount option, which would then get correctly clamped at 9.
To fix, just hard-code the compress level at 0, forcing the property's algorithm default.
--- fs/btrfs/inode.c | 2 ++ 1 file changed, 2 insertions(+)
fs/btrfs/inode.c +2 -0
@@ -938,6 +938,7 @@ static void compress_file_range(struct btrfs_work *work) compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ; }/* Compression level is applied here. */
@@ -2326,6 +2327,7 @@ static int run_delalloc_inline(struct btrfs_inode *inode, struct folio *locked_f compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ; } cb =btrfs_compress_bio (inode,0 , blocksize, compress_type, compress_level,0 );if (IS_ERR (cb)) {
Signature · 1 line
2.55.0
Re: [PATCH v3 1/5] btrfs: fix -o compress= level getting inherited by compression xattr property
Quoted 23 lines · koraynilay
When setting the btrfs.compression xattr, btrfs uses the mount option level (since levels in the xattr aren't supported yet) to compress the data, even if the compression algorithm between the property and the compress= option are different (!!).
For example, when mounting with compress=zstd:15 and setting btrfs.compression=zlib, the data would get compressed at zlib:9, because btrfs would take "zlib" from the xattr but "15" from the mount option, which would then get correctly clamped at 9.To fix, just hard-code the compress level at 0, forcing the property's algorithm default.
Reported-by Zygo Blaxell <[email protected]>Signed-off-by koraynilay <[email protected]>--- fs/btrfs/inode.c | 2 ++ 1 file changed, 2 insertions(+)fs/btrfs/inode.c +0
-0
fs/btrfs/inode.c static void compress_file_range(struct btrfs_work *work) · line 941
Quoted 4 lines
fs/btrfs/inode.c +1
-0@@ -938,6 +938,7 @@ static void compress_file_range(struct btrfs_work *work) compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ;
This set to zero should be conditional on inode->prop_compress != fs_info->compress_type; otherwise, it prevents the compress-level mount option from working in the common case where 'btrfs.compression=zstd' or 'chattr +c' has been set.
Quoted 15 lines
fs/btrfs/inode.c +1
-0}/* Compression level is applied here. */ @@ -2326,6 +2327,7 @@ static int run_delalloc_inline(struct btrfs_inode *inode, struct folio *locked_f compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ; } cb =btrfs_compress_bio (inode,0 , blocksize, compress_type, compress_level,0 );if (IS_ERR (cb)) {Signature · 1 line
2.55.0
Re: [PATCH v3 1/5] btrfs: fix -o compress= level getting inherited by compression xattr property
Quoted 19 lines · Zygo Blaxell
Quoted 18 lines · koraynilay
When setting the btrfs.compression xattr, btrfs uses the mount option level (since levels in the xattr aren't supported yet) to compress the data, even if the compression algorithm between the property and the compress= option are different (!!).
For example, when mounting with compress=zstd:15 and setting btrfs.compression=zlib, the data would get compressed at zlib:9, because btrfs would take "zlib" from the xattr but "15" from the mount option, which would then get correctly clamped at 9.To fix, just hard-code the compress level at 0, forcing the property's algorithm default.
Reported-by Zygo Blaxell <[email protected]>Signed-off-by koraynilay <[email protected]>--- fs/btrfs/inode.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 14 lines
fs/btrfs/inode.c static void compress_file_range(struct btrfs_work *work) · line 941
Quoted 9 lines
fs/btrfs/inode.c +1
-0@@ -938,6 +938,7 @@ static void compress_file_range(struct btrfs_work *work) compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ;This set to zero should be conditional on
inode->prop_compress!=fs_info->compress_type; otherwise, it prevents the compress-level mount option from working in the common case where 'btrfs.compression=zstd' or 'chattr +c' has been set.
No, read this discussion.
https://lore.kernel.org/linux-btrfs/[email protected]/T/#m79e6c1ecbcc61ebb57dca5cbb3eceabd052ecdbc
Quoted 16 lines
Quoted 16 lines
fs/btrfs/inode.c +1
-0}/* Compression level is applied here. */ @@ -2326,6 +2327,7 @@ static int run_delalloc_inline(struct btrfs_inode *inode, struct folio *locked_f compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; + compress_level =0 ; } cb =btrfs_compress_bio (inode,0 , blocksize, compress_type, compress_level,0 );if (IS_ERR (cb)) {Signature · 1 line
2.55.0
PATCH v3 2/5 btrfs: export btrfs_match_compress_type(), move it to compression.h
In reply to koraynilayExport btrfs_match_compress_type() from being a static and locally used function in super.c and move it to compression.c, exporting it in compression.h. This allows it to be used in the next patches as validation for compression algorithm names.
--- fs/btrfs/compression.c | 17 +++++++++++++++++ fs/btrfs/compression.h | 1 + fs/btrfs/super.c | 8 -------- 3 files changed, 18 insertions(+), 8 deletions(-)
fs/btrfs/compression.c +17 -0
@@ -1660,3 +1660,20 @@ int btrfs_compress_str2level(unsigned int type, const char *str, int *level_ret) *level_ret =btrfs_compress_set_level (type, level);return 0 ; } + +/** +* btrfs_match_compress_type - Check if the string matches the compression type +* @string: The string to check +* @type: The compression type string (name) to match against (e.g. "zstd") +* @may_have_level: If true, the string may have a level suffix (e.g., ":1") +* +* Return: %true if the string matches the type and, if %may_have_level is %true, +* has a level suffix, %false otherwise. +*/ +bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level) +{ +const int len =strlen (type); + +return (strncmp (string, type, len) ==0 ) && + ((may_have_level && string[len] ==':' ) || string[len] =='\0' ); +}
fs/btrfs/compression.h +1 -0
@@ -97,6 +97,7 @@ void btrfs_submit_compressed_write(struct btrfs_ordered_extent *ordered,void btrfs_submit_compressed_read (struct btrfs_bio *bbio);int btrfs_compress_str2level (unsigned int type,const char *str,int *level_ret); +bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level);struct folio *btrfs_alloc_compr_folio (struct btrfs_fs_info *fs_info,gfp_t gfp);void btrfs_free_compr_folio (struct folio *folio);
fs/btrfs/super.c +0 -8
@@ -264,14 +264,6 @@ static const struct fs_parameter_spec btrfs_fs_parameters[] = { {} }; -static bool btrfs_match_compress_type (const char *string,const char *type,bool may_have_level) -{ -const int len =strlen (type); - -return (strncmp (string, type, len) ==0 ) && - ((may_have_level && string[len] ==':' ) || string[len] =='\0' ); -} -static int btrfs_parse_compress (struct btrfs_fs_context *ctx,const struct fs_parameter *param,int opt) {
Signature · 1 line
2.55.0
PATCH v3 3/5 btrfs: also validate compression levels in btrfs_compress_is_valid_type()
In reply to koraynilayChange btrfs_compress_is_valid_type() to use btrfs_match_compress_type() and btrfs_compress_str2level() instead of a simple strncmp(), which would validate even incorrect strings (e.g. "zstd:invalid" or "zstdinvalid").
This also makes the function validate levels in the same way the compress= option gets parsed, allowing bigger or smaller values, but still clamping them to the min or max supported.
Furthermore, remove the len parameter, since now it requires a NUL-terminated string because of btrfs_compress_str2level(); this change is fine because btrfs_compress_is_valid_type() is used only once in props.c by prop_compression_validate(), which now uses kmemdup_nul() to convert the user-supplied xattr value to a NUL-terminated string.
--- fs/btrfs/compression.c | 18 +++++++++++++++--- fs/btrfs/compression.h | 2 +- fs/btrfs/props.c | 16 ++++++++++++---- 3 files changed, 28 insertions(+), 8 deletions(-)
fs/btrfs/compression.c +15 -3
@@ -70,18 +70,30 @@ static struct compressed_bio *alloc_compressed_bio(struct btrfs_inode *inode,return to_compressed_bio (bbio); } -bool btrfs_compress_is_valid_type (const char *str,size_t len) +/** +* btrfs_compress_is_valid_type - Check if a string is a valid compression type. +* @str: The compression string to check. Must be null-terminated. +* +* Return: %true if the string is a valid compression algorithm (optionally with +* a level suffix, e.g. "zstd" or "zstd:9"), %false otherwise. +*/ +bool btrfs_compress_is_valid_type (const char *str) { +size_t len =strlen (str);int i;for (i =1 ; i <ARRAY_SIZE (btrfs_compress_types); i++) {size_t comp_len =strlen (btrfs_compress_types[i]); +const char *comp_type = btrfs_compress_types[i]; +int tmp_level;if (len < comp_len)continue ; -if (!strncmp (btrfs_compress_types[i], str, comp_len)) -return true; +if (btrfs_match_compress_type (str, comp_type, true)) { +if (btrfs_compress_str2level (i, str + comp_len, &tmp_level) ==0 ) +return true; + } }return false; }
fs/btrfs/compression.h +1 -1
@@ -132,7 +132,7 @@ extern const struct btrfs_compress_levels btrfs_lzo_compress;extern const struct btrfs_compress_levels btrfs_zstd_compress;const char *btrfs_compress_type2str (enum btrfs_compression_type type); -bool btrfs_compress_is_valid_type (const char *str,size_t len); +bool btrfs_compress_is_valid_type (const char *str);int btrfs_compress_heuristic (struct btrfs_inode *inode,u64 start,u64 end);
fs/btrfs/props.c +12 -4
@@ -295,20 +295,28 @@ int btrfs_load_inode_props(struct btrfs_inode *inode, struct btrfs_path *path)static int prop_compression_validate (const struct btrfs_inode *inode,const char *value,size_t len) { +int ret = -EINVAL ; +const char *value_str; +if (!btrfs_inode_can_compress (inode))return -EINVAL ;if (!value)return 0 ; -if (btrfs_compress_is_valid_type (value, len)) -return 0 ; -if ((len ==2 &&strncmp ("no" , value,2 ) ==0 ) || (len ==4 &&strncmp ("none" , value,4 ) ==0 ))return 0 ; -return -EINVAL ; + value_str =kmemdup_nul (value, len,GFP_KERNEL ); +if (!value_str) +return -ENOMEM ; + +if (btrfs_compress_is_valid_type (value_str)) + ret =0 ; + +kfree (value_str); +return ret; } static int prop_compression_apply(struct btrfs_inode *inode, const char *value,
Signature · 1 line
2.55.0
PATCH v3 4/5 btrfs: add per-inode compression levels in xattrs
In reply to koraynilayAdd support for specifying algo:level using
btrfs property set /path/to/file compression "algo:level".
Add a signed 8 bit prop_compress_level property to btrfs_inode, which can support from level -128 to 127, plenty for the currently supported algo:level(s).
Change prop_compression_apply() to use the already present btrfs_match_compress_type() and btrfs_compress_str2level() to parse the xattr, using kmemdup_nul() to convert the user-supplied xattr value to a NUL-terminated string to be used by btrfs_compress_str2level().
--- fs/btrfs/btrfs_inode.h | 1 + fs/btrfs/inode.c | 4 ++-- fs/btrfs/props.c | 27 ++++++++++++++++++++++----- 3 files changed, 25 insertions(+), 7 deletions(-)
fs/btrfs/btrfs_inode.h +1 -0
@@ -130,6 +130,7 @@ struct btrfs_inode {/* Cached value of inode property 'compression'. */ u8 prop_compress; +s8 prop_compress_level;/* * Force compression on the file using the defrag ioctl, could be
fs/btrfs/inode.c +2 -2
@@ -938,7 +938,7 @@ static void compress_file_range(struct btrfs_work *work) compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; - compress_level =0 ; + compress_level = inode->prop_compress_level; }/* Compression level is applied here. */
@@ -2327,7 +2327,7 @@ static int run_delalloc_inline(struct btrfs_inode *inode, struct folio *locked_f compress_level = inode->defrag_compress_level; }else if (inode->prop_compress) { compress_type = inode->prop_compress; - compress_level =0 ; + compress_level = inode->prop_compress_level; } cb =btrfs_compress_bio (inode,0 , blocksize, compress_type, compress_level,0 );if (IS_ERR (cb)) {
fs/btrfs/props.c +22 -5
@@ -324,6 +324,9 @@ static int prop_compression_apply(struct btrfs_inode *inode, const char *value, {struct btrfs_fs_info *fs_info = inode->root->fs_info;int type; +int level =0 ; +int ret =0 ; +const char *value_str;/* Reset to defaults */ if (len ==0 ) {
@@ -343,23 +346,37 @@ static int prop_compression_apply(struct btrfs_inode *inode, const char *value,return 0 ; } -if (!strncmp ("lzo" , value,3 )) { + value_str =kmemdup_nul (value, len,GFP_KERNEL ); +if (!value_str) +return -ENOMEM ; + +if (btrfs_match_compress_type (value_str,"lzo" , true)) { type =BTRFS_COMPRESS_LZO ;btrfs_set_fs_incompat (fs_info,COMPRESS_LZO ); - }else if (!strncmp ("zlib" , value,4 )) { + }else if (btrfs_match_compress_type (value_str,"zlib" , true)) { type =BTRFS_COMPRESS_ZLIB ; - }else if (!strncmp ("zstd" , value,4 )) { + ret =btrfs_compress_str2level (type, value_str +4 , &level); +if (ret <0 ) + goto out; + }else if (btrfs_match_compress_type (value_str,"zstd" , true)) { type =BTRFS_COMPRESS_ZSTD ; + ret =btrfs_compress_str2level (type, value_str +4 , &level); +if (ret <0 ) + goto out;btrfs_set_fs_incompat (fs_info,COMPRESS_ZSTD ); }else { -return -EINVAL ; + ret = -EINVAL ; + goto out; } inode->flags &= ~BTRFS_INODE_NOCOMPRESS ; inode->flags |=BTRFS_INODE_COMPRESS ; inode->prop_compress = type; + inode->prop_compress_level = level; -return 0 ; +out: +kfree (value_str); +return ret; }static bool prop_compression_ignore (const struct btrfs_inode *inode)
Signature · 1 line
2.55.0
PATCH v3 5/5 btrfs: support inheritance for per-inode compression levels
In reply to koraynilayChange prop_handler's extract() signature to take an output buffer and its length. This allows for the algo:level string to be generated dynamically, but now the caller is in charge of managing that buffer memory. Before this patch prop_compression_extract() would return only the address of statically compiled string without being able to add the level, now it concatenates the level to that statically compiled string.
This signature change is fine because the only currently supported prop is compression, so there is only 1 extract() function.
--- fs/btrfs/props.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
fs/btrfs/props.c +20 -3
@@ -18,6 +18,15 @@#include "super.h" #include "dir-item.h" +/* +* Max length of compression algorithm:level string. +* +* For now the longest possible string is "zstd:-15", which is +* 8 characters + 1 terminating null byte. +* Rounding it up to the closest power of 2 gives 16. +*/ +#define BTRFS_COMPRESS_PROP_MAX_LEN 16 +#define BTRFS_PROP_HANDLERS_HT_BITS 8 static DEFINE_HASHTABLE(prop_handlers_ht, BTRFS_PROP_HANDLERS_HT_BITS);
@@ -27,7 +36,7 @@ struct prop_handler {int (*validate)(const struct btrfs_inode *inode,const char *value,size_t len);int (*apply)(struct btrfs_inode *inode,const char *value,size_t len); -const char *(*extract)(const struct btrfs_inode *inode); +const char *(*extract)(const struct btrfs_inode *inode,char *buf,size_t len);bool (*ignore)(const struct btrfs_inode *inode);int inheritable; };
@@ -395,12 +404,19 @@ static bool prop_compression_ignore(const struct btrfs_inode *inode)return false; } -static const char *prop_compression_extract (const struct btrfs_inode *inode) +static const char *prop_compression_extract (const struct btrfs_inode *inode, +char *buf,size_t len) {switch (inode->prop_compress) {case BTRFS_COMPRESS_ZLIB :case BTRFS_COMPRESS_LZO :case BTRFS_COMPRESS_ZSTD : +if (inode->prop_compress_level) { +snprintf (buf, len,"%s:%d" , +btrfs_compress_type2str (inode->prop_compress), + inode->prop_compress_level); +return buf; + }return btrfs_compress_type2str (inode->prop_compress);default :break ;
@@ -437,6 +453,7 @@ int btrfs_inode_inherit_props(struct btrfs_trans_handle *trans,const struct prop_handler *h = &prop_handlers[i];const char *value;u64 num_bytes =0 ; +char buf[BTRFS_COMPRESS_PROP_MAX_LEN ];if (!h->inheritable)continue ;
@@ -444,7 +461,7 @@ int btrfs_inode_inherit_props(struct btrfs_trans_handle *trans,if (h->ignore (inode))continue ; - value = h->extract (parent); + value = h->extract (parent, buf,sizeof (buf));if (!value)continue ;
Signature · 1 line
2.55.0