btrfs-progs: Add zstd support
Subsystems (4)
Nick Terrell
maintainerPATCH v2 1/1 btrfs-progs: Add zstd support
Adds zstd support to the btrfs program. An optional dependency on libzstd
Quoted 1 line
= 1.0.0 is added. Autoconf accepts
--enable-zstd' or--disable-zstd' and
defaults to detecting if libzstd is present using `pkg-config'. Adds tests for the new features based on a prebuilt btrfs image with a zstd compressed file.
The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI with the new tests. The prebuilt binary is available there.
I haven't updated Android.mk.
[1] https://github.com/terrelln/btrfs-progs/tree/devel
Signed-off-by Nick Terrell <[email protected]>
---
v1 -> v2:
- Allow zstd support to be disabled, defaulting to auto detection
- Print if zstd is enabled/disabled when `./configure' finishes
- Add zstd test to patch with prebuilt image`
.travis.yml | 9 ++++ Documentation/btrfs-filesystem.asciidoc | 2 +- Documentation/btrfs-man5.asciidoc | 8 +++- Documentation/btrfs-property.asciidoc | 2 +- INSTALL | 1 + Makefile | 1 + Makefile.inc.in | 5 +- cmds-filesystem.c | 16 ++++--- cmds-inspect-dump-super.c | 2 +- cmds-restore.c | 50 ++++++++++++++++++++ configure.ac | 41 ++++++++++++----- ctree.h | 15 ++---- fsfeatures.h | 2 +- print-tree.c | 3 ++ .../022-zstd-compression/compress.raw.xz | Bin 0 -> 18256 bytes tests/misc-tests/022-zstd-compression/test.sh | 51 +++++++++++++++++++++ 16 files changed, 172 insertions(+), 36 deletions(-) create mode 100644 tests/misc-tests/022-zstd-compression/compress.raw.xz create mode 100755 tests/misc-tests/022-zstd-compression/test.sh
.travis.yml +9 -0
@@ -59,6 +59,15 @@ before_install: sudo make install; cd ../.. " + - "mkdir tmp-zstd; + cd tmp-zstd; + wget https://github.com/facebook/zstd/archive/v1.3.1.tar.gz; + tar xf v1.3.1.tar.gz; + cd zstd-1.3.1; + make; + sudo make install PREFIX=/usr; + cd ../.. + " - "./autogen.sh && ./configure --disable-documentation && make" addons:
Documentation/btrfs-filesystem.asciidoc +1 -1
@@ -112,7 +112,7 @@ KiB, MiB, GiB, TiB, PiB, or EiB, respectively (case does not matter). be verbose, print file names as they're submitted for defragmentation -c[<algo>]:::: compress file contents while defragmenting. Optional argument selects the compression -algorithm, 'zlib' (default) or 'lzo'. Currently it's not possible to select no +algorithm, 'zlib' (default), 'lzo' or 'zstd'. Currently it's not possible to select no compression. See also section 'EXAMPLES'. -r:::: defragment files recursively in given directories
Documentation/btrfs-man5.asciidoc +7 -1
@@ -118,7 +118,7 @@ but a warning is printed if it's more than 300 seconds (5 minutes). (default: off) + Control BTRFS file data compression. Type may be specified as 'zlib', -'lzo' or 'no' (for no compression, used for remounting). If no type +'lzo', 'zstd' or 'no' (for no compression, used for remounting). If no type is specified, 'zlib' is used. If 'compress-force' is specified, the compression will allways be attempted, but the data may end up uncompressed if the compression would make them larger.
@@ -472,6 +472,12 @@ page size the 'lzo' compression has been used on the filesystem, either as a mount option or via *btrfs filesystem defrag*. +*compress_zstd*:: +(since: 4.14) ++ +the 'zstd' compression has been used on the filesystem, either as a mount option +or via *btrfs filesystem defrag*. + *default_subvol*:: (since: 2.6.34) +
Documentation/btrfs-property.asciidoc +1 -1
@@ -43,7 +43,7 @@ read-only flag of subvolume: true or false label:::: label of device compression:::: -compression setting for an inode: lzo, zlib, or "" (empty string) +compression setting for an inode: lzo, zlib, zstd, or "" (empty string) *list* [-t <type>] <object>:: Lists available properties with their descriptions for the given object.
INSTALL +1 -0
@@ -7,6 +7,7 @@ The Btrfs utility programs require the following libraries/tools to build: - libblkid - block device id library - liblzo2 - LZO data compression library - zlib - ZLIB data compression library +- libzstd - ZSTD data compression library version >= 1.0.0 (optional) For the btrfs-convert utility:
Makefile +1 -0
@@ -209,6 +209,7 @@ btrfs_fragments_libs = -lgd -lpng -ljpeg -lfreetype btrfs_debug_tree_objects = cmds-inspect-dump-tree.o btrfs_show_super_objects = cmds-inspect-dump-super.o btrfs_calc_size_objects = cmds-inspect-tree-stats.o +cmds_restore_cflags = -DBTRFSRESTORE_ZSTD=$(BTRFSRESTORE_ZSTD) # collect values of the variables above standalone_deps = $(foreach dep,$(patsubst %,%_objects,$(subst -,_,$(filter btrfs-%, $(progs)))),$($(dep)))
Makefile.inc.in +3 -2
@@ -13,14 +13,15 @@ DISABLE_DOCUMENTATION = @DISABLE_DOCUMENTATION@ DISABLE_BTRFSCONVERT = @DISABLE_BTRFSCONVERT@ BTRFSCONVERT_EXT2 = @BTRFSCONVERT_EXT2@ BTRFSCONVERT_REISERFS = @BTRFSCONVERT_REISERFS@ +BTRFSRESTORE_ZSTD = @BTRFSRESTORE_ZSTD@ SUBST_CFLAGS = @CFLAGS@ SUBST_LDFLAGS = @LDFLAGS@ LIBS_BASE = @UUID_LIBS@ @BLKID_LIBS@ -L. -pthread -LIBS_COMP = @ZLIB_LIBS@ @LZO2_LIBS@ +LIBS_COMP = @ZLIB_LIBS@ @LZO2_LIBS@ @ZSTD_LIBS@ STATIC_LIBS_BASE = @UUID_LIBS_STATIC@ @BLKID_LIBS_STATIC@ -L. -pthread -STATIC_LIBS_COMP = @ZLIB_LIBS_STATIC@ @LZO2_LIBS_STATIC@ +STATIC_LIBS_COMP = @ZLIB_LIBS_STATIC@ @LZO2_LIBS_STATIC@ @ZSTD_LIBS_STATIC@ prefix ?= @prefix@ exec_prefix = @exec_prefix@
cmds-filesystem.c +9 -7
@@ -952,6 +952,8 @@ static int parse_compress_type(char *s)return BTRFS_COMPRESS_ZLIB ; else if (strcmp(optarg, "lzo") == 0) return BTRFS_COMPRESS_LZO; + else if (strcmp(optarg, "zstd") == 0) + return BTRFS_COMPRESS_ZSTD; else {error ("unknown compression type %s" , s);exit (1 );
@@ -962,13 +964,13 @@ static const char * const cmd_filesystem_defrag_usage[] = {"btrfs filesystem defragment [options] <file>|<dir> [<file>|<dir>...]" ,"Defragment a file or a directory" ,"" , -"-v be verbose" , -"-r defragment files recursively" , -"-c[zlib,lzo] compress the file while defragmenting" , -"-f flush data to disk immediately after defragmenting" , -"-s start defragment only from byte onward" , -"-l len defragment only up to len bytes" , -"-t size target extent size hint (default: 32M)" , +"-v be verbose" , +"-r defragment files recursively" , +"-c[zlib,lzo,zstd] compress the file while defragmenting" , +"-f flush data to disk immediately after defragmenting" , +"-s start defragment only from byte onward" , +"-l len defragment only up to len bytes" , +"-t size target extent size hint (default: 32M)" ,NULL };
cmds-inspect-dump-super.c +1 -1
@@ -223,7 +223,7 @@ static struct readable_flag_entry incompat_flags_array[] = {DEF_INCOMPAT_FLAG_ENTRY (DEFAULT_SUBVOL ),DEF_INCOMPAT_FLAG_ENTRY (MIXED_GROUPS ),DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_LZO ), -DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_LZOv2), +DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_ZSTD ),DEF_INCOMPAT_FLAG_ENTRY (BIG_METADATA ),DEF_INCOMPAT_FLAG_ENTRY (EXTENDED_IREF ), DEF_INCOMPAT_FLAG_ENTRY(RAID56),
cmds-restore.c +50 -0
68 lines not on this page · raw message
configure.ac +29 -12
56 lines not on this page · raw message
ctree.h +5 -10
@@ -482,14 +482,7 @@ struct btrfs_super_block {#define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL (1ULL << 1)#define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS (1ULL << 2)#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO (1ULL << 3) - -/* -* some patches floated around with a second compression method -* lets save that incompat here for when they do get in -* Note we don't actually support it, we're just reserving the -* number -*/ -#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZOv2 (1ULL << 4) +#define BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD (1ULL << 4)/* * older kernels tried to do bigger metadata blocks, but the
@@ -514,6 +507,7 @@ struct btrfs_super_block { (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF | \ BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL | \ BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO | \ + BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD | \ BTRFS_FEATURE_INCOMPAT_BIG_METADATA | \ BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \ BTRFS_FEATURE_INCOMPAT_RAID56 | \
@@ -675,8 +669,9 @@ typedef enum {BTRFS_COMPRESS_NONE =0 ,BTRFS_COMPRESS_ZLIB =1 ,BTRFS_COMPRESS_LZO =2 , -BTRFS_COMPRESS_TYPES =2 , -BTRFS_COMPRESS_LAST =3 , +BTRFS_COMPRESS_ZSTD =3 , +BTRFS_COMPRESS_TYPES =3 , +BTRFS_COMPRESS_LAST =4 , } btrfs_compression_type;/* we don't understand any encryption methods right now */
fsfeatures.h +1 -1
@@ -31,7 +31,7 @@ (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF \ |BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL \ |BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO \ - | BTRFS_FEATURE_INCOMPAT_COMPRESS_LZOv2 \ + |BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD \ |BTRFS_FEATURE_INCOMPAT_BIG_METADATA \ |BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF \ | BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA \
print-tree.c +3 -0
@@ -316,6 +316,9 @@ static void compress_type_to_str(u8 compress_type, char *ret)case BTRFS_COMPRESS_LZO :strcpy (ret,"lzo" );break ; +case BTRFS_COMPRESS_ZSTD : +strcpy (ret,"zstd" ); +break ;default :sprintf (ret,"UNKNOWN.%d" , compress_type); }
tests/misc-tests/022-zstd-compression/compress.raw.xz new file binary +0 -0
tests/misc-tests/022-zstd-compression/test.sh new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.9.5
Quoted 11 lines · Nick Terrell
Adds zstd support to the btrfs program. An optional dependency on libzstd
Quoted 1 line
= 1.0.0 is added. Autoconf accepts
--enable-zstd' or--disable-zstd' anddefaults to detecting if libzstd is present using `pkg-config'. Adds tests for the new features based on a prebuilt btrfs image with a zstd compressed file.
The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI with the new tests. The prebuilt binary is available there.
I haven't updated Android.mk.
Quoted 8 lines
[1] https://github.com/terrelln/btrfs-progs/tree/devel
Signed-off-by Nick Terrell <[email protected]>---
v1 -> v2:
- Allow zstd support to be disabled, defaulting to auto detection
- Print if zstd is enabled/disabled when `./configure' finishes
- Add zstd test to patch with prebuilt image`
Applied, with some changes. I've split the tests to own commit and cleaned up test.sh so it follows the coding patterns (dd instead of head, grep -q, minor things). The run_check should be used if possible, but dumping the 200k of zeros to log via run_check_stdout is OTOH not recomended. But, it's merged.