btrfs-progs: Add zstd support
Subsystems (3)
PATCH 1/1 btrfs-progs: Add zstd support
Adds zstd support to the btrfs program, and a dependency on libzstd >= 1.0.0.
The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI. I tested each command that is effected in my test script [2].
I haven't updated Android.mk since I have no way to test it, and am not certain if it is used.
[1] https://github.com/terrelln/btrfs-progs/tree/devel
[2] https://gist.github.com/terrelln/4136a369c5d10092956781433eed0a23
--- .travis.yml | 9 +++++++ Documentation/btrfs-filesystem.asciidoc | 2 +- Documentation/btrfs-man5.asciidoc | 8 +++++- Documentation/btrfs-property.asciidoc | 2 +- INSTALL | 1 + Makefile.inc.in | 4 +-- cmds-filesystem.c | 16 ++++++------ cmds-inspect-dump-super.c | 2 +- cmds-restore.c | 44 +++++++++++++++++++++++++++++++++ configure.ac | 4 ++- ctree.h | 15 ++++------- fsfeatures.h | 2 +- print-tree.c | 3 +++ 13 files changed, 87 insertions(+), 25 deletions(-)
.travis.yml +9 -0
@@ -59,6 +59,15 @@ before_install: sudo make install; cd ../.. " + - "mkdir tmp-zstd; + cd tmp-zstd; + wget https://github.com/facebook/zstd/archive/v1.3.1.tar.gz; + tar xf v1.3.1.tar.gz; + cd zstd-1.3.1; + make; + sudo make install PREFIX=/usr; + cd ../.. + " - "./autogen.sh && ./configure --disable-documentation && make" addons:
Documentation/btrfs-filesystem.asciidoc +1 -1
@@ -112,7 +112,7 @@ KiB, MiB, GiB, TiB, PiB, or EiB, respectively (case does not matter). be verbose, print file names as they're submitted for defragmentation -c[<algo>]:::: compress file contents while defragmenting. Optional argument selects the compression -algorithm, 'zlib' (default) or 'lzo'. Currently it's not possible to select no +algorithm, 'zlib' (default), 'lzo' or 'zstd'. Currently it's not possible to select no compression. See also section 'EXAMPLES'. -r:::: defragment files recursively in given directories
Documentation/btrfs-man5.asciidoc +7 -1
@@ -118,7 +118,7 @@ but a warning is printed if it's more than 300 seconds (5 minutes). (default: off) + Control BTRFS file data compression. Type may be specified as 'zlib', -'lzo' or 'no' (for no compression, used for remounting). If no type +'lzo', 'zstd' or 'no' (for no compression, used for remounting). If no type is specified, 'zlib' is used. If 'compress-force' is specified, the compression will allways be attempted, but the data may end up uncompressed if the compression would make them larger.
@@ -472,6 +472,12 @@ page size the 'lzo' compression has been used on the filesystem, either as a mount option or via *btrfs filesystem defrag*. +*compress_zstd*:: +(since: 4.14) ++ +the 'zstd' compression has been used on the filesystem, either as a mount option +or via *btrfs filesystem defrag*. + *default_subvol*:: (since: 2.6.34) +
Documentation/btrfs-property.asciidoc +1 -1
@@ -43,7 +43,7 @@ read-only flag of subvolume: true or false label:::: label of device compression:::: -compression setting for an inode: lzo, zlib, or "" (empty string) +compression setting for an inode: lzo, zlib, zstd, or "" (empty string) *list* [-t <type>] <object>:: Lists available properties with their descriptions for the given object.
INSTALL +1 -0
@@ -7,6 +7,7 @@ The Btrfs utility programs require the following libraries/tools to build: - libblkid - block device id library - liblzo2 - LZO data compression library - zlib - ZLIB data compression library +- libzstd - ZSTD data compression library version >= 1.0.0 For the btrfs-convert utility:
Makefile.inc.in +2 -2
@@ -18,9 +18,9 @@ SUBST_CFLAGS = @CFLAGS@ SUBST_LDFLAGS = @LDFLAGS@ LIBS_BASE = @UUID_LIBS@ @BLKID_LIBS@ -L. -pthread -LIBS_COMP = @ZLIB_LIBS@ @LZO2_LIBS@ +LIBS_COMP = @ZLIB_LIBS@ @LZO2_LIBS@ @ZSTD_LIBS@ STATIC_LIBS_BASE = @UUID_LIBS_STATIC@ @BLKID_LIBS_STATIC@ -L. -pthread -STATIC_LIBS_COMP = @ZLIB_LIBS_STATIC@ @LZO2_LIBS_STATIC@ +STATIC_LIBS_COMP = @ZLIB_LIBS_STATIC@ @LZO2_LIBS_STATIC@ @ZSTD_LIBS_STATIC@ prefix ?= @prefix@ exec_prefix = @exec_prefix@
cmds-filesystem.c +9 -7
@@ -952,6 +952,8 @@ static int parse_compress_type(char *s)return BTRFS_COMPRESS_ZLIB ; else if (strcmp(optarg, "lzo") == 0) return BTRFS_COMPRESS_LZO; + else if (strcmp(optarg, "zstd") == 0) + return BTRFS_COMPRESS_ZSTD; else {error ("unknown compression type %s" , s);exit (1 );
@@ -962,13 +964,13 @@ static const char * const cmd_filesystem_defrag_usage[] = {"btrfs filesystem defragment [options] <file>|<dir> [<file>|<dir>...]" ,"Defragment a file or a directory" ,"" , -"-v be verbose" , -"-r defragment files recursively" , -"-c[zlib,lzo] compress the file while defragmenting" , -"-f flush data to disk immediately after defragmenting" , -"-s start defragment only from byte onward" , -"-l len defragment only up to len bytes" , -"-t size target extent size hint (default: 32M)" , +"-v be verbose" , +"-r defragment files recursively" , +"-c[zlib,lzo,zstd] compress the file while defragmenting" , +"-f flush data to disk immediately after defragmenting" , +"-s start defragment only from byte onward" , +"-l len defragment only up to len bytes" , +"-t size target extent size hint (default: 32M)" ,NULL };
cmds-inspect-dump-super.c +1 -1
@@ -223,7 +223,7 @@ static struct readable_flag_entry incompat_flags_array[] = {DEF_INCOMPAT_FLAG_ENTRY (DEFAULT_SUBVOL ),DEF_INCOMPAT_FLAG_ENTRY (MIXED_GROUPS ),DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_LZO ), -DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_LZOv2), +DEF_INCOMPAT_FLAG_ENTRY (COMPRESS_ZSTD ),DEF_INCOMPAT_FLAG_ENTRY (BIG_METADATA ),DEF_INCOMPAT_FLAG_ENTRY (EXTENDED_IREF ), DEF_INCOMPAT_FLAG_ENTRY(RAID56),
cmds-restore.c +44 -0
65 lines not on this page · raw message
configure.ac +3 -1
@@ -182,6 +182,9 @@ PKG_STATIC(UUID_LIBS_STATIC, [uuid]) PKG_CHECK_MODULES(ZLIB, [zlib]) PKG_STATIC(ZLIB_LIBS_STATIC, [zlib]) +PKG_CHECK_MODULES(ZSTD, [libzstd]) +PKG_STATIC(ZSTD_LIBS_STATIC, [libzstd]) + # udev v190 introduced the btrfs builtin and a udev rule to use it. # Our udev rule gives us the friendly dm names but isn't required (or valid) # on earlier releases.
@@ -238,4 +241,3 @@ AC_MSG_RESULT([ Type 'make' to compile. ]) -
ctree.h +5 -10
@@ -482,14 +482,7 @@ struct btrfs_super_block {#define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL (1ULL << 1)#define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS (1ULL << 2)#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO (1ULL << 3) - -/* -* some patches floated around with a second compression method -* lets save that incompat here for when they do get in -* Note we don't actually support it, we're just reserving the -* number -*/ -#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZOv2 (1ULL << 4) +#define BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD (1ULL << 4)/* * older kernels tried to do bigger metadata blocks, but the
@@ -514,6 +507,7 @@ struct btrfs_super_block { (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF | \ BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL | \ BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO | \ + BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD | \ BTRFS_FEATURE_INCOMPAT_BIG_METADATA | \ BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \ BTRFS_FEATURE_INCOMPAT_RAID56 | \
@@ -675,8 +669,9 @@ typedef enum {BTRFS_COMPRESS_NONE =0 ,BTRFS_COMPRESS_ZLIB =1 ,BTRFS_COMPRESS_LZO =2 , -BTRFS_COMPRESS_TYPES =2 , -BTRFS_COMPRESS_LAST =3 , +BTRFS_COMPRESS_ZSTD =3 , +BTRFS_COMPRESS_TYPES =3 , +BTRFS_COMPRESS_LAST =4 , } btrfs_compression_type;/* we don't understand any encryption methods right now */
fsfeatures.h +1 -1
@@ -31,7 +31,7 @@ (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF \ |BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL \ |BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO \ - | BTRFS_FEATURE_INCOMPAT_COMPRESS_LZOv2 \ + |BTRFS_FEATURE_INCOMPAT_COMPRESS_ZSTD \ |BTRFS_FEATURE_INCOMPAT_BIG_METADATA \ |BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF \ | BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA \
print-tree.c +3 -0
@@ -316,6 +316,9 @@ static void compress_type_to_str(u8 compress_type, char *ret)case BTRFS_COMPRESS_LZO :strcpy (ret,"lzo" );break ; +case BTRFS_COMPRESS_ZSTD : +strcpy (ret,"zstd" ); +break ;default :sprintf (ret,"UNKNOWN.%d" , compress_type); }
--
2.9.5
Quoted 2 lines · Nick Terrell
Adds zstd support to the btrfs program, and a dependency on libzstd >= 1.0.0.
I'm afraid we'll have to make the build optional for now, as the distros may not provide it and I'd like to give at least some heads-up first. My idea is:
- check if zstd library is found, use it
- otherwise warn that's it's going to be on by default in near future
'btrfs restore' would need to be enhanced to dump what's built in (in a similar way convert now prints the supported filesystem), and possibly warn that there are data compressed by zstd but there's no decrompression support.
Quoted 2 lines
The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI. I tested each command that is effected in my test script [2].
The test requires zstd kernel support, which may be not available for some time in the testing environments. But as restore is userspace-only, we can create and use various images with zstd compressed data.
Quoted 2 lines
I haven't updated Android.mk since I have no way to test it, and am not certain if it is used.
The anodroid build verification will work eventually through travis and I'm aware it could be out of sync, you don't need to care about that right now.
Quoted 10 lines · On 9/4/17, 11:03 AM, "[email protected] on behalf of David…
Quoted 2 lines · Nick Terrell
Adds zstd support to the btrfs program, and a dependency on libzstd >= 1.0.0.
I'm afraid we'll have to make the build optional for now, as the distros may not provide it and I'd like to give at least some heads-up first. My idea is:
- check if zstd library is found, use it
- otherwise warn that's it's going to be on by default in near future
I can see a few options, what do you want to see?
1. Require zstd for building `btrfs' but statically link it so users don't need to have zstd installed, unless they install from source.
2. Require zstd to build btrfs', but dynamically link and use dlopen' and `dlsym' to access the decompression function (and check the library version).
3. Make zstd optional in the build process, and completely disable it if unavailable. Otherwise dynamically link and do the same as (2).
4. Same as (3) but statically link if it is available.
I'm thinking that (3) is the best option for consistency with the other libraries, at the expense of making the user install zstd if they want to use it. We can put a nice warning message in `btrfs restore' that says you need to install zstd.
Quoted 4 lines
'btrfs restore' would need to be enhanced to dump what's built in (in a similar way convert now prints the supported filesystem), and possibly warn that there are data compressed by zstd but there's no decrompression support.
Sounds good.
Quoted 6 lines
Quoted 2 lines
The patch is also available in my fork of btrfs-progs [1], which passes Travis-CI. I tested each command that is effected in my test script [2].
The test requires zstd kernel support, which may be not available for some time in the testing environments. But as restore is userspace-only, we can create and use various images with zstd compressed data.
I'll modify my tests and add them to the test suite.
Quoted 6 lines
Quoted 2 lines
I haven't updated Android.mk since I have no way to test it, and am not certain if it is used.
The anodroid build verification will work eventually through travis and I'm aware it could be out of sync, you don't need to care about that right now.
���{.n�+�������+%�����ݶ��w��{.n�+����{���k~���^n�r���z���h����&���z��z�ޗ�+��+zf���h���~����i��������z_��j:+v���)ߣ�m