Add the snappy-c compressor to lib

PATCH In review Andi Kleen 3 participants 6 messages Oct 20, 2011 – Oct 30, 2011

Subsystems (4)

Library code Memory technology devices (MTD) Btrfs file system Crypto API

lore

Share & tools

lore

Andi Kleen

author

2011-10-20 23:38 UTC #0 lore raw

PATCH 1/3 Add the snappy-c compressor to lib

From: Andi Kleen <[email protected]>

This is a C port of the google snappy compressor. It has roughly comparable compression to LZO, but is significantly faster on many file types. For example it beats all other compressors on already compressed data.

I ported the original C++ code over to C and did some changes to make it better fit into the kernel. It preallocates the worst case memory consumption now. While the code being larger than lzo it is still reasonable (about 5K on x86).

Decompression needs very little memory, Compression currently 192K on 64bit and 128K on 32bit. For comparison LZO compression needs 128K on 64bit and 64K on 32bit.

[This could be lowered significantly by not preallocating for most use cases, typically the footprint is much lower. The original C++ version only allocated most of this when (rarely) needed, but this is more problematic in the kernel]

There are some minor divergences from the Linux coding standards:
in particular I kept the C++/C99 style mixed statement/declarations. This was mainly to not diverge too much from the reference C++ source, so that improvements from there can be easily ported. There are some other left overs from the google style, but very little now.

checkpatch.pl has some complaints, but they are either caused by the above or checkpatch.pl bugs (it misparsed #define foo do {} while(0))

Performance:

The compressor performs best on 64bit-LE systems, but is also quite good on 32bit. I haven't tested BE, but I don't expect that to add a lot of overhead.

Here is some performance data (32bit, Nehalem):
c/b = cycles/byte; lower numbers are better.

x86-64 executable: (compression minimally slower than qlz, but much better at decompression, lzo is left in the dust):

snappy: emacs-gtk: 11007968 b: ratio 0.38: comp 8.13 uncomp 2.65 c/b
lzo : emacs-gtk: 11007968 b: ratio 0.33: comp 12.74 uncomp 4.70 c/b
zlib1 : emacs-gtk: 11007968 b: ratio 0.27: comp 49.96 uncomp 13.14 c/b zlib3 : emacs-gtk: 11007968 b: ratio 0.26: comp 64.17 uncomp 12.33 c/b
lzf : emacs-gtk: 11007968 b: ratio 0.37: comp 9.85 uncomp 4.33 c/b
qlz : emacs-gtk: 11007968 b: ratio 0.34: comp 7.51 uncomp 6.28 c/b
fastlz: emacs-gtk: 11007968 b: ratio 0.37: comp 10.73 uncomp 4.97 c/b

Compressed data (beats everything else):

snappy: udev-151.tar.gz: 634842 b: ratio 1.00: comp 0.99 uncomp 0.33 c/b
lzo : udev-151.tar.gz: 634842 b: ratio 1.00: comp 41.44 uncomp 0.66 c/b
zlib1 : udev-151.tar.gz: 634842 b: ratio 1.00: comp 116.99 uncomp 3.94 c/b zlib3 : udev-151.tar.gz: 634842 b: ratio 1.00: comp 117.68 uncomp 3.94 c/b
lzf : udev-151.tar.gz: 634842 b: ratio 1.03: comp 16.32 uncomp 1.14 c/b
qlz : udev-151.tar.gz: 634842 b: ratio 1.00: comp 10.42 uncomp 0.42 c/b
fastlz: udev-151.tar.gz: 634842 b: ratio 1.03: comp 19.35 uncomp 2.07 c/b

Text file (compression somewhat slower than qlz, but decompression much better, lzo is much worse):

snappy: manual.txt: 445343 b: ratio 0.47: comp 12.01 uncomp 3.12 c/b
lzo : manual.txt: 445343 b: ratio 0.44: comp 16.32 uncomp 7.53 c/b
zlib1 : manual.txt: 445343 b: ratio 0.35: comp 56.37 uncomp 15.59 c/b zlib3 : manual.txt: 445343 b: ratio 0.31: comp 73.45 uncomp 13.99 c/b
lzf : manual.txt: 445343 b: ratio 0.46: comp 13.43 uncomp 5.47 c/b
qlz : manual.txt: 445343 b: ratio 0.44: comp 9.16 uncomp 8.19 c/b
fastlz: manual.txt: 445343 b: ratio 0.46: comp 14.22 uncomp 7.28 c/b

As you can see snappy is a good all-around compressor.

On 64bit the compression is even faster and beats everything else easily:

snappy: emacs-gtk: 11007968 b: ratio 0.38: comp 4.90 uncomp 2.65 c/b
lzo : emacs-gtk: 11007968 b: ratio 0.33: comp 11.24 uncomp 4.46 c/b
zlib1 : emacs-gtk: 11007968 b: ratio 0.27: comp 41.67 uncomp 11.13 c/b zlib3 : emacs-gtk: 11007968 b: ratio 0.26: comp 51.80 uncomp 10.54 c/b
lzf : emacs-gtk: 11007968 b: ratio 0.37: comp 8.79 uncomp 4.05 c/b
qlz : emacs-gtk: 11007968 b: ratio 0.34: comp 5.44 uncomp 5.46 c/b
fastlz: emacs-gtk: 11007968 b: ratio 0.37: comp 9.91 uncomp 4.77 c/b

On 64bit it's now nearly as fast as qlz on the text file too:

snappy: manual.txt: 445343 b: ratio 0.47: comp 7.79 uncomp 3.47 c/b
lzo : manual.txt: 445343 b: ratio 0.44: comp 15.46 uncomp 7.27 c/b
zlib1 : manual.txt: 445343 b: ratio 0.35: comp 45.79 uncomp 12.78 c/b zlib3 : manual.txt: 445343 b: ratio 0.31: comp 60.52 uncomp 11.72 c/b
lzf : manual.txt: 445343 b: ratio 0.46: comp 12.62 uncomp 5.30 c/b
qlz : manual.txt: 445343 b: ratio 0.44: comp 6.81 uncomp 7.65 c/b
fastlz: manual.txt: 445343 b: ratio 0.46: comp 13.75 uncomp 6.52 c/b

Overall it's a good alternative to lzo, with the only drawback being the somewhat higher memory use. The memory use can be fixed for most cases (e.g. some of the current buffers are only used for SG), but isn't yet in this version.

Open: it's pretty easy to add scatter-gather support since input/output is quite abstracted. This would benefit some users who could avoid temporary buffers.

Signed-off-by Andi Kleen <[email protected]>

---
 include/linux/snappy.h |   26 +
 lib/Kconfig            |    8 +
 lib/Makefile           |    4 +
 lib/snappy.c           | 1294 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 1332 insertions(+), 0 deletions(-)
 create mode 100644 include/linux/snappy.h
 create mode 100644 lib/snappy.c

include/linux/snappy.h new file +26 -0

new file mode 100644
index 0000000..4119803
@@ -0,0 +1,26 @@
+#ifndef _LINUX_SNAPPY_H
+#define _LINUX_SNAPPY_H 1
+
+#include <linux/types.h>
+
+/* Only needed for compression. This preallocates the worst case */
+struct snappy_env {
+	u16 *hash_table;
+	void *scratch;
+	void *scratch_output;
+};
+
+int snappy_init_env(struct snappy_env *env);
+void snappy_free_env(struct snappy_env *env);
+bool snappy_uncompress(const char *compressed, size_t n, char *uncompressed);
+int snappy_compress(struct snappy_env *env,
+		    const char *input,
+		    size_t input_length,
+		    char *compressed,
+		    size_t *compressed_length);
+bool snappy_uncompressed_length(const char *buf, size_t len, size_t *result);
+size_t snappy_max_compressed_length(size_t source_len);
+
+
+
+#endif

lib/Kconfig +8 -0

index 6c695ff..416dc08 100644
@@ -279,4 +279,12 @@ config CORDIC
 config LLIST
 	bool
 
+config SNAPPY
+       tristate "Snappy compressor"
+       help
+          Add the snappy compressor. This is a reasonable compressor that
+	  compresses and decompresses extremly fast. In general it's a better
+	  replacement for LZO.
+
+
 endmenu

lib/Makefile +4 -0

index 3f5bc6d..e38e580 100644
@@ -117,6 +117,10 @@ obj-$(CONFIG_CORDIC) += cordic.o
 
 obj-$(CONFIG_LLIST) += llist.o
 
+CFLAGS_snappy.o += $(call cc-disable-warning, declaration-after-statement) \
+		   -DNDEBUG=1
+obj-$(CONFIG_SNAPPY) += snappy.o
+
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h

lib/snappy.c new file +1294 -0

1,294 lines not on this page · raw message

Signature · 1 line

1.7.4.4

Andi Kleen

author

2011-10-20 23:39 UTC #1 lore raw

PATCH 2/3 BTRFS: Add snappy support

From: Andi Kleen <[email protected]>

Add support in btrfs for snappy compression.

This is based on the lzo code with minor modifications. The btrfs glue code could be significantly improved over LZO by exploiting some snappy features, but hasn't so far.

Open: implement scatter-gather support and get rid of the temporary buffers.

Some performance numbers (thanks to Jacob Sowles for running them)

bonnie++, Core i7-64bit

    block output rewrite  block input random
       K/sec     K/sec    K/sec       K/sec
None   100%      100%     100%        100%
zlib    +6.3%    +5.4%    +6.6%       +11.6%
lzo    +11.5%    +4.6%    +12.4%      +6.7%
snappy +19.3%    +28.1%   +32.6%      +9.3%

Snappy does extremly well on the 64bit architecture, outperforming everything else, sometimes with a healthy margin.

bonnie++, Atom-32bit

block output rewrite block input random

K/sec K/sec K/sec K/sec

None   100%      100%     100%        100%
zlib   -43.1%    -24.2%   -19.0%      +12.0%
lzo    +0.8%     +2.6%    +6.8%       +14.8%
snappy +19.5%    +16.2%   +24.0%      +15.7%

zlib does very poorly on Atom, actually degrading performance. snappy is generally faster or similar to LZO. The difference is not as big as on the 64bit CPU though, but still visible.

bonnie++, files, Core-i7-64bit

sequential create delete random create delete
files/sec

None       100%      100%          100%   100%
zlib       +8.3%     +10.5%        +10.3%  +1.4%
lzo        +3.8%     +3.3%         +5.4%   -3.89%
snappy     +23.7%    +37.2%        +21.8%  +23.8%

bonnie++, files, Atom-32bit

sequential create delete random create delete
files/sec

None       100%      100%          100%   100%
zlib       +3.0%     +7.9%         +5.2%  +5.1%
lzo        +8.2%     +5.9%         +4.8%  +4.6%
snappy     +3.1%     +8.5%         +5.7%  +1.3%

Creation/Deletion on Atom is a case where snappy loses to LZO, however the loss is small. On 64bit Core it's a win.

I should add that these benchmarks mainly use 0 filled IO, however FFSB was also quickly tested with more random data and the differences were similar. See also the micro benchmarks in the algorithm description for the behaviour with different data types.

FFSB, 4 threads, stair case data pattern, Reads MB/s

Core i7-64bit                 Atom-32bit
       MB/s                   MB/s
None   100%                   100%
zlib   +8.0%                  +4.2%
lzo    +9.3%                  +4.8%
snappy +12.4%                 +7.9%

In general snappy is a better replacement for LZO, especially on 64bit, but even on 32bit.

Cc [email protected]

Signed-off-by Andi Kleen <[email protected]>

---
 fs/btrfs/Kconfig       |    1 +
 fs/btrfs/Makefile      |    3 +-
 fs/btrfs/compression.c |    1 +
 fs/btrfs/compression.h |    1 +
 fs/btrfs/ctree.h       |    9 +-
 fs/btrfs/disk-io.c     |    2 +
 fs/btrfs/ioctl.c       |    4 +
 fs/btrfs/snappy.c      |  435 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/super.c       |    9 +-
 lib/Makefile           |    1 +
 10 files changed, 461 insertions(+), 5 deletions(-)
 create mode 100644 fs/btrfs/snappy.c

fs/btrfs/Kconfig +1 -0

index ecb9fd3..d55df9c 100644
@@ -6,6 +6,7 @@ config BTRFS_FS
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
 	select LZO_DECOMPRESS
+	select SNAPPY
 	help
 	  Btrfs is a new filesystem with extents, writable snapshotting,
 	  support for multiple devices and many more features.

fs/btrfs/Makefile +2 -1

index 40e6ac0..7cd86e7 100644
@@ -7,6 +7,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
 	   export.o tree-log.o free-space-cache.o zlib.o lzo.o \
-	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o
+	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
+	   snappy.o
 
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o

fs/btrfs/compression.c +1 -0

index 8ec5d86..b171858 100644
@@ -729,6 +729,7 @@ static wait_queue_head_t comp_workspace_wait[BTRFS_COMPRESS_TYPES];
 struct btrfs_compress_op *btrfs_compress_op[] = {
 	&btrfs_zlib_compress,
 	&btrfs_lzo_compress,
+	&btrfs_snappy_compress,
 };
 
 int __init btrfs_init_compress(void)

fs/btrfs/compression.h +1 -0

index a12059f..971a425 100644
@@ -79,5 +79,6 @@ struct btrfs_compress_op {
 
 extern struct btrfs_compress_op btrfs_zlib_compress;
 extern struct btrfs_compress_op btrfs_lzo_compress;
+extern struct btrfs_compress_op btrfs_snappy_compress;
 
 #endif

fs/btrfs/ctree.h +6 -3

index 03912c5..7ebdae3 100644
@@ -415,6 +415,7 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
 #define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS	(1ULL << 2)
 #define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO	(1ULL << 3)
+#define BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY	(1ULL << 4)
 
 #define BTRFS_FEATURE_COMPAT_SUPP		0ULL
 #define BTRFS_FEATURE_COMPAT_RO_SUPP		0ULL

@@ -422,7 +423,8 @@ struct btrfs_super_block {
 	(BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF |		\
 	 BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL |	\
 	 BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS |		\
-	 BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO)
+	 BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO |		\
+	 BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY)
 
 /*
  * A leaf is full of items. offset and size tell us where to find

@@ -578,8 +580,9 @@ enum btrfs_compression_type {
 	BTRFS_COMPRESS_NONE  = 0,
 	BTRFS_COMPRESS_ZLIB  = 1,
 	BTRFS_COMPRESS_LZO   = 2,
-	BTRFS_COMPRESS_TYPES = 2,
-	BTRFS_COMPRESS_LAST  = 3,
+	BTRFS_COMPRESS_SNAPPY = 3,
+	BTRFS_COMPRESS_TYPES = 3,
+	BTRFS_COMPRESS_LAST  = 4,
 };
 
 struct btrfs_inode_item {

fs/btrfs/disk-io.c +2 -0

index 07b3ac6..4c04228 100644
@@ -1808,6 +1808,8 @@ struct btrfs_root *open_ctree(struct super_block *sb,
 	features |= BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF;
 	if (tree_root->fs_info->compress_type & BTRFS_COMPRESS_LZO)
 		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
+	if (tree_root->fs_info->compress_type & BTRFS_COMPRESS_SNAPPY)
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY;
 	btrfs_set_super_incompat_flags(disk_super, features);
 
 	features = btrfs_super_compat_ro_flags(disk_super) &

fs/btrfs/ioctl.c +4 -0

index 538f65a..3f15175 100644
@@ -1133,6 +1133,10 @@ int btrfs_defrag_file(struct inode *inode, struct file *file,
 		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
 		btrfs_set_super_incompat_flags(disk_super, features);
 	}
+	if (range->compress_type == BTRFS_COMPRESS_SNAPPY) {
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY;
+		btrfs_set_super_incompat_flags(disk_super, features);
+	}
 
 	if (!file)
 		kfree(ra);

fs/btrfs/snappy.c new file +435 -0

435 lines not on this page · raw message

fs/btrfs/super.c +8 -1

index 15634d4..d3a2d10 100644
@@ -267,6 +267,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 			} else if (strcmp(args[0].from, "lzo") == 0) {
 				compress_type = "lzo";
 				info->compress_type = BTRFS_COMPRESS_LZO;
+			} else if (strcmp(args[0].from, "snappy") == 0) {
+				compress_type = "snappy";
+				info->compress_type = BTRFS_COMPRESS_SNAPPY;
 			} else {
 				ret = -EINVAL;
 				goto out;

@@ -696,8 +699,12 @@ static int btrfs_show_options(struct seq_file *seq, struct vfsmount *vfs)
 	if (btrfs_test_opt(root, COMPRESS)) {
 		if (info->compress_type == BTRFS_COMPRESS_ZLIB)
 			compress_type = "zlib";
-		else
+		else if (info->compress_type == BTRFS_COMPRESS_SNAPPY)
+			compress_type = "snappy";
+		else if (info->compress_type == BTRFS_COMPRESS_LZO)
 			compress_type = "lzo";
+		else
+			compress_type = "?";
 		if (btrfs_test_opt(root, FORCE_COMPRESS))
 			seq_printf(seq, ",compress-force=%s", compress_type);
 		else

lib/Makefile +1 -0

index e38e580..2fad385 100644
@@ -113,6 +113,7 @@ obj-$(CONFIG_AVERAGE) += average.o
 
 obj-$(CONFIG_CPU_RMAP) += cpu_rmap.o
 
+CFLAGS_snappy.o += $(call cc-disable-warning, declaration-after-statement) -DNDEBUG=1
 obj-$(CONFIG_CORDIC) += cordic.o
 
 obj-$(CONFIG_LLIST) += llist.o

Signature · 1 line

1.7.4.4

Andi Kleen

author

2011-10-20 23:39 UTC #2 lore raw

PATCH 3/3 Add snappy interface to crypto API

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

Mainly so that ubifs can use it.

Snappy is a better compressor in the same niche as LZO.

Only lightly tested so far. Experiences welcome.

Cc [email protected]

Signed-off-by Andi Kleen <[email protected]>

---
 crypto/Kconfig  |    9 +++++
 crypto/Makefile |    1 +
 crypto/snappy.c |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 109 insertions(+), 0 deletions(-)
 create mode 100644 crypto/snappy.c

crypto/Kconfig +9 -0

index ae27b753..2c85991 100644
@@ -823,6 +823,15 @@ config CRYPTO_LZO
 	help
 	  This is the LZO algorithm.
 
+config CRYPTO_SNAPPY
+	tristate "Snappy compression algorithm"
+	select CRYPTO_ALGAPI
+	select SNAPPY
+	help
+	  snappy is a faster alternative to the lzo compression algorithm
+	  with comparable compression. It is very fast on 64bit systems, but also
+	  good on 32bit systems. It especially excels at already compressed data.
+
 comment "Random Number Generation"
 
 config CRYPTO_ANSI_CPRNG

crypto/Makefile +1 -0

index ce5a813..cf92f6f 100644
@@ -80,6 +80,7 @@ obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
 obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o
 obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o authencesn.o
 obj-$(CONFIG_CRYPTO_LZO) += lzo.o
+obj-$(CONFIG_CRYPTO_SNAPPY) += snappy.o
 obj-$(CONFIG_CRYPTO_RNG2) += rng.o
 obj-$(CONFIG_CRYPTO_RNG2) += krng.o
 obj-$(CONFIG_CRYPTO_ANSI_CPRNG) += ansi_cprng.o

crypto/snappy.c new file +99 -0

99 lines not on this page · raw message

Signature · 1 line

1.7.4.4

Herbert Xu

Acked-by on v1 3/3

2011-10-21 13:24 UTC #3 lore raw

Re: [PATCH 3/3] Add snappy interface to crypto API

Quoted 4 lines · Andi Kleen

From: Andi Kleen <[email protected]>
Mainly so that ubifs can use it.

Quoted 8 lines

Snappy is a better compressor in the same niche as LZO.
Only lightly tested so far. Experiences welcome.
Cc [email protected]
Cc [email protected]
Cc [email protected]
Signed-off-by Andi Kleen <[email protected]>

Acked-by Herbert Xu <[email protected]>

Cheers,

Signature · 3 lines

Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Artem Bityutskiy

2011-10-30 12:38 UTC #4 lore raw

PATCH 2/3 BTRFS: Add snappy support

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

Add support in btrfs for snappy compression.

This is based on the lzo code with minor modifications. The btrfs glue code could be significantly improved over LZO by exploiting some snappy features, but hasn't so far.

Open: implement scatter-gather support and get rid of the temporary buffers.

Some performance numbers (thanks to Jacob Sowles for running them)

bonnie++, Core i7-64bit

    block output rewrite  block input random
       K/sec     K/sec    K/sec       K/sec
None   100%      100%     100%        100%
zlib    +6.3%    +5.4%    +6.6%       +11.6%
lzo    +11.5%    +4.6%    +12.4%      +6.7%
snappy +19.3%    +28.1%   +32.6%      +9.3%

Snappy does extremly well on the 64bit architecture, outperforming everything else, sometimes with a healthy margin.

bonnie++, Atom-32bit

block output rewrite block input random

K/sec K/sec K/sec K/sec

None   100%      100%     100%        100%
zlib   -43.1%    -24.2%   -19.0%      +12.0%
lzo    +0.8%     +2.6%    +6.8%       +14.8%
snappy +19.5%    +16.2%   +24.0%      +15.7%

zlib does very poorly on Atom, actually degrading performance. snappy is generally faster or similar to LZO. The difference is not as big as on the 64bit CPU though, but still visible.

bonnie++, files, Core-i7-64bit

sequential create delete random create delete
files/sec

None       100%      100%          100%   100%
zlib       +8.3%     +10.5%        +10.3%  +1.4%
lzo        +3.8%     +3.3%         +5.4%   -3.89%
snappy     +23.7%    +37.2%        +21.8%  +23.8%

bonnie++, files, Atom-32bit

sequential create delete random create delete
files/sec

None       100%      100%          100%   100%
zlib       +3.0%     +7.9%         +5.2%  +5.1%
lzo        +8.2%     +5.9%         +4.8%  +4.6%
snappy     +3.1%     +8.5%         +5.7%  +1.3%

Creation/Deletion on Atom is a case where snappy loses to LZO, however the loss is small. On 64bit Core it's a win.

FFSB, 4 threads, stair case data pattern, Reads MB/s

Core i7-64bit                 Atom-32bit
       MB/s                   MB/s
None   100%                   100%
zlib   +8.0%                  +4.2%
lzo    +9.3%                  +4.8%
snappy +12.4%                 +7.9%

In general snappy is a better replacement for LZO, especially on 64bit, but even on 32bit.

Cc [email protected]

Signed-off-by Andi Kleen <[email protected]>

---
 fs/btrfs/Kconfig       |    1 +
 fs/btrfs/Makefile      |    3 +-
 fs/btrfs/compression.c |    1 +
 fs/btrfs/compression.h |    1 +
 fs/btrfs/ctree.h       |    9 +-
 fs/btrfs/disk-io.c     |    2 +
 fs/btrfs/ioctl.c       |    4 +
 fs/btrfs/snappy.c      |  435 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/super.c       |    9 +-
 lib/Makefile           |    1 +
 10 files changed, 461 insertions(+), 5 deletions(-)
 create mode 100644 fs/btrfs/snappy.c

fs/btrfs/Kconfig +1 -0

index ecb9fd3..d55df9c 100644
@@ -6,6 +6,7 @@ config BTRFS_FS
 	select ZLIB_DEFLATE
 	select LZO_COMPRESS
 	select LZO_DECOMPRESS
+	select SNAPPY
 	help
 	  Btrfs is a new filesystem with extents, writable snapshotting,
 	  support for multiple devices and many more features.

fs/btrfs/Makefile +2 -1

index 40e6ac0..7cd86e7 100644
@@ -7,6 +7,7 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
 	   export.o tree-log.o free-space-cache.o zlib.o lzo.o \
-	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o
+	   compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
+	   snappy.o
 
 btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o

fs/btrfs/compression.c +1 -0

index 8ec5d86..b171858 100644
@@ -729,6 +729,7 @@ static wait_queue_head_t comp_workspace_wait[BTRFS_COMPRESS_TYPES];
 struct btrfs_compress_op *btrfs_compress_op[] = {
 	&btrfs_zlib_compress,
 	&btrfs_lzo_compress,
+	&btrfs_snappy_compress,
 };
 
 int __init btrfs_init_compress(void)

fs/btrfs/compression.h +1 -0

index a12059f..971a425 100644
@@ -79,5 +79,6 @@ struct btrfs_compress_op {
 
 extern struct btrfs_compress_op btrfs_zlib_compress;
 extern struct btrfs_compress_op btrfs_lzo_compress;
+extern struct btrfs_compress_op btrfs_snappy_compress;
 
 #endif

fs/btrfs/ctree.h +6 -3

index 03912c5..7ebdae3 100644
@@ -415,6 +415,7 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
 #define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS	(1ULL << 2)
 #define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO	(1ULL << 3)
+#define BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY	(1ULL << 4)
 
 #define BTRFS_FEATURE_COMPAT_SUPP		0ULL
 #define BTRFS_FEATURE_COMPAT_RO_SUPP		0ULL

@@ -422,7 +423,8 @@ struct btrfs_super_block {
 	(BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF |		\
 	 BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL |	\
 	 BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS |		\
-	 BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO)
+	 BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO |		\
+	 BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY)
 
 /*
  * A leaf is full of items. offset and size tell us where to find

@@ -578,8 +580,9 @@ enum btrfs_compression_type {
 	BTRFS_COMPRESS_NONE  = 0,
 	BTRFS_COMPRESS_ZLIB  = 1,
 	BTRFS_COMPRESS_LZO   = 2,
-	BTRFS_COMPRESS_TYPES = 2,
-	BTRFS_COMPRESS_LAST  = 3,
+	BTRFS_COMPRESS_SNAPPY = 3,
+	BTRFS_COMPRESS_TYPES = 3,
+	BTRFS_COMPRESS_LAST  = 4,
 };
 
 struct btrfs_inode_item {

fs/btrfs/disk-io.c +2 -0

index 07b3ac6..4c04228 100644
@@ -1808,6 +1808,8 @@ struct btrfs_root *open_ctree(struct super_block *sb,
 	features |= BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF;
 	if (tree_root->fs_info->compress_type & BTRFS_COMPRESS_LZO)
 		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
+	if (tree_root->fs_info->compress_type & BTRFS_COMPRESS_SNAPPY)
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY;
 	btrfs_set_super_incompat_flags(disk_super, features);
 
 	features = btrfs_super_compat_ro_flags(disk_super) &

fs/btrfs/ioctl.c +4 -0

index 538f65a..3f15175 100644
@@ -1133,6 +1133,10 @@ int btrfs_defrag_file(struct inode *inode, struct file *file,
 		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
 		btrfs_set_super_incompat_flags(disk_super, features);
 	}
+	if (range->compress_type == BTRFS_COMPRESS_SNAPPY) {
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_SNAPPY;
+		btrfs_set_super_incompat_flags(disk_super, features);
+	}
 
 	if (!file)
 		kfree(ra);

fs/btrfs/snappy.c new file +435 -0

435 lines not on this page · raw message

fs/btrfs/super.c +8 -1

index 15634d4..d3a2d10 100644
@@ -267,6 +267,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 			} else if (strcmp(args[0].from, "lzo") == 0) {
 				compress_type = "lzo";
 				info->compress_type = BTRFS_COMPRESS_LZO;
+			} else if (strcmp(args[0].from, "snappy") == 0) {
+				compress_type = "snappy";
+				info->compress_type = BTRFS_COMPRESS_SNAPPY;
 			} else {
 				ret = -EINVAL;
 				goto out;

@@ -696,8 +699,12 @@ static int btrfs_show_options(struct seq_file *seq, struct vfsmount *vfs)
 	if (btrfs_test_opt(root, COMPRESS)) {
 		if (info->compress_type == BTRFS_COMPRESS_ZLIB)
 			compress_type = "zlib";
-		else
+		else if (info->compress_type == BTRFS_COMPRESS_SNAPPY)
+			compress_type = "snappy";
+		else if (info->compress_type == BTRFS_COMPRESS_LZO)
 			compress_type = "lzo";
+		else
+			compress_type = "?";
 		if (btrfs_test_opt(root, FORCE_COMPRESS))
 			seq_printf(seq, ",compress-force=%s", compress_type);
 		else

lib/Makefile +1 -0

index e38e580..2fad385 100644
@@ -113,6 +113,7 @@ obj-$(CONFIG_AVERAGE) += average.o
 
 obj-$(CONFIG_CPU_RMAP) += cpu_rmap.o
 
+CFLAGS_snappy.o += $(call cc-disable-warning, declaration-after-statement) -DNDEBUG=1
 obj-$(CONFIG_CORDIC) += cordic.o
 
 obj-$(CONFIG_LLIST) += llist.o

Signature · 1 line

1.7.4.4

Artem Bityutskiy

2011-10-30 12:38 UTC #5 lore raw

PATCH 3/3 Add snappy interface to crypto API

In reply to Andi Kleen

From: Andi Kleen <[email protected]>

Mainly so that ubifs can use it.

Snappy is a better compressor in the same niche as LZO.

Only lightly tested so far. Experiences welcome.

Cc [email protected]

Signed-off-by Andi Kleen <[email protected]>

---
 crypto/Kconfig  |    9 +++++
 crypto/Makefile |    1 +
 crypto/snappy.c |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 109 insertions(+), 0 deletions(-)
 create mode 100644 crypto/snappy.c

crypto/Kconfig +9 -0

index ae27b753..2c85991 100644
@@ -823,6 +823,15 @@ config CRYPTO_LZO
 	help
 	  This is the LZO algorithm.
 
+config CRYPTO_SNAPPY
+	tristate "Snappy compression algorithm"
+	select CRYPTO_ALGAPI
+	select SNAPPY
+	help
+	  snappy is a faster alternative to the lzo compression algorithm
+	  with comparable compression. It is very fast on 64bit systems, but also
+	  good on 32bit systems. It especially excels at already compressed data.
+
 comment "Random Number Generation"
 
 config CRYPTO_ANSI_CPRNG

crypto/Makefile +1 -0

index ce5a813..cf92f6f 100644
@@ -80,6 +80,7 @@ obj-$(CONFIG_CRYPTO_MICHAEL_MIC) += michael_mic.o
 obj-$(CONFIG_CRYPTO_CRC32C) += crc32c.o
 obj-$(CONFIG_CRYPTO_AUTHENC) += authenc.o authencesn.o
 obj-$(CONFIG_CRYPTO_LZO) += lzo.o
+obj-$(CONFIG_CRYPTO_SNAPPY) += snappy.o
 obj-$(CONFIG_CRYPTO_RNG2) += rng.o
 obj-$(CONFIG_CRYPTO_RNG2) += krng.o
 obj-$(CONFIG_CRYPTO_ANSI_CPRNG) += ansi_cprng.o

crypto/snappy.c new file +99 -0

99 lines not on this page · raw message

Signature · 1 line

1.7.4.4