Skip to contentSkip to conversation

btrfs: Add lzo compression support

GIT PULL New Li Zefan 4 participants 16 messages Nov 18, 2010 – Dec 26, 2010 Btrfs file system

lore
Share & tools
lore

Li Zefan

author
#0 lore raw

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Usage:

# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Compatibility:

If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.

(time in second)

           lzo        zlib        nocompress
copy:      10.6       21.7        14.9
extract:   70.1       94.4        66.6

(data size in MB)

           lzo        zlib        nocompress
copy:      185.87     108.69      394.49
extract:   193.80     132.36      381.21

Test:

Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.

Other:

The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.

Main change from v1:

- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.

---------------->
Li Zefan (6):
btrfs: Fix bugs in zlib workspace
btrfs: Fix error handling in zlib
btrfs: Allow to add new compression algorithm
btrfs: Add lzo compression support
btrfs: Allow to specify compress method when defrag
btrfs: Extract duplicate decompress code

 fs/btrfs/Makefile       |    2 +-
 fs/btrfs/btrfs_inode.h  |    2 +-
 fs/btrfs/compression.c  |  329 +++++++++++++++++++++++++++++++++++++-
 fs/btrfs/compression.h  |   72 ++++++--
 fs/btrfs/ctree.h        |   11 +-
 fs/btrfs/extent_io.c    |    5 +-
 fs/btrfs/extent_io.h    |   17 ++-
 fs/btrfs/extent_map.c   |    2 +
 fs/btrfs/extent_map.h   |    3 +-
 fs/btrfs/file.c         |    2 +
 fs/btrfs/inode.c        |   82 ++++++----
 fs/btrfs/ioctl.c        |   10 +-
 fs/btrfs/ioctl.h        |    9 +-
 fs/btrfs/lzo.c          |  409 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/ordered-data.c |   18 ++-
 fs/btrfs/ordered-data.h |    8 +-
 fs/btrfs/super.c        |   47 +++++-
 fs/btrfs/zlib.c         |  361 +++++++----------------------------------
 18 files changed, 1013 insertions(+), 376 deletions(-)
author
#1 lore raw

PATCH v2 1/6 btrfs: Fix bugs in zlib workspace

- Fix a race that can result in alloc_workspace > cpus.
- Fix to check num_workspace after wakeup.

---
 fs/btrfs/zlib.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
fs/btrfs/zlib.c +5 -2
index b9cd544..e5b8b22 100644
@@ -75,16 +75,19 @@ again:
 		return workspace;
 
 	}
-	spin_unlock(&workspace_lock);
 	if (atomic_read(&alloc_workspace) > cpus) {
 		DEFINE_WAIT(wait);
+
+		spin_unlock(&workspace_lock);
 		prepare_to_wait(&workspace_wait, &wait, TASK_UNINTERRUPTIBLE);
-		if (atomic_read(&alloc_workspace) > cpus)
+		if (atomic_read(&alloc_workspace) > cpus && !num_workspace)
 			schedule();
 		finish_wait(&workspace_wait, &wait);
 		goto again;
 	}
 	atomic_inc(&alloc_workspace);
+	spin_unlock(&workspace_lock);
+
 	workspace = kzalloc(sizeof(*workspace), GFP_NOFS);
 	if (!workspace) {
 		ret = -ENOMEM;

-- 1.7.3.1

author
#2 lore raw

PATCH v2 2/6 btrfs: Fix error handling in zlib

In reply to Li Zefan

Return failure if alloc_page() fails to allocate memory, and the upper code will just give up compression.

---
 fs/btrfs/zlib.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
fs/btrfs/zlib.c +8 -0
index e5b8b22..5a8ba17 100644
@@ -225,6 +225,10 @@ int btrfs_zlib_compress_pages(struct address_space *mapping,
 	data_in = kmap(in_page);
 
 	out_page = alloc_page(GFP_NOFS | __GFP_HIGHMEM);
+	if (out_page == NULL) {
+		ret = -ENOMEM;
+		goto out;
+	}
 	cpage_out = kmap(out_page);
 	pages[0] = out_page;
 	nr_pages = 1;
@@ -263,6 +267,10 @@ int btrfs_zlib_compress_pages(struct address_space *mapping,
 				goto out;
 			}
 			out_page = alloc_page(GFP_NOFS | __GFP_HIGHMEM);
+			if (out_page == NULL) {
+				ret = -ENOMEM;
+				goto out;
+			}
 			cpage_out = kmap(out_page);
 			pages[nr_pages] = out_page;
 			nr_pages++;

-- 1.7.3.1

author
#3 lore raw

PATCH v2 3/6 btrfs: Allow to add new compression algorithm

In reply to Li Zefan

Make the code aware of compression type, instead of always assuming zlib compression.

Also make the zlib workspace function as common code for all compression types.

---
 fs/btrfs/btrfs_inode.h  |    2 +-
 fs/btrfs/compression.c  |  236 ++++++++++++++++++++++++++++++++++++++++++-
 fs/btrfs/compression.h  |   66 +++++++++----
 fs/btrfs/ctree.h        |   10 +-
 fs/btrfs/extent_io.c    |    5 +-
 fs/btrfs/extent_io.h    |   17 +++-
 fs/btrfs/extent_map.c   |    2 +
 fs/btrfs/extent_map.h   |    3 +-
 fs/btrfs/file.c         |    2 +
 fs/btrfs/inode.c        |   82 ++++++++++------
 fs/btrfs/ioctl.c        |    4 +-
 fs/btrfs/ordered-data.c |   18 +++-
 fs/btrfs/ordered-data.h |    8 ++-
 fs/btrfs/super.c        |   47 +++++++--
 fs/btrfs/zlib.c         |  253 +++++++++-------------------------------------
 15 files changed, 473 insertions(+), 282 deletions(-)
fs/btrfs/btrfs_inode.h +1 -1
index 6ad63f1..ccc991c 100644
@@ -157,7 +157,7 @@ struct btrfs_inode {
 	/*
 	 * always compress this one file
 	 */
-	unsigned force_compress:1;
+	unsigned force_compress:4;
 
 	struct inode vfs_inode;
 };
fs/btrfs/compression.c +231 -5

257 lines not on this page · raw message

fs/btrfs/compression.h +48 -18

76 lines not on this page · raw message

fs/btrfs/ctree.h +6 -4
index 8db9234..a8983f2 100644
@@ -551,9 +551,10 @@ struct btrfs_timespec {
 } __attribute__ ((__packed__));
 
 enum btrfs_compression_type {
-	BTRFS_COMPRESS_NONE = 0,
-	BTRFS_COMPRESS_ZLIB = 1,
-	BTRFS_COMPRESS_LAST = 2,
+	BTRFS_COMPRESS_NONE  = 0,
+	BTRFS_COMPRESS_ZLIB  = 1,
+	BTRFS_COMPRESS_TYPES = 1,
+	BTRFS_COMPRESS_LAST  = 2,
 };
 
 struct btrfs_inode_item {
@@ -895,7 +896,8 @@ struct btrfs_fs_info {
 	 */
 	u64 last_trans_log_full_commit;
 	u64 open_ioctl_trans;
-	unsigned long mount_opt;
+	unsigned long mount_opt:20;
+	unsigned long compress_type:4;
 	u64 max_inline;
 	u64 alloc_start;
 	struct btrfs_transaction *running_transaction;
fs/btrfs/extent_io.c +4 -1
index 3b7eaee..d9de849 100644
@@ -2028,8 +2028,11 @@ static int __extent_read_full_page(struct extent_io_tree *tree,
 		BUG_ON(extent_map_end(em) <= cur);
 		BUG_ON(end < cur);
 
-		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags))
+		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags)) {
 			this_bio_flag = EXTENT_BIO_COMPRESSED;
+			extent_set_compress_type(&this_bio_flag,
+						 em->compress_type);
+		}
 
 		iosize = min(extent_map_end(em) - cur, end - cur + 1);
 		cur_end = min(extent_map_end(em) - 1, end);
fs/btrfs/extent_io.h +16 -1
index 1c6d4f3..2116be0 100644
@@ -20,8 +20,12 @@
 #define EXTENT_IOBITS (EXTENT_LOCKED | EXTENT_WRITEBACK)
 #define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING | EXTENT_FIRST_DELALLOC)
 
-/* flags for bio submission */
+/*
+ * flags for bio submission. The high bits indicate the compression
+ * type for this bio
+ */
 #define EXTENT_BIO_COMPRESSED 1
+#define EXTENT_BIO_FLAG_SHIFT 16
 
 /* these are bit numbers for test/set bit */
 #define EXTENT_BUFFER_UPTODATE 0
@@ -135,6 +139,17 @@ struct extent_buffer {
 	wait_queue_head_t lock_wq;
 };
 
+static inline void extent_set_compress_type(unsigned long *bio_flags,
+					    int compress_type)
+{
+	*bio_flags |= compress_type << EXTENT_BIO_FLAG_SHIFT;
+}
+
+static inline int extent_compress_type(unsigned long bio_flags)
+{
+	return bio_flags >> EXTENT_BIO_FLAG_SHIFT;
+}
+
 struct extent_map_tree;
 
 static inline struct extent_state *extent_state_next(struct extent_state *state)
fs/btrfs/extent_map.c +2 -0
index 23cb8da..b0e1fce 100644
@@ -3,6 +3,7 @@
 #include <linux/module.h>
 #include <linux/spinlock.h>
 #include <linux/hardirq.h>
+#include "ctree.h"
 #include "extent_map.h"
 
 
@@ -54,6 +55,7 @@ struct extent_map *alloc_extent_map(gfp_t mask)
 		return em;
 	em->in_tree = 0;
 	em->flags = 0;
+	em->compress_type = BTRFS_COMPRESS_NONE;
 	atomic_set(&em->refs, 1);
 	return em;
 }
fs/btrfs/extent_map.h +2 -1
index ab6d74b..28b44db 100644
@@ -26,7 +26,8 @@ struct extent_map {
 	unsigned long flags;
 	struct block_device *bdev;
 	atomic_t refs;
-	int in_tree;
+	unsigned int in_tree:1;
+	unsigned int compress_type:4;
 };
 
 struct extent_map_tree {
fs/btrfs/file.c +2 -0
index e354c33..a67768c 100644
@@ -220,6 +220,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end,
 
 			split->bdev = em->bdev;
 			split->flags = flags;
+			split->compress_type = em->compress_type;
 			ret = add_extent_mapping(em_tree, split);
 			BUG_ON(ret);
 			free_extent_map(split);
@@ -234,6 +235,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end,
 			split->len = em->start + em->len - (start + len);
 			split->bdev = em->bdev;
 			split->flags = flags;
+			split->compress_type = em->compress_type;
 
 			if (compressed) {
 				split->block_len = em->block_len;
fs/btrfs/inode.c +51 -31

229 lines not on this page · raw message

fs/btrfs/ioctl.c +2 -2
index 463d91b..460bd70 100644
@@ -671,7 +671,7 @@ static int btrfs_defrag_file(struct file *file,
 		total_read++;
 		mutex_lock(&inode->i_mutex);
 		if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS)
-			BTRFS_I(inode)->force_compress = 1;
+			BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_ZLIB;
 
 		ret  = btrfs_delalloc_reserve_space(inode, PAGE_CACHE_SIZE);
 		if (ret)
@@ -769,7 +769,7 @@ loop_unlock:
 		atomic_dec(&root->fs_info->async_submit_draining);
 
 		mutex_lock(&inode->i_mutex);
-		BTRFS_I(inode)->force_compress = 0;
+		BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_NONE;
 		mutex_unlock(&inode->i_mutex);
 	}
 
fs/btrfs/ordered-data.c +15 -3
index f4621f6..298b851 100644
@@ -172,7 +172,7 @@ static inline struct rb_node *tree_search(struct btrfs_ordered_inode_tree *tree,
  */
 static int __btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 				      u64 start, u64 len, u64 disk_len,
-				      int type, int dio)
+				      int type, int dio, int compress_type)
 {
 	struct btrfs_ordered_inode_tree *tree;
 	struct rb_node *node;
@@ -189,6 +189,7 @@ static int __btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 	entry->disk_len = disk_len;
 	entry->bytes_left = len;
 	entry->inode = inode;
+	entry->compress_type = compress_type;
 	if (type != BTRFS_ORDERED_IO_DONE && type != BTRFS_ORDERED_COMPLETE)
 		set_bit(type, &entry->flags);
 
@@ -220,14 +221,25 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 			     u64 start, u64 len, u64 disk_len, int type)
 {
 	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
-					  disk_len, type, 0);
+					  disk_len, type, 0,
+					  BTRFS_COMPRESS_NONE);
 }
 
 int btrfs_add_ordered_extent_dio(struct inode *inode, u64 file_offset,
 				 u64 start, u64 len, u64 disk_len, int type)
 {
 	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
-					  disk_len, type, 1);
+					  disk_len, type, 1,
+					  BTRFS_COMPRESS_NONE);
+}
+
+int btrfs_add_ordered_extent_compress(struct inode *inode, u64 file_offset,
+				      u64 start, u64 len, u64 disk_len,
+				      int type, int compress_type)
+{
+	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
+					  disk_len, type, 0,
+					  compress_type);
 }
 
 /*
fs/btrfs/ordered-data.h +7 -1
index 8ac3654..47ba440 100644
@@ -68,7 +68,7 @@ struct btrfs_ordered_sum {
 
 #define BTRFS_ORDERED_NOCOW 2 /* set when we want to write in place */
 
-#define BTRFS_ORDERED_COMPRESSED 3 /* writing a compressed extent */
+#define BTRFS_ORDERED_COMPRESSED 3 /* writing a zlib compressed extent */
 
 #define BTRFS_ORDERED_PREALLOC 4 /* set when writing to prealloced extent */
 
@@ -93,6 +93,9 @@ struct btrfs_ordered_extent {
 	/* flags (described above) */
 	unsigned long flags;
 
+	/* compression algorithm */
+	int compress_type;
+
 	/* reference count */
 	atomic_t refs;
 
@@ -145,6 +148,9 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 			     u64 start, u64 len, u64 disk_len, int type);
 int btrfs_add_ordered_extent_dio(struct inode *inode, u64 file_offset,
 				 u64 start, u64 len, u64 disk_len, int type);
+int btrfs_add_ordered_extent_compress(struct inode *inode, u64 file_offset,
+				      u64 start, u64 len, u64 disk_len,
+				      int type, int compress_type);
 int btrfs_add_ordered_sum(struct inode *inode,
 			  struct btrfs_ordered_extent *entry,
 			  struct btrfs_ordered_sum *sum);
fs/btrfs/super.c +36 -11

95 lines not on this page · raw message

fs/btrfs/zlib.c +50 -203

338 lines not on this page · raw message

-- 1.7.3.1

author
#4 lore raw

PATCH v2 4/6 btrfs: Add lzo compression support

In reply to Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and speed for different applications

Usage:

 # mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
 # mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Compatibility:

If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted it.

(time in second)

           lzo        zlib        nocompress
copy:      10.6       21.7        14.9
extract:   70.1       94.4        66.6

(data size in MB)

           lzo        zlib        nocompress
copy:      185.87     108.69      394.49
extract:   193.80     132.36      381.21

Changelog:

v1 -> v2:
- Select LZO_COMPRESS and LZO_DECOMPRESS in btrfs Kconfig.
- Add incompability flag.
- Fix error handling in compress code.

---
 fs/btrfs/Kconfig       |    2 +
 fs/btrfs/Makefile      |    2 +-
 fs/btrfs/compression.c |    1 +
 fs/btrfs/compression.h |    1 +
 fs/btrfs/ctree.h       |    9 +-
 fs/btrfs/disk-io.c     |    8 +-
 fs/btrfs/lzo.c         |  509 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/super.c       |    3 +
 8 files changed, 527 insertions(+), 8 deletions(-)
 create mode 100644 fs/btrfs/lzo.c
fs/btrfs/Kconfig +2 -0
index 7bb3c02..ecb9fd3 100644
@@ -4,6 +4,8 @@ config BTRFS_FS
 	select LIBCRC32C
 	select ZLIB_INFLATE
 	select ZLIB_DEFLATE
+	select LZO_COMPRESS
+	select LZO_DECOMPRESS
 	help
 	  Btrfs is a new filesystem with extents, writable snapshotting,
 	  support for multiple devices and many more features.
fs/btrfs/Makefile +1 -1
index a35eb36..31610ea 100644
@@ -6,5 +6,5 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   transaction.o inode.o file.o tree-defrag.o \
 	   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
-	   export.o tree-log.o acl.o free-space-cache.o zlib.o \
+	   export.o tree-log.o acl.o free-space-cache.o zlib.o lzo.o \
 	   compression.o delayed-ref.o relocation.o
fs/btrfs/compression.c +1 -0
index a8c8292..d814891 100644
@@ -704,6 +704,7 @@ static wait_queue_head_t comp_workspace_wait[BTRFS_COMPRESS_TYPES];
 
 struct btrfs_compress_op *btrfs_compress_op[] = {
 	&btrfs_zlib_compress,
+	&btrfs_lzo_compress,
 };
 
 int __init btrfs_init_compress(void)
fs/btrfs/compression.h +1 -0
index 9b5f2f3..f7ce217 100644
@@ -73,5 +73,6 @@ struct btrfs_compress_op {
 };
 
 extern struct btrfs_compress_op btrfs_zlib_compress;
+extern struct btrfs_compress_op btrfs_lzo_compress;
 
 #endif
fs/btrfs/ctree.h +6 -3
index a8983f2..2b54650 100644
@@ -398,13 +398,15 @@ struct btrfs_super_block {
 #define BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF	(1ULL << 0)
 #define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL	(1ULL << 1)
 #define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS	(1ULL << 2)
+#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO	(1ULL << 3)
 
 #define BTRFS_FEATURE_COMPAT_SUPP		0ULL
 #define BTRFS_FEATURE_COMPAT_RO_SUPP		0ULL
 #define BTRFS_FEATURE_INCOMPAT_SUPP			\
 	(BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF |		\
 	 BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL |	\
-	 BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS)
+	 BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS |		\
+	 BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO)
 
 /*
  * A leaf is full of items. offset and size tell us where to find
@@ -553,8 +555,9 @@ struct btrfs_timespec {
 enum btrfs_compression_type {
 	BTRFS_COMPRESS_NONE  = 0,
 	BTRFS_COMPRESS_ZLIB  = 1,
-	BTRFS_COMPRESS_TYPES = 1,
-	BTRFS_COMPRESS_LAST  = 2,
+	BTRFS_COMPRESS_LZO   = 2,
+	BTRFS_COMPRESS_TYPES = 2,
+	BTRFS_COMPRESS_LAST  = 3,
 };
 
 struct btrfs_inode_item {
fs/btrfs/disk-io.c +4 -4
index b40dfe4..394060f 100644
@@ -1717,10 +1717,10 @@ struct btrfs_root *open_ctree(struct super_block *sb,
 	}
 
 	features = btrfs_super_incompat_flags(disk_super);
-	if (!(features & BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF)) {
-		features |= BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF;
-		btrfs_set_super_incompat_flags(disk_super, features);
-	}
+	features |= BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF;
+	if (tree_root->fs_info->compress_type & BTRFS_COMPRESS_LZO)
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
+	btrfs_set_super_incompat_flags(disk_super, features);
 
 	features = btrfs_super_compat_ro_flags(disk_super) &
 		~BTRFS_FEATURE_COMPAT_RO_SUPP;
fs/btrfs/lzo.c new file +509 -0

509 lines not on this page · raw message

fs/btrfs/super.c +3 -0
index 0318596..0e998a5 100644
@@ -168,6 +168,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 			    strcmp(args[0].from, "zlib") == 0) {
 				compress_type = "zlib";
 				info->compress_type = BTRFS_COMPRESS_ZLIB;
+			} else if (strcmp(args[0].from, "lzo") == 0) {
+				compress_type = "lzo";
+				info->compress_type = BTRFS_COMPRESS_LZO;
 			} else {
 				ret = -EINVAL;
 				goto out;

-- 1.7.3.1

author
#5 lore raw

PATCH v2 5/6 btrfs: Allow to specify compress method when defrag

In reply to Li Zefan

Update defrag ioctl, so one can choose lzo or zlib when turning on compression in defrag operation.

Changelog:

v1 -> v2
- Add incompability flag.
- Fix to check invalid compress type.

---
 fs/btrfs/ioctl.c |   24 +++++++++++++++++++++++-
 fs/btrfs/ioctl.h |    9 ++++++++-
 2 files changed, 31 insertions(+), 2 deletions(-)
fs/btrfs/ioctl.c +23 -1
index 460bd70..1714c48 100644
@@ -626,9 +626,11 @@ static int btrfs_defrag_file(struct file *file,
 	struct extent_io_tree *io_tree = &BTRFS_I(inode)->io_tree;
 	struct btrfs_ordered_extent *ordered;
 	struct page *page;
+	struct btrfs_super_block *disk_super;
 	unsigned long last_index;
 	unsigned long ra_pages = root->fs_info->bdi.ra_pages;
 	unsigned long total_read = 0;
+	u64 features;
 	u64 page_start;
 	u64 page_end;
 	u64 last_len = 0;
@@ -636,6 +638,18 @@ static int btrfs_defrag_file(struct file *file,
 	u64 defrag_end = 0;
 	unsigned long i;
 	int ret;
+	int compress_type = BTRFS_COMPRESS_ZLIB;
+
+	if (range->compress_type > BTRFS_COMPRESS_TYPES)
+		return -EINVAL;
+
+	/*
+	 * Compression will be enabled only if BTRFS_DEFRAG_RANGE_COMPRESS
+	 * is set, and the default is zlib if compression method is not
+	 * specified.
+	 */
+	if (range->compress_type)
+		compress_type = range->compress_type;
 
 	if (inode->i_size == 0)
 		return 0;
@@ -671,7 +685,7 @@ static int btrfs_defrag_file(struct file *file,
 		total_read++;
 		mutex_lock(&inode->i_mutex);
 		if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS)
-			BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_ZLIB;
+			BTRFS_I(inode)->force_compress = compress_type;
 
 		ret  = btrfs_delalloc_reserve_space(inode, PAGE_CACHE_SIZE);
 		if (ret)
@@ -773,6 +787,14 @@ loop_unlock:
 		mutex_unlock(&inode->i_mutex);
 	}
 
+	disk_super = &root->fs_info->super_copy;
+	features = btrfs_super_incompat_flags(disk_super);
+	if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS &&
+	    compress_type == BTRFS_COMPRESS_LZO) {
+		features |= BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO;
+		btrfs_set_super_incompat_flags(disk_super, features);
+	}
+
 	return 0;
 
 err_reservations:
fs/btrfs/ioctl.h +8 -1
index 17c99eb..0821367 100644
@@ -129,8 +129,15 @@ struct btrfs_ioctl_defrag_range_args {
 	 */
 	__u32 extent_thresh;
 
+	/*
+	 * which compression method to use if turning on compression
+	 * for this defrag operation.  If unspecified, zlib will
+	 * be used
+	 */
+	__u32 compress_type;
+
 	/* spare for later */
-	__u32 unused[5];
+	__u32 unused[4];
 };
 
 struct btrfs_ioctl_space_info {

-- 1.7.3.1

author
#6 lore raw

PATCH v2 6/6 btrfs: Extract duplicate decompress code

In reply to Li Zefan

Add a common function to copy decompressed data from working buffer to bio pages.

---
 fs/btrfs/compression.c |   92 +++++++++++++++++++++++++++++++++++++++
 fs/btrfs/compression.h |    5 ++
 fs/btrfs/lzo.c         |  101 +++-----------------------------------------
 fs/btrfs/zlib.c        |  111 +++++------------------------------------------
 4 files changed, 115 insertions(+), 194 deletions(-)
fs/btrfs/compression.c +92 -0
index d814891..e58da2e 100644
@@ -917,3 +917,95 @@ void __exit btrfs_exit_compress(void)
 {
 	free_workspaces();
 }
+
+/*
+ * Copy uncompressed data from working buffer to pages.
+ *
+ * buf_start is the byte offset we're of the start of our workspace buffer.
+ *
+ * total_out is the last byte of the buffer
+ */
+int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
+			      unsigned long total_out, u64 disk_start,
+			      struct bio_vec *bvec, int vcnt,
+			      unsigned long *page_index,
+			      unsigned long *pg_offset)
+{
+	unsigned long buf_offset;
+	unsigned long current_buf_start;
+	unsigned long start_byte;
+	unsigned long working_bytes = total_out - buf_start;
+	unsigned long bytes;
+	char *kaddr;
+	struct page *page_out = bvec[*page_index].bv_page;
+
+	/*
+	 * start byte is the first byte of the page we're currently
+	 * copying into relative to the start of the compressed data.
+	 */
+	start_byte = page_offset(page_out) - disk_start;
+
+	/* we haven't yet hit data corresponding to this page */
+	if (total_out <= start_byte)
+		return 1;
+
+	/*
+	 * the start of the data we care about is offset into
+	 * the middle of our working buffer
+	 */
+	if (total_out > start_byte && buf_start < start_byte) {
+		buf_offset = start_byte - buf_start;
+		working_bytes -= buf_offset;
+	} else {
+		buf_offset = 0;
+	}
+	current_buf_start = buf_start;
+
+	/* copy bytes from the working buffer into the pages */
+	while (working_bytes > 0) {
+		bytes = min(PAGE_CACHE_SIZE - *pg_offset,
+			    PAGE_CACHE_SIZE - buf_offset);
+		bytes = min(bytes, working_bytes);
+		kaddr = kmap_atomic(page_out, KM_USER0);
+		memcpy(kaddr + *pg_offset, buf + buf_offset, bytes);
+		kunmap_atomic(kaddr, KM_USER0);
+		flush_dcache_page(page_out);
+
+		*pg_offset += bytes;
+		buf_offset += bytes;
+		working_bytes -= bytes;
+		current_buf_start += bytes;
+
+		/* check if we need to pick another page */
+		if (*pg_offset == PAGE_CACHE_SIZE) {
+			(*page_index)++;
+			if (*page_index >= vcnt)
+				return 0;
+
+			page_out = bvec[*page_index].bv_page;
+			*pg_offset = 0;
+			start_byte = page_offset(page_out) - disk_start;
+
+			/*
+			 * make sure our new page is covered by this
+			 * working buffer
+			 */
+			if (total_out <= start_byte)
+				return 1;
+
+			/*
+			 * the next page in the biovec might not be adjacent
+			 * to the last page, but it might still be found
+			 * inside this working buffer. bump our offset pointer
+			 */
+			if (total_out > start_byte &&
+			    current_buf_start < start_byte) {
+				buf_offset = start_byte - buf_start;
+				working_bytes = total_out - start_byte;
+				current_buf_start = buf_start + buf_offset;
+			}
+		}
+	}
+
+	return 1;
+}
fs/btrfs/compression.h +5 -0
index f7ce217..5100017 100644
@@ -34,6 +34,11 @@ int btrfs_decompress_biovec(int type, struct page **pages_in, u64 disk_start,
 			    struct bio_vec *bvec, int vcnt, size_t srclen);
 int btrfs_decompress(int type, unsigned char *data_in, struct page *dest_page,
 		     unsigned long start_byte, size_t srclen, size_t destlen);
+int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
+			      unsigned long total_out, u64 disk_start,
+			      struct bio_vec *bvec, int vcnt,
+			      unsigned long *page_index,
+			      unsigned long *pg_offset);
 
 int btrfs_submit_compressed_write(struct inode *inode, u64 start,
 				  unsigned long len, u64 disk_start,
fs/btrfs/lzo.c +6 -95

132 lines not on this page · raw message

fs/btrfs/zlib.c +12 -99

144 lines not on this page · raw message

-- 1.7.3.1

#7 lore raw
In reply to Li Zefan
Quoted 70 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Usage:

# mount -t btrfs -o compress[=3D<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=3D<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Compatibility:

If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.

(time in second)
=A0 =A0 =A0 =A0 =A0 lzo =A0 =A0 =A0 =A0zlib =A0 =A0 =A0 =A0nocompress
copy: =A0 =A0 =A010.6 =A0 =A0 =A0 21.7 =A0 =A0 =A0 =A014.9
extract: =A0 70.1 =A0 =A0 =A0 94.4 =A0 =A0 =A0 =A066.6

(data size in MB)
=A0 =A0 =A0 =A0 =A0 lzo =A0 =A0 =A0 =A0zlib =A0 =A0 =A0 =A0nocompress
copy: =A0 =A0 =A0185.87 =A0 =A0 108.69 =A0 =A0 =A0394.49
extract: =A0 193.80 =A0 =A0 132.36 =A0 =A0 =A0381.21

Test:

Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.

Other:

The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.

Main change from v1:

- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.

---------------->
Li Zefan (6):
=A0 =A0 =A0btrfs: Fix bugs in zlib workspace
=A0 =A0 =A0btrfs: Fix error handling in zlib
=A0 =A0 =A0btrfs: Allow to add new compression algorithm =A0 =A0 =A0btrfs: Add lzo compression support
=A0 =A0 =A0btrfs: Allow to specify compress method when defrag =A0 =A0 =A0btrfs: Extract duplicate decompress code

Quoted 3 lines

=A0fs/btrfs/Makefile =A0 =A0 =A0 | =A0 =A02 +- =A0fs/btrfs/btrfs_inode.h =A0| =A0 =A02 +- =A0fs/btrfs/compression.c =A0| =A0329 +++++++++++++++++++++++++++++++=

++++++-

Quoted 11 lines

=A0fs/btrfs/compression.h =A0| =A0 72 ++++++--
=A0fs/btrfs/ctree.h =A0 =A0 =A0 =A0| =A0 11 +-
=A0fs/btrfs/extent_io.c =A0 =A0| =A0 =A05 +-
=A0fs/btrfs/extent_io.h =A0 =A0| =A0 17 ++-
=A0fs/btrfs/extent_map.c =A0 | =A0 =A02 +
=A0fs/btrfs/extent_map.h =A0 | =A0 =A03 +-

=A0fs/btrfs/file.c =A0 =A0 =A0 =A0 | =A0 =A02 +
=A0fs/btrfs/inode.c =A0 =A0 =A0 =A0| =A0 82 ++++++----
=A0fs/btrfs/ioctl.c =A0 =A0 =A0 =A0| =A0 10 +-
=A0fs/btrfs/ioctl.h =A0 =A0 =A0 =A0| =A0 =A09 +-

=A0fs/btrfs/lzo.c =A0 =A0 =A0 =A0 =A0| =A0409 +++++++++++++++++++++++=

++++++++++++++++++++++++

Quoted 4 lines

=A0fs/btrfs/ordered-data.c | =A0 18 ++- =A0fs/btrfs/ordered-data.h | =A0 =A08 +- =A0fs/btrfs/super.c =A0 =A0 =A0 =A0| =A0 47 +++++- =A0fs/btrfs/zlib.c =A0 =A0 =A0 =A0 | =A0361 +++++++------------------=

----------------

Quoted 1 line

=A018 files changed, 1013 insertions(+), 376 deletions(-)

is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.

what's the proper place (repo/branch) to see what is pending?

C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#8 lore raw
Quoted 91 lines · C Anthony Risinger
Quoted 81 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Usage:

# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Compatibility:

If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.

(time in second)
lzo zlib nocompress

copy:      10.6       21.7        14.9
extract:   70.1       94.4        66.6

(data size in MB)
lzo zlib nocompress

copy:      185.87     108.69      394.49
extract:   193.80     132.36      381.21

Test:

Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.

Other:

The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.

Main change from v1:

- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.

---------------->
Li Zefan (6):
btrfs: Fix bugs in zlib workspace
btrfs: Fix error handling in zlib
btrfs: Allow to add new compression algorithm
btrfs: Add lzo compression support
btrfs: Allow to specify compress method when defrag
btrfs: Extract duplicate decompress code

 fs/btrfs/Makefile       |    2 +-
 fs/btrfs/btrfs_inode.h  |    2 +-
 fs/btrfs/compression.c  |  329 +++++++++++++++++++++++++++++++++++++-
 fs/btrfs/compression.h  |   72 ++++++--
 fs/btrfs/ctree.h        |   11 +-
 fs/btrfs/extent_io.c    |    5 +-
 fs/btrfs/extent_io.h    |   17 ++-
 fs/btrfs/extent_map.c   |    2 +
 fs/btrfs/extent_map.h   |    3 +-
 fs/btrfs/file.c         |    2 +
 fs/btrfs/inode.c        |   82 ++++++----
Quoted 8 lines
 fs/btrfs/ioctl.c        |   10 +-
 fs/btrfs/ioctl.h        |    9 +-
 fs/btrfs/lzo.c          |  409 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/ordered-data.c |   18 ++-
 fs/btrfs/ordered-data.h |    8 +-
 fs/btrfs/super.c        |   47 +++++-
 fs/btrfs/zlib.c         |  361 +++++++----------------------------------
 18 files changed, 1013 insertions(+), 376 deletions(-)
Quoted 5 lines

is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.

what's the proper place (repo/branch) to see what is pending?

As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.

Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.

#9 lore raw
Quoted 2 lines · Li Zefan
Quoted 1 line · C Anthony Risinger

On Wed, Nov 17, 2010 at 8:08 PM, Li Zefan <[email protected]> wrot=

e:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

You can pull from:

=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allo=

w

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

is this in a branch somewhere? =A0or for inclusion in .37/.38? =A0th=

is is

Quoted 6 lines
Quoted 3 lines

a very attractive feature.

what's the proper place (repo/branch) to see what is pending?

As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.

:-(

yeah i thought it was already too late for .37, but i'd love to see this for .38... should add even more kick to my eee s101 netbook.

Quoted 3 lines

Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.

i knew about Chris' tree, i probably should have been more clear; instead of "pending" i should have said "awaiting review", ie. is there a repo Chris or someone else keeps with all the potential patches (such as this, or the recent readonly ones, etc.)?

i keep a pretty good eye on the list, but i thought maybe someone already had a nice central point to see these "up and coming" patches that aren't necessarily "pending" for mainline yet.

thanks Li,

C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#10 lore raw
Quoted 23 lines · C Anthony Risinger
Quoted 10 lines · Li Zefan
Quoted 9 lines · C Anthony Risinger
Quoted 8 lines · Li Zefan

You can pull from:

git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Quoted 6 lines
Quoted 4 lines

is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.

what's the proper place (repo/branch) to see what is pending?

As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.

:-(

yeah i thought it was already too late for .37, but i'd love to see this for .38... should add even more kick to my eee s101 netbook.

Quoted 9 lines
Quoted 3 lines

Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.

i knew about Chris' tree, i probably should have been more clear; instead of "pending" i should have said "awaiting review", ie. is there a repo Chris or someone else keeps with all the potential patches (such as this, or the recent readonly ones, etc.)?

Unfortunately no..

Quoted 4 lines

i keep a pretty good eye on the list, but i thought maybe someone already had a nice central point to see these "up and coming" patches that aren't necessarily "pending" for mainline yet.

But btrfs-unstable tree should have acted as this role. We put those patches that are ready for next release in different branches in that tree, and merge them into master branch later. And add that tree into linux-next for wider testing..

#11 lore raw
In reply to Li Zefan
Quoted 15 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Quoted 7 lines

Usage:

# mount -t btrfs -o compress[=3D<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=3D<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

This doesn't seem to work with -o remount. "mount / -o remount,compress=3Dlzo" fails with "[90192.345382] btrfs: unrecognized mount option 'compress=3Dlzo'". The -clzo option to defrag works fine. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#12 lore raw
Quoted 22 lines · cwillu
Quoted 21 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

git://repo.or.cz/linux-btrfs-devel.git lzo-support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Usage:

# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt
Quoted 6 lines
Quoted 1 line

"-o compress" without argument is still allowed for compatability.

This doesn't seem to work with -o remount. "mount / -o remount,compress=lzo" fails with "[90192.345382] btrfs: unrecognized mount option 'compress=lzo'". The -clzo option to defrag works fine.

It works for me:

# mount -t btrfs -o compress=zlib /dev/loop1 /mnt
# mount -o remount,compress=lzo /dev/loop1 /mnt
# dmesg
...
device fsid c142fef10429f78f-540f99158993f7a2 devid 1 transid 31 /dev/loop1
btrfs: use zlib compression
btrfs: use lzo compression
#13 lore raw

Excerpts from Li Zefan's message of 2010-12-24 01:55:53 -0500:

Quoted 11 lines
Quoted 10 lines · cwillu
Quoted 9 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

git://repo.or.cz/linux-btrfs-devel.git lzo-support

I pulled this in last night and an all night run completed without troubles. I'm trying a few more this morning, but it looks really good.

I think I'll also bump the compressed extent size up to 512KB, which will cut down on the number of extents in files that compress well.

-chris

#14 lore raw

Excerpts from Chris Mason's message of 2010-12-24 07:32:16 -0500:

Quoted 11 lines

Excerpts from Li Zefan's message of 2010-12-24 01:55:53 -0500:

Quoted 10 lines
Quoted 9 lines · cwillu
Quoted 8 lines · Li Zefan

Hi Chris,

Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.

You can pull from:

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

git://repo.or.cz/linux-btrfs-devel.git lzo-support

I pulled this in last night and an all night run completed without troubles. I'm trying a few more this morning, but it looks really good.

I think I'll also bump the compressed extent size up to 512KB, which will cut down on the number of extents in files that compress well.

The next branch of the btrfs unstable tree has lzo and the other fujitsu work so far. This is going to be the building block for our 2.6.38 pull request to Linus.

It might get rebased, but it seems good so far.

-chris

#15 lore raw
Quoted 1 line · Chris Mason

This is going to be the building block > for our 2.6.38 pull

request to Linus.

hooray christmas came early this year! what a nice gift, thanks!

heh, thanks for all the great work guys. I'm doing my best to spread the good btrfs word all over the net, correcting misconceptions, writing guides/hooks/scripts, and generally helping those that need it; the majority are most impressed, and looking forward to the onset.

Just thought I'd say that my travels have found many satisfied users, who are, including myself, very appreciative of the work being done here.

So thanks! Happy holidays! And to all a good night!

C Anthony [mobile]

Quoted 1 line