Skip to contentSkip to conversation

btrfs: Add lzo compression support

PATCH Superseded Li Zefan 3 participants 11 messages Oct 25, 2010 – Nov 16, 2010 Btrfs file system

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Li Zefan

author
#0 lore raw

cover btrfs: Add lzo compression support

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

Usage:

# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.

(time in second)

           lzo        zlib        nocompress
copy:      10.6       21.7        14.9
extract:   70.1       94.4        66.6

(data size in MB)

           lzo        zlib        nocompress
copy:      185.87     108.69      394.49
extract:   193.80     132.36      381.21

Other:

The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.

TODO:

- Update btrfs-progs accordingly.

---
 fs/btrfs/Makefile       |    2 +-
 fs/btrfs/btrfs_inode.h  |    2 +-
 fs/btrfs/compression.c  |  329 +++++++++++++++++++++++++++++++++++++-
 fs/btrfs/compression.h  |   72 ++++++--
 fs/btrfs/ctree.h        |   11 +-
 fs/btrfs/extent_io.c    |    5 +-
 fs/btrfs/extent_io.h    |   17 ++-
 fs/btrfs/extent_map.c   |    2 +
 fs/btrfs/extent_map.h   |    3 +-
 fs/btrfs/file.c         |    2 +
 fs/btrfs/inode.c        |   82 ++++++----
 fs/btrfs/ioctl.c        |   10 +-
 fs/btrfs/ioctl.h        |    9 +-
 fs/btrfs/lzo.c          |  409 +++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/ordered-data.c |   18 ++-
 fs/btrfs/ordered-data.h |    8 +-
 fs/btrfs/super.c        |   47 +++++-
 fs/btrfs/zlib.c         |  361 +++++++----------------------------------
 18 files changed, 1013 insertions(+), 376 deletions(-)
author
#1 lore raw

PATCH 1/5 btrfs: Fix bugs in zlib workspace

- Fix a race that can result in alloc_workspace > cpus.
- Fix to check num_workspace after wakeup.

---
 fs/btrfs/zlib.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
fs/btrfs/zlib.c +5 -2
index b9cd544..e5b8b22 100644
@@ -75,16 +75,19 @@ again:
 		return workspace;
 
 	}
-	spin_unlock(&workspace_lock);
 	if (atomic_read(&alloc_workspace) > cpus) {
 		DEFINE_WAIT(wait);
+
+		spin_unlock(&workspace_lock);
 		prepare_to_wait(&workspace_wait, &wait, TASK_UNINTERRUPTIBLE);
-		if (atomic_read(&alloc_workspace) > cpus)
+		if (atomic_read(&alloc_workspace) > cpus && !num_workspace)
 			schedule();
 		finish_wait(&workspace_wait, &wait);
 		goto again;
 	}
 	atomic_inc(&alloc_workspace);
+	spin_unlock(&workspace_lock);
+
 	workspace = kzalloc(sizeof(*workspace), GFP_NOFS);
 	if (!workspace) {
 		ret = -ENOMEM;
Signature · 1 line
1.7.0.1
author
#2 lore raw

PATCH 2/5 btrfs: Allow to add new compression algorithm

In reply to Li Zefan

Make the code aware of compression type, instead of always assuming zlib compression.

Also make the zlib workspace function as common code for all compression types.

---
 fs/btrfs/btrfs_inode.h  |    2 +-
 fs/btrfs/compression.c  |  236 ++++++++++++++++++++++++++++++++++++++++++-
 fs/btrfs/compression.h  |   66 +++++++++----
 fs/btrfs/ctree.h        |   10 +-
 fs/btrfs/extent_io.c    |    5 +-
 fs/btrfs/extent_io.h    |   17 +++-
 fs/btrfs/extent_map.c   |    2 +
 fs/btrfs/extent_map.h   |    3 +-
 fs/btrfs/file.c         |    2 +
 fs/btrfs/inode.c        |   82 ++++++++++------
 fs/btrfs/ioctl.c        |    4 +-
 fs/btrfs/ordered-data.c |   18 +++-
 fs/btrfs/ordered-data.h |    8 ++-
 fs/btrfs/super.c        |   41 +++++++-
 fs/btrfs/zlib.c         |  253 +++++++++-------------------------------------
 15 files changed, 472 insertions(+), 277 deletions(-)
fs/btrfs/btrfs_inode.h +1 -1
index 6ad63f1..ccc991c 100644
@@ -157,7 +157,7 @@ struct btrfs_inode {
 	/*
 	 * always compress this one file
 	 */
-	unsigned force_compress:1;
+	unsigned force_compress:4;
 
 	struct inode vfs_inode;
 };
fs/btrfs/compression.c +231 -5

257 lines not on this page · raw message

fs/btrfs/compression.h +48 -18

76 lines not on this page · raw message

fs/btrfs/ctree.h +6 -4
index e9bf864..b0f8b4d 100644
@@ -528,9 +528,10 @@ struct btrfs_timespec {
 } __attribute__ ((__packed__));
 
 enum btrfs_compression_type {
-	BTRFS_COMPRESS_NONE = 0,
-	BTRFS_COMPRESS_ZLIB = 1,
-	BTRFS_COMPRESS_LAST = 2,
+	BTRFS_COMPRESS_NONE  = 0,
+	BTRFS_COMPRESS_ZLIB  = 1,
+	BTRFS_COMPRESS_TYPES = 1,
+	BTRFS_COMPRESS_LAST  = 2,
 };
 
 struct btrfs_inode_item {
@@ -857,7 +858,8 @@ struct btrfs_fs_info {
 	 */
 	u64 last_trans_log_full_commit;
 	u64 open_ioctl_trans;
-	unsigned long mount_opt;
+	unsigned long mount_opt:20;
+	unsigned long compress_type:4;
 	u64 max_inline;
 	u64 alloc_start;
 	struct btrfs_transaction *running_transaction;
fs/btrfs/extent_io.c +4 -1
index 41277d6..56eba19 100644
@@ -2072,8 +2072,11 @@ static int __extent_read_full_page(struct extent_io_tree *tree,
 		BUG_ON(extent_map_end(em) <= cur);
 		BUG_ON(end < cur);
 
-		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags))
+		if (test_bit(EXTENT_FLAG_COMPRESSED, &em->flags)) {
 			this_bio_flag = EXTENT_BIO_COMPRESSED;
+			extent_set_compress_type(&this_bio_flag,
+						 em->compress_type);
+		}
 
 		iosize = min(extent_map_end(em) - cur, end - cur + 1);
 		cur_end = min(extent_map_end(em) - 1, end);
fs/btrfs/extent_io.h +16 -1
index 5691c7b..65f06c8 100644
@@ -20,8 +20,12 @@
 #define EXTENT_IOBITS (EXTENT_LOCKED | EXTENT_WRITEBACK)
 #define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING | EXTENT_FIRST_DELALLOC)
 
-/* flags for bio submission */
+/*
+ * flags for bio submission. The high bits indicate the compression
+ * type for this bio
+ */
 #define EXTENT_BIO_COMPRESSED 1
+#define EXTENT_BIO_FLAG_SHIFT 16
 
 /* these are bit numbers for test/set bit */
 #define EXTENT_BUFFER_UPTODATE 0
@@ -135,6 +139,17 @@ struct extent_buffer {
 	wait_queue_head_t lock_wq;
 };
 
+static inline void extent_set_compress_type(unsigned long *bio_flags,
+					    int compress_type)
+{
+	*bio_flags |= compress_type << EXTENT_BIO_FLAG_SHIFT;
+}
+
+static inline int extent_compress_type(unsigned long bio_flags)
+{
+	return bio_flags >> EXTENT_BIO_FLAG_SHIFT;
+}
+
 struct extent_map_tree;
 
 static inline struct extent_state *extent_state_next(struct extent_state *state)
fs/btrfs/extent_map.c +2 -0
index 23cb8da..b0e1fce 100644
@@ -3,6 +3,7 @@
 #include <linux/module.h>
 #include <linux/spinlock.h>
 #include <linux/hardirq.h>
+#include "ctree.h"
 #include "extent_map.h"
 
 
@@ -54,6 +55,7 @@ struct extent_map *alloc_extent_map(gfp_t mask)
 		return em;
 	em->in_tree = 0;
 	em->flags = 0;
+	em->compress_type = BTRFS_COMPRESS_NONE;
 	atomic_set(&em->refs, 1);
 	return em;
 }
fs/btrfs/extent_map.h +2 -1
index ab6d74b..28b44db 100644
@@ -26,7 +26,8 @@ struct extent_map {
 	unsigned long flags;
 	struct block_device *bdev;
 	atomic_t refs;
-	int in_tree;
+	unsigned int in_tree:1;
+	unsigned int compress_type:4;
 };
 
 struct extent_map_tree {
fs/btrfs/file.c +2 -0
index 7f29464..8585a72 100644
@@ -220,6 +220,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end,
 
 			split->bdev = em->bdev;
 			split->flags = flags;
+			split->compress_type = em->compress_type;
 			ret = add_extent_mapping(em_tree, split);
 			BUG_ON(ret);
 			free_extent_map(split);
@@ -234,6 +235,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end,
 			split->len = em->start + em->len - (start + len);
 			split->bdev = em->bdev;
 			split->flags = flags;
+			split->compress_type = em->compress_type;
 
 			if (compressed) {
 				split->block_len = em->block_len;
fs/btrfs/inode.c +51 -31

229 lines not on this page · raw message

fs/btrfs/ioctl.c +2 -2
index ca52cfe..ecc49ee 100644
@@ -585,7 +585,7 @@ static int btrfs_defrag_file(struct file *file,
 		total_read++;
 		mutex_lock(&inode->i_mutex);
 		if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS)
-			BTRFS_I(inode)->force_compress = 1;
+			BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_ZLIB;
 
 		ret  = btrfs_delalloc_reserve_space(inode, PAGE_CACHE_SIZE);
 		if (ret)
@@ -683,7 +683,7 @@ loop_unlock:
 		atomic_dec(&root->fs_info->async_submit_draining);
 
 		mutex_lock(&inode->i_mutex);
-		BTRFS_I(inode)->force_compress = 0;
+		BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_NONE;
 		mutex_unlock(&inode->i_mutex);
 	}
 
fs/btrfs/ordered-data.c +15 -3
index 5ebffc4..68c88ea 100644
@@ -172,7 +172,7 @@ static inline struct rb_node *tree_search(struct btrfs_ordered_inode_tree *tree,
  */
 static int __btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 				      u64 start, u64 len, u64 disk_len,
-				      int type, int dio)
+				      int type, int dio, int compress_type)
 {
 	struct btrfs_ordered_inode_tree *tree;
 	struct rb_node *node;
@@ -189,6 +189,7 @@ static int __btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 	entry->disk_len = disk_len;
 	entry->bytes_left = len;
 	entry->inode = inode;
+	entry->compress_type = compress_type;
 	if (type != BTRFS_ORDERED_IO_DONE && type != BTRFS_ORDERED_COMPLETE)
 		set_bit(type, &entry->flags);
 
@@ -220,14 +221,25 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 			     u64 start, u64 len, u64 disk_len, int type)
 {
 	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
-					  disk_len, type, 0);
+					  disk_len, type, 0,
+					  BTRFS_COMPRESS_NONE);
 }
 
 int btrfs_add_ordered_extent_dio(struct inode *inode, u64 file_offset,
 				 u64 start, u64 len, u64 disk_len, int type)
 {
 	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
-					  disk_len, type, 1);
+					  disk_len, type, 1,
+					  BTRFS_COMPRESS_NONE);
+}
+
+int btrfs_add_ordered_extent_compress(struct inode *inode, u64 file_offset,
+				      u64 start, u64 len, u64 disk_len,
+				      int type, int compress_type)
+{
+	return __btrfs_add_ordered_extent(inode, file_offset, start, len,
+					  disk_len, type, 0,
+					  compress_type);
 }
 
 /*
fs/btrfs/ordered-data.h +7 -1
index 8ac3654..47ba440 100644
@@ -68,7 +68,7 @@ struct btrfs_ordered_sum {
 
 #define BTRFS_ORDERED_NOCOW 2 /* set when we want to write in place */
 
-#define BTRFS_ORDERED_COMPRESSED 3 /* writing a compressed extent */
+#define BTRFS_ORDERED_COMPRESSED 3 /* writing a zlib compressed extent */
 
 #define BTRFS_ORDERED_PREALLOC 4 /* set when writing to prealloced extent */
 
@@ -93,6 +93,9 @@ struct btrfs_ordered_extent {
 	/* flags (described above) */
 	unsigned long flags;
 
+	/* compression algorithm */
+	int compress_type;
+
 	/* reference count */
 	atomic_t refs;
 
@@ -145,6 +148,9 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset,
 			     u64 start, u64 len, u64 disk_len, int type);
 int btrfs_add_ordered_extent_dio(struct inode *inode, u64 file_offset,
 				 u64 start, u64 len, u64 disk_len, int type);
+int btrfs_add_ordered_extent_compress(struct inode *inode, u64 file_offset,
+				      u64 start, u64 len, u64 disk_len,
+				      int type, int compress_type);
 int btrfs_add_ordered_sum(struct inode *inode,
 			  struct btrfs_ordered_extent *entry,
 			  struct btrfs_ordered_sum *sum);
fs/btrfs/super.c +35 -6

90 lines not on this page · raw message

fs/btrfs/zlib.c +50 -203

338 lines not on this page · raw message

Signature · 1 line
1.7.0.1
author
#3 lore raw

PATCH 3/5 btrfs: Add lzo compression support

In reply to Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and speed for different applications

Usage:

# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt
or
# mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt

"-o compress" without argument is still allowed for compatability.

Performance:

The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted it.

(time in second)

           lzo        zlib        nocompress
copy:      10.6       21.7        14.9
extract:   70.1       94.4        66.6

(data size in MB)

           lzo        zlib        nocompress
copy:      185.87     108.69      394.49
extract:   193.80     132.36      381.21
---
 fs/btrfs/Makefile      |    2 +-
 fs/btrfs/compression.c |    1 +
 fs/btrfs/compression.h |    1 +
 fs/btrfs/ctree.h       |    5 +-
 fs/btrfs/lzo.c         |  498 ++++++++++++++++++++++++++++++++++++++++++++++++
 fs/btrfs/super.c       |    6 +
 6 files changed, 510 insertions(+), 3 deletions(-)
 create mode 100644 fs/btrfs/lzo.c
fs/btrfs/Makefile +1 -1
index a35eb36..31610ea 100644
@@ -6,5 +6,5 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \
 	   transaction.o inode.o file.o tree-defrag.o \
 	   extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \
 	   extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \
-	   export.o tree-log.o acl.o free-space-cache.o zlib.o \
+	   export.o tree-log.o acl.o free-space-cache.o zlib.o lzo.o \
 	   compression.o delayed-ref.o relocation.o
fs/btrfs/compression.c +1 -0
index 4c6068b..84c5490 100644
@@ -714,6 +714,7 @@ static wait_queue_head_t comp_workspace_wait[BTRFS_COMPRESS_TYPES];
 
 struct btrfs_compress_op *btrfs_compress_op[] = {
 	&btrfs_zlib_compress,
+	&btrfs_lzo_compress,
 };
 
 int __init btrfs_init_compress(void)
fs/btrfs/compression.h +1 -0
index 9b5f2f3..f7ce217 100644
@@ -73,5 +73,6 @@ struct btrfs_compress_op {
 };
 
 extern struct btrfs_compress_op btrfs_zlib_compress;
+extern struct btrfs_compress_op btrfs_lzo_compress;
 
 #endif
fs/btrfs/ctree.h +3 -2
index b0f8b4d..f9c2b00 100644
@@ -530,8 +530,9 @@ struct btrfs_timespec {
 enum btrfs_compression_type {
 	BTRFS_COMPRESS_NONE  = 0,
 	BTRFS_COMPRESS_ZLIB  = 1,
-	BTRFS_COMPRESS_TYPES = 1,
-	BTRFS_COMPRESS_LAST  = 2,
+	BTRFS_COMPRESS_LZO   = 2,
+	BTRFS_COMPRESS_TYPES = 2,
+	BTRFS_COMPRESS_LAST  = 3,
 };
 
 struct btrfs_inode_item {
fs/btrfs/lzo.c new file +498 -0

498 lines not on this page · raw message

fs/btrfs/super.c +6 -0
index a2e5c79..5670f30 100644
@@ -159,6 +159,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 			    strcmp(args[0].from, "zlib") == 0) {
 				compress_type = "zlib";
 				info->compress_type = BTRFS_COMPRESS_ZLIB;
+			} else if (strcmp(args[0].from, "lzo") == 0) {
+				compress_type = "lzo";
+				info->compress_type = BTRFS_COMPRESS_LZO;
 			} else {
 				ret = -EINVAL;
 				goto out;
@@ -173,6 +176,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options)
 			    strcmp(args[0].from, "zlib") == 0) {
 				compress_type = "zlib";
 				info->compress_type = BTRFS_COMPRESS_ZLIB;
+			} else if (strcmp(args[0].from, "lzo") == 0) {
+				compress_type = "lzo";
+				info->compress_type = BTRFS_COMPRESS_LZO;
 			} else {
 				ret = -EINVAL;
 				goto out;
Signature · 1 line
1.7.0.1
author
#4 lore raw

PATCH 4/5 btrfs: Allow to specify compress method when defrag

In reply to Li Zefan

Update defrag ioctl, so one can choose lzo or zlib when turning on compression in defrag operation.

---
 fs/btrfs/ioctl.c |    6 +++++-
 fs/btrfs/ioctl.h |    9 ++++++++-
 2 files changed, 13 insertions(+), 2 deletions(-)
fs/btrfs/ioctl.c +5 -1
index ecc49ee..2ea05df 100644
@@ -584,8 +584,12 @@ static int btrfs_defrag_file(struct file *file,
 		}
 		total_read++;
 		mutex_lock(&inode->i_mutex);
-		if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS)
+		if (range->flags & BTRFS_DEFRAG_RANGE_COMPRESS) {
 			BTRFS_I(inode)->force_compress = BTRFS_COMPRESS_ZLIB;
+			if (range->compress_type)
+				BTRFS_I(inode)->force_compress =
+					range->compress_type;
+		}
 
 		ret  = btrfs_delalloc_reserve_space(inode, PAGE_CACHE_SIZE);
 		if (ret)
fs/btrfs/ioctl.h +8 -1
index 424694a..776a2b4 100644
@@ -122,8 +122,15 @@ struct btrfs_ioctl_defrag_range_args {
 	 */
 	__u32 extent_thresh;
 
+	/*
+	 * which compression method to use if turning on compression
+	 * for this defrag operation.  If unspecified, zlib will
+	 * be used
+	 */
+	__u32 compress_type;
+
 	/* spare for later */
-	__u32 unused[5];
+	__u32 unused[4];
 };
 
 struct btrfs_ioctl_space_info {
Signature · 1 line
1.7.0.1
author
#5 lore raw

PATCH 5/5 btrfs: Extract duplicate decompress code

In reply to Li Zefan

Add a common function to copy decompressed data from working buffer to bio pages.

---
 fs/btrfs/compression.c |   92 +++++++++++++++++++++++++++++++++++++++
 fs/btrfs/compression.h |    5 ++
 fs/btrfs/lzo.c         |  101 +++-----------------------------------------
 fs/btrfs/zlib.c        |  111 +++++------------------------------------------
 4 files changed, 115 insertions(+), 194 deletions(-)
fs/btrfs/compression.c +92 -0
index 84c5490..c7f695b 100644
@@ -927,3 +927,95 @@ void __exit btrfs_exit_compress(void)
 {
     free_workspaces();
 }
+
+/*
+ * Copy uncompressed data from working buffer to pages.
+ *
+ * buf_start is the byte offset we're of the start of our workspace buffer.
+ *
+ * total_out is the last byte of the buffer
+ */
+int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
+			      unsigned long total_out, u64 disk_start,
+			      struct bio_vec *bvec, int vcnt,
+			      unsigned long *page_index,
+			      unsigned long *pg_offset)
+{
+	unsigned long buf_offset;
+	unsigned long current_buf_start;
+	unsigned long start_byte;
+	unsigned long working_bytes = total_out - buf_start;
+	unsigned long bytes;
+	char *kaddr;
+	struct page *page_out = bvec[*page_index].bv_page;
+
+	/*
+	 * start byte is the first byte of the page we're currently
+	 * copying into relative to the start of the compressed data.
+	 */
+	start_byte = page_offset(page_out) - disk_start;
+
+	/* we haven't yet hit data corresponding to this page */
+	if (total_out <= start_byte)
+		return 1;
+
+	/*
+	 * the start of the data we care about is offset into
+	 * the middle of our working buffer
+	 */
+	if (total_out > start_byte && buf_start < start_byte) {
+		buf_offset = start_byte - buf_start;
+		working_bytes -= buf_offset;
+	} else {
+		buf_offset = 0;
+	}
+	current_buf_start = buf_start;
+
+	/* copy bytes from the working buffer into the pages */
+	while (working_bytes > 0) {
+		bytes = min(PAGE_CACHE_SIZE - *pg_offset,
+			    PAGE_CACHE_SIZE - buf_offset);
+		bytes = min(bytes, working_bytes);
+		kaddr = kmap_atomic(page_out, KM_USER0);
+		memcpy(kaddr + *pg_offset, buf + buf_offset, bytes);
+		kunmap_atomic(kaddr, KM_USER0);
+		flush_dcache_page(page_out);
+
+		*pg_offset += bytes;
+		buf_offset += bytes;
+		working_bytes -= bytes;
+		current_buf_start += bytes;
+
+		/* check if we need to pick another page */
+		if (*pg_offset == PAGE_CACHE_SIZE) {
+			(*page_index)++;
+			if (*page_index >= vcnt)
+				return 0;
+
+			page_out = bvec[*page_index].bv_page;
+			*pg_offset = 0;
+			start_byte = page_offset(page_out) - disk_start;
+
+			/*
+			 * make sure our new page is covered by this
+			 * working buffer
+			 */
+			if (total_out <= start_byte)
+				return 1;
+
+			/*
+			 * the next page in the biovec might not be adjacent
+			 * to the last page, but it might still be found
+			 * inside this working buffer. bump our offset pointer
+			 */
+			if (total_out > start_byte &&
+			    current_buf_start < start_byte) {
+				buf_offset = start_byte - buf_start;
+				working_bytes = total_out - start_byte;
+				current_buf_start = buf_start + buf_offset;
+			}
+		}
+	}
+
+	return 1;
+}
fs/btrfs/compression.h +5 -0
index f7ce217..5100017 100644
@@ -34,6 +34,11 @@ int btrfs_decompress_biovec(int type, struct page **pages_in, u64 disk_start,
 			    struct bio_vec *bvec, int vcnt, size_t srclen);
 int btrfs_decompress(int type, unsigned char *data_in, struct page *dest_page,
 		     unsigned long start_byte, size_t srclen, size_t destlen);
+int btrfs_decompress_buf2page(char *buf, unsigned long buf_start,
+			      unsigned long total_out, u64 disk_start,
+			      struct bio_vec *bvec, int vcnt,
+			      unsigned long *page_index,
+			      unsigned long *pg_offset);
 
 int btrfs_submit_compressed_write(struct inode *inode, u64 start,
 				  unsigned long len, u64 disk_start,
fs/btrfs/lzo.c +6 -95

132 lines not on this page · raw message

fs/btrfs/zlib.c +12 -99

144 lines not on this page · raw message

Signature · 1 line
1.7.0.1
#6 lore raw
In reply to Li Zefan
Quoted 3 lines · Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

This is also much smaller than I expected, really nice. It looks like older kernels won't properly deal (nicely give EIO) with lzo compressed files?

We can add compatbits to deal with that if it is the case.

-chris

author
#7 lore raw
Quoted 11 lines · Chris Mason
Quoted 3 lines · Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

This is also much smaller than I expected, really nice. It looks like older kernels won't properly deal (nicely give EIO) with lzo compressed files?

We can add compatbits to deal with that if it is the case.

I forgot compatibility issue with older kernels..

Though I didn't test older kernels, I don't think they can deal with lzo compressed files properly, at least not for inlined extents, in which case I think btrfs will just show compressed data to the users.

So yes, an incompat flag is needed.

#8 lore raw
Quoted 3 lines · Li Zefan
Quoted 2 lines · Chris Mason
Quoted 1 line · Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allo=

w

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

This is also much smaller than I expected, really nice. =A0It looks =

like

Quoted 1 line
Quoted 1 line

older kernels won't properly deal (nicely give EIO) with lzo compres=

sed

Quoted 8 lines
Quoted 4 lines

files?

We can add compatbits to deal with that if it is the case.

I forgot compatibility issue with older kernels..

Though I didn't test older kernels, I don't think they can deal with =

lzo

Quoted 4 lines

compressed files properly, at least not for inlined extents, in which case I think btrfs will just show compressed data to the users.

So yes, an incompat flag is needed.

I've been testing this set of patches on a 2.6.36 kernel with the latest btrfs-unstable patches, and I wanted to provide my positive feedback.

The lzo compression patches worked as expected, and I did not encounter any stability problems in my testing (which mainly consisted of decompressing a root partition archive and doing some compiling chores in a chroot environment on that drive).

I did some crude benchmarking on decompressing an archived root partition to an empty btrfs drive, and my results were similar to the results posted by Li Zefan in the original post. The lzo compression did not compress quite as well as zlib, but was much faster.

I did some test mounting of a lzo-compressed partition with an older non-lzo-patched kernel, and as expected, the contents of the files were garbled. But I did not encounter kernel oops or other issues when trying to read the garbled files.

I also wanted to confirm that my other zlib-compressed partitions did not encounter any issues when bouncing back and forth between a lzo-patched kernel and older kernels. As expected, I did not see any issues in the partitions that had only been mounted with zlib compression.

You'll want an incompat flag as indicated in previous posts, but I look forward to seeing lzo compression in the future.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#9 lore raw
Quoted 39 lines · Mitch Harder
Quoted 11 lines · Li Zefan
Quoted 10 lines · Chris Mason
Quoted 3 lines · Li Zefan

Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

This is also much smaller than I expected, really nice. It looks like older kernels won't properly deal (nicely give EIO) with lzo compressed files?

We can add compatbits to deal with that if it is the case.

Quoted 7 lines

I forgot compatibility issue with older kernels..

Though I didn't test older kernels, I don't think they can deal with lzo compressed files properly, at least not for inlined extents, in which case I think btrfs will just show compressed data to the users.

So yes, an incompat flag is needed.

I've been testing this set of patches on a 2.6.36 kernel with the latest btrfs-unstable patches, and I wanted to provide my positive feedback.

The lzo compression patches worked as expected, and I did not encounter any stability problems in my testing (which mainly consisted of decompressing a root partition archive and doing some compiling chores in a chroot environment on that drive).

I did some crude benchmarking on decompressing an archived root partition to an empty btrfs drive, and my results were similar to the results posted by Li Zefan in the original post. The lzo compression did not compress quite as well as zlib, but was much faster.

I did some test mounting of a lzo-compressed partition with an older non-lzo-patched kernel, and as expected, the contents of the files were garbled. But I did not encounter kernel oops or other issues when trying to read the garbled files.

Quoted 9 lines

I also wanted to confirm that my other zlib-compressed partitions did not encounter any issues when bouncing back and forth between a lzo-patched kernel and older kernels. As expected, I did not see any issues in the partitions that had only been mounted with zlib compression.

You'll want an incompat flag as indicated in previous posts, but I look forward to seeing lzo compression in the future.

Thanks for testing!

I had been occupied by other things, so have been silent on this. Actually I've added the incompat flag, but haven't done userspace programs.

I'm setting up a git tree, and will send out a pull request to Chris this week. (probabaly)

And I think I can add your Tested-by in the whole patchset? Except for the ioctl change, which I guess you didn't test.

We really need more review and test for btrfs patches, so what you've done is really appreciatated. :)

--
Li Zefan

#10 lore raw
Quoted 2 lines · Li Zefan
Quoted 1 line · Mitch Harder

On Mon, Oct 25, 2010 at 9:34 PM, Li Zefan <[email protected]> wrot=

e:

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines · Chris Mason
Quoted 1 line · Li Zefan

Lzo is a much faster compression algorithm than gzib, so would al=

low

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

more users to enable transparent compression, and some users can choose from compression ratio and compression speed.

This is also much smaller than I expected, really nice. =A0It look=

s like

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

older kernels won't properly deal (nicely give EIO) with lzo compr=

essed

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

files?

We can add compatbits to deal with that if it is the case.

I forgot compatibility issue with older kernels..

Though I didn't test older kernels, I don't think they can deal wit=

h lzo

Quoted 1 line
Quoted 1 line
Quoted 1 line

compressed files properly, at least not for inlined extents, in whi=

ch

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

case I think btrfs will just show compressed data to the users.

So yes, an incompat flag is needed.

I've been testing this set of patches on a 2.6.36 kernel with the latest btrfs-unstable patches, and I wanted to provide my positive feedback.

The lzo compression patches worked as expected, and I did not encounter any stability problems in my testing (which mainly consist=

ed

Quoted 5 lines
Quoted 5 lines

of decompressing a root partition archive and doing some compiling chores in a chroot environment on that drive).

I did some crude benchmarking on decompressing an archived root partition to an empty btrfs drive, and my results were similar to th=

e

Quoted 1 line
Quoted 1 line

results posted by Li Zefan in the original post. =A0The lzo compress=

ion

Quoted 8 lines
Quoted 8 lines

did not compress quite as well as zlib, but was much faster.

I did some test mounting of a lzo-compressed partition with an older non-lzo-patched kernel, and as expected, the contents of the files were garbled. =A0But I did not encounter kernel oops or other issues when trying to read the garbled files.

I also wanted to confirm that my other zlib-compressed partitions di=

d

Quoted 2 lines
Quoted 2 lines

not encounter any issues when bouncing back and forth between a lzo-patched kernel and older kernels. =A0As expected, I did not see =

any

Quoted 13 lines
Quoted 6 lines

issues in the partitions that had only been mounted with zlib compression.

You'll want an incompat flag as indicated in previous posts, but I look forward to seeing lzo compression in the future.

Thanks for testing!

I had been occupied by other things, so have been silent on this. Actually I've added the incompat flag, but haven't done userspace programs.

Quoted 9 lines

I'm setting up a git tree, and will send out a pull request to Chris this week. (probabaly)

And I think I can add your Tested-by in the whole patchset? Except for the ioctl change, which I guess you didn't test.

We really need more review and test for btrfs patches, so what you've done is really appreciatated. :)

Glad to help. Feel free to add "Tested-by Mitch Harder <[email protected]>"

Regarding the "ioctl change", I tested the five patches submitted to the M/L on Oct. 25, 2010. Some minor merging changes were required on patch 2/5 for the enum containing the new lzo options in super.c (due to some other recent modifications to that enum for space_cache option).

So if there was an additional "ioctl change", I did not test it. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html