btrfs: Add lzo compression support
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
git://repo.or.cz/linux-btrfs-devel.git lzo-support
Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Usage:
# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt
"-o compress" without argument is still allowed for compatability.
Compatibility:
If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.
Performance:
The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.
(time in second)
lzo zlib nocompress copy: 10.6 21.7 14.9 extract: 70.1 94.4 66.6
(data size in MB)
lzo zlib nocompress copy: 185.87 108.69 394.49 extract: 193.80 132.36 381.21
Test:
Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.
Other:
The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.
Main change from v1:
- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.
---------------->
Li Zefan (6):
btrfs: Fix bugs in zlib workspace
btrfs: Fix error handling in zlib
btrfs: Allow to add new compression algorithm
btrfs: Add lzo compression support
btrfs: Allow to specify compress method when defrag
btrfs: Extract duplicate decompress code
fs/btrfs/Makefile | 2 +- fs/btrfs/btrfs_inode.h | 2 +- fs/btrfs/compression.c | 329 +++++++++++++++++++++++++++++++++++++- fs/btrfs/compression.h | 72 ++++++-- fs/btrfs/ctree.h | 11 +- fs/btrfs/extent_io.c | 5 +- fs/btrfs/extent_io.h | 17 ++- fs/btrfs/extent_map.c | 2 + fs/btrfs/extent_map.h | 3 +- fs/btrfs/file.c | 2 + fs/btrfs/inode.c | 82 ++++++---- fs/btrfs/ioctl.c | 10 +- fs/btrfs/ioctl.h | 9 +- fs/btrfs/lzo.c | 409 +++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/ordered-data.c | 18 ++- fs/btrfs/ordered-data.h | 8 +- fs/btrfs/super.c | 47 +++++- fs/btrfs/zlib.c | 361 +++++++---------------------------------- 18 files changed, 1013 insertions(+), 376 deletions(-)
PATCH v2 1/6 btrfs: Fix bugs in zlib workspace
- Fix a race that can result in alloc_workspace > cpus.
- Fix to check num_workspace after wakeup.
--- fs/btrfs/zlib.c | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
fs/btrfs/zlib.c +5 -2
@@ -75,16 +75,19 @@ again:return workspace; } -spin_unlock (&workspace_lock);if (atomic_read (&alloc_workspace) > cpus) {DEFINE_WAIT (wait); + +spin_unlock (&workspace_lock);prepare_to_wait (&workspace_wait, &wait,TASK_UNINTERRUPTIBLE ); -if (atomic_read (&alloc_workspace) > cpus) +if (atomic_read (&alloc_workspace) > cpus && !num_workspace)schedule ();finish_wait (&workspace_wait, &wait); goto again; }atomic_inc (&alloc_workspace); +spin_unlock (&workspace_lock); + workspace =kzalloc (sizeof (*workspace),GFP_NOFS );if (!workspace) { ret = -ENOMEM ;
-- 1.7.3.1
PATCH v2 2/6 btrfs: Fix error handling in zlib
In reply to Li ZefanReturn failure if alloc_page() fails to allocate memory, and the upper code will just give up compression.
--- fs/btrfs/zlib.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
fs/btrfs/zlib.c +8 -0
@@ -225,6 +225,10 @@ int btrfs_zlib_compress_pages(struct address_space *mapping, data_in =kmap (in_page); out_page =alloc_page (GFP_NOFS | __GFP_HIGHMEM); +if (out_page ==NULL ) { + ret = -ENOMEM ; + goto out; + } cpage_out =kmap (out_page); pages[0 ] = out_page; nr_pages =1 ;
@@ -263,6 +267,10 @@ int btrfs_zlib_compress_pages(struct address_space *mapping, goto out; } out_page =alloc_page (GFP_NOFS | __GFP_HIGHMEM); +if (out_page ==NULL ) { + ret = -ENOMEM ; + goto out; + } cpage_out =kmap (out_page); pages[nr_pages] = out_page; nr_pages++;
-- 1.7.3.1
PATCH v2 3/6 btrfs: Allow to add new compression algorithm
In reply to Li ZefanMake the code aware of compression type, instead of always assuming zlib compression.
Also make the zlib workspace function as common code for all compression types.
--- fs/btrfs/btrfs_inode.h | 2 +- fs/btrfs/compression.c | 236 ++++++++++++++++++++++++++++++++++++++++++- fs/btrfs/compression.h | 66 +++++++++---- fs/btrfs/ctree.h | 10 +- fs/btrfs/extent_io.c | 5 +- fs/btrfs/extent_io.h | 17 +++- fs/btrfs/extent_map.c | 2 + fs/btrfs/extent_map.h | 3 +- fs/btrfs/file.c | 2 + fs/btrfs/inode.c | 82 ++++++++++------ fs/btrfs/ioctl.c | 4 +- fs/btrfs/ordered-data.c | 18 +++- fs/btrfs/ordered-data.h | 8 ++- fs/btrfs/super.c | 47 +++++++-- fs/btrfs/zlib.c | 253 +++++++++------------------------------------- 15 files changed, 473 insertions(+), 282 deletions(-)
fs/btrfs/btrfs_inode.h +1 -1
@@ -157,7 +157,7 @@ struct btrfs_inode {/* * always compress this one file */ - unsigned force_compress:1; + unsigned force_compress:4;struct inode vfs_inode; };
fs/btrfs/compression.c +231 -5
257 lines not on this page · raw message
fs/btrfs/compression.h +48 -18
76 lines not on this page · raw message
fs/btrfs/ctree.h +6 -4
@@ -551,9 +551,10 @@ struct btrfs_timespec { } __attribute__ ((__packed__));enum btrfs_compression_type { -BTRFS_COMPRESS_NONE =0 , -BTRFS_COMPRESS_ZLIB =1 , -BTRFS_COMPRESS_LAST =2 , +BTRFS_COMPRESS_NONE =0 , +BTRFS_COMPRESS_ZLIB =1 , +BTRFS_COMPRESS_TYPES =1 , +BTRFS_COMPRESS_LAST =2 , };struct btrfs_inode_item {
@@ -895,7 +896,8 @@ struct btrfs_fs_info {*/ u64 last_trans_log_full_commit;u64 open_ioctl_trans; -unsigned long mount_opt; + unsigned long mount_opt:20; + unsigned long compress_type:4;u64 max_inline;u64 alloc_start;struct btrfs_transaction *running_transaction;
fs/btrfs/extent_io.c +4 -1
@@ -2028,8 +2028,11 @@ static int __extent_read_full_page(struct extent_io_tree *tree,BUG_ON (extent_map_end (em) <= cur);BUG_ON (end < cur); -if (test_bit (EXTENT_FLAG_COMPRESSED , &em->flags)) +if (test_bit (EXTENT_FLAG_COMPRESSED , &em->flags)) { this_bio_flag =EXTENT_BIO_COMPRESSED ; +extent_set_compress_type (&this_bio_flag, + em->compress_type); + } iosize =min (extent_map_end (em) - cur, end - cur +1 ); cur_end =min (extent_map_end (em) -1 , end);
fs/btrfs/extent_io.h +16 -1
@@ -20,8 +20,12 @@#define EXTENT_IOBITS (EXTENT_LOCKED | EXTENT_WRITEBACK)#define EXTENT_CTLBITS (EXTENT_DO_ACCOUNTING | EXTENT_FIRST_DELALLOC) -/* flags for bio submission */ +/* +* flags for bio submission. The high bits indicate the compression +* type for this bio +*/ #define EXTENT_BIO_COMPRESSED 1 +#define EXTENT_BIO_FLAG_SHIFT 16/* these are bit numbers for test/set bit */ #define EXTENT_BUFFER_UPTODATE 0
@@ -135,6 +139,17 @@ struct extent_buffer {wait_queue_head_t lock_wq; }; +static inline void extent_set_compress_type (unsigned long *bio_flags, +int compress_type) +{ + *bio_flags |= compress_type <<EXTENT_BIO_FLAG_SHIFT ; +} + +static inline int extent_compress_type (unsigned long bio_flags) +{ +return bio_flags >>EXTENT_BIO_FLAG_SHIFT ; +} +struct extent_map_tree ;static inline struct extent_state *extent_state_next (struct extent_state *state)
fs/btrfs/extent_map.c +2 -0
@@ -3,6 +3,7 @@#include <linux/module.h> #include <linux/spinlock.h> #include <linux/hardirq.h> +#include "ctree.h" #include "extent_map.h"
@@ -54,6 +55,7 @@ struct extent_map *alloc_extent_map(gfp_t mask)return em; em->in_tree =0 ; em->flags =0 ; + em->compress_type =BTRFS_COMPRESS_NONE ;atomic_set (&em->refs,1 );return em; }
fs/btrfs/extent_map.h +2 -1
@@ -26,7 +26,8 @@ struct extent_map {unsigned long flags;struct block_device *bdev;atomic_t refs; -int in_tree; + unsigned int in_tree:1; + unsigned int compress_type:4; };struct extent_map_tree {
fs/btrfs/file.c +2 -0
@@ -220,6 +220,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end, split->bdev = em->bdev; split->flags = flags; + split->compress_type = em->compress_type; ret =add_extent_mapping (em_tree, split);BUG_ON (ret);free_extent_map (split);
@@ -234,6 +235,7 @@ int btrfs_drop_extent_cache(struct inode *inode, u64 start, u64 end, split->len = em->start + em->len - (start + len); split->bdev = em->bdev; split->flags = flags; + split->compress_type = em->compress_type;if (compressed) { split->block_len = em->block_len;
fs/btrfs/inode.c +51 -31
229 lines not on this page · raw message
fs/btrfs/ioctl.c +2 -2
@@ -671,7 +671,7 @@ static int btrfs_defrag_file(struct file *file, total_read++;mutex_lock (&inode->i_mutex);if (range->flags &BTRFS_DEFRAG_RANGE_COMPRESS ) -BTRFS_I (inode)->force_compress =1 ; +BTRFS_I (inode)->force_compress =BTRFS_COMPRESS_ZLIB ; ret =btrfs_delalloc_reserve_space (inode,PAGE_CACHE_SIZE );if (ret)
@@ -769,7 +769,7 @@ loop_unlock:atomic_dec (&root->fs_info->async_submit_draining);mutex_lock (&inode->i_mutex); -BTRFS_I (inode)->force_compress =0 ; +BTRFS_I (inode)->force_compress =BTRFS_COMPRESS_NONE ;mutex_unlock (&inode->i_mutex); }
fs/btrfs/ordered-data.c +15 -3
@@ -172,7 +172,7 @@ static inline struct rb_node *tree_search(struct btrfs_ordered_inode_tree *tree,*/ static int __btrfs_add_ordered_extent (struct inode *inode,u64 file_offset,u64 start,u64 len,u64 disk_len, -int type,int dio) +int type,int dio,int compress_type) {struct btrfs_ordered_inode_tree *tree;struct rb_node *node;
@@ -189,6 +189,7 @@ static int __btrfs_add_ordered_extent(struct inode *inode, u64 file_offset, entry->disk_len = disk_len; entry->bytes_left = len; entry->inode = inode; + entry->compress_type = compress_type;if (type !=BTRFS_ORDERED_IO_DONE && type !=BTRFS_ORDERED_COMPLETE )set_bit (type, &entry->flags);
@@ -220,14 +221,25 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset, u64 start, u64 len, u64 disk_len, int type) { return __btrfs_add_ordered_extent(inode, file_offset, start, len, - disk_len, type, 0); + disk_len, type, 0, +BTRFS_COMPRESS_NONE ); }int btrfs_add_ordered_extent_dio (struct inode *inode,u64 file_offset,u64 start,u64 len,u64 disk_len,int type) {return __btrfs_add_ordered_extent (inode, file_offset, start, len, - disk_len, type,1 ); + disk_len, type,1 , +BTRFS_COMPRESS_NONE ); +} + +int btrfs_add_ordered_extent_compress (struct inode *inode,u64 file_offset, +u64 start,u64 len,u64 disk_len, +int type,int compress_type) +{ +return __btrfs_add_ordered_extent (inode, file_offset, start, len, + disk_len, type,0 , + compress_type); }/*
fs/btrfs/ordered-data.h +7 -1
@@ -68,7 +68,7 @@ struct btrfs_ordered_sum {#define BTRFS_ORDERED_NOCOW 2/* set when we want to write in place */ -#define BTRFS_ORDERED_COMPRESSED 3/* writing a compressed extent */ +#define BTRFS_ORDERED_COMPRESSED 3/* writing a zlib compressed extent */ #define BTRFS_ORDERED_PREALLOC 4/* set when writing to prealloced extent */
@@ -93,6 +93,9 @@ struct btrfs_ordered_extent {/* flags (described above) */ unsigned long flags; +/* compression algorithm */ +int compress_type; +/* reference count */ atomic_t refs;
@@ -145,6 +148,9 @@ int btrfs_add_ordered_extent(struct inode *inode, u64 file_offset, u64 start, u64 len, u64 disk_len, int type);int btrfs_add_ordered_extent_dio (struct inode *inode,u64 file_offset,u64 start,u64 len,u64 disk_len,int type); +int btrfs_add_ordered_extent_compress (struct inode *inode,u64 file_offset, +u64 start,u64 len,u64 disk_len, +int type,int compress_type);int btrfs_add_ordered_sum (struct inode *inode,struct btrfs_ordered_extent *entry,struct btrfs_ordered_sum *sum);
fs/btrfs/super.c +36 -11
95 lines not on this page · raw message
fs/btrfs/zlib.c +50 -203
338 lines not on this page · raw message
-- 1.7.3.1
PATCH v2 4/6 btrfs: Add lzo compression support
In reply to Li ZefanLzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and speed for different applications
Usage:
# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt
"-o compress" without argument is still allowed for compatability.
Compatibility:
If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.
Performance:
The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted it.
(time in second)
lzo zlib nocompress copy: 10.6 21.7 14.9 extract: 70.1 94.4 66.6
(data size in MB)
lzo zlib nocompress copy: 185.87 108.69 394.49 extract: 193.80 132.36 381.21
Changelog:
v1 -> v2:
- Select LZO_COMPRESS and LZO_DECOMPRESS in btrfs Kconfig.
- Add incompability flag.
- Fix error handling in compress code.
--- fs/btrfs/Kconfig | 2 + fs/btrfs/Makefile | 2 +- fs/btrfs/compression.c | 1 + fs/btrfs/compression.h | 1 + fs/btrfs/ctree.h | 9 +- fs/btrfs/disk-io.c | 8 +- fs/btrfs/lzo.c | 509 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/super.c | 3 + 8 files changed, 527 insertions(+), 8 deletions(-) create mode 100644 fs/btrfs/lzo.c
fs/btrfs/Kconfig +2 -0
@@ -4,6 +4,8 @@ config BTRFS_FS select LIBCRC32C select ZLIB_INFLATE select ZLIB_DEFLATE + select LZO_COMPRESS + select LZO_DECOMPRESS help Btrfs is a new filesystem with extents, writable snapshotting, support for multiple devices and many more features.
fs/btrfs/Makefile +1 -1
@@ -6,5 +6,5 @@ btrfs-y += super.o ctree.o extent-tree.o print-tree.o root-tree.o dir-item.o \ transaction.o inode.o file.o tree-defrag.o \ extent_map.o sysfs.o struct-funcs.o xattr.o ordered-data.o \ extent_io.o volumes.o async-thread.o ioctl.o locking.o orphan.o \ - export.o tree-log.o acl.o free-space-cache.o zlib.o \ + export.o tree-log.o acl.o free-space-cache.o zlib.o lzo.o \ compression.o delayed-ref.o relocation.o
fs/btrfs/compression.c +1 -0
@@ -704,6 +704,7 @@ static wait_queue_head_t comp_workspace_wait[BTRFS_COMPRESS_TYPES];struct btrfs_compress_op *btrfs_compress_op[] = { &btrfs_zlib_compress, + &btrfs_lzo_compress, };int __init btrfs_init_compress (void )
fs/btrfs/compression.h +1 -0
@@ -73,5 +73,6 @@ struct btrfs_compress_op { };extern struct btrfs_compress_op btrfs_zlib_compress; +extern struct btrfs_compress_op btrfs_lzo_compress;#endif
fs/btrfs/ctree.h +6 -3
@@ -398,13 +398,15 @@ struct btrfs_super_block {#define BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF (1ULL << 0)#define BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL (1ULL << 1)#define BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS (1ULL << 2) +#define BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO (1ULL << 3)#define BTRFS_FEATURE_COMPAT_SUPP 0ULL#define BTRFS_FEATURE_COMPAT_RO_SUPP 0ULL#define BTRFS_FEATURE_INCOMPAT_SUPP \ (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF | \ BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL | \ - BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS) + BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS | \ + BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO)/* * A leaf is full of items. offset and size tell us where to find
@@ -553,8 +555,9 @@ struct btrfs_timespec {enum btrfs_compression_type {BTRFS_COMPRESS_NONE =0 ,BTRFS_COMPRESS_ZLIB =1 , -BTRFS_COMPRESS_TYPES =1 , -BTRFS_COMPRESS_LAST =2 , +BTRFS_COMPRESS_LZO =2 , +BTRFS_COMPRESS_TYPES =2 , +BTRFS_COMPRESS_LAST =3 , };struct btrfs_inode_item {
fs/btrfs/disk-io.c +4 -4
@@ -1717,10 +1717,10 @@ struct btrfs_root *open_ctree(struct super_block *sb, } features =btrfs_super_incompat_flags (disk_super); -if (!(features &BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF )) { - features |=BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF ; -btrfs_set_super_incompat_flags (disk_super, features); - } + features |=BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF ; +if (tree_root->fs_info->compress_type &BTRFS_COMPRESS_LZO ) + features |=BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO ; +btrfs_set_super_incompat_flags (disk_super, features); features =btrfs_super_compat_ro_flags (disk_super) & ~BTRFS_FEATURE_COMPAT_RO_SUPP ;
fs/btrfs/lzo.c new file +509 -0
509 lines not on this page · raw message
fs/btrfs/super.c +3 -0
@@ -168,6 +168,9 @@ int btrfs_parse_options(struct btrfs_root *root, char *options) strcmp(args[0].from, "zlib") == 0) { compress_type ="zlib" ; info->compress_type =BTRFS_COMPRESS_ZLIB ; + } else if (strcmp(args[0].from, "lzo") == 0) { + compress_type = "lzo"; + info->compress_type = BTRFS_COMPRESS_LZO; } else { ret = -EINVAL; goto out;
-- 1.7.3.1
PATCH v2 5/6 btrfs: Allow to specify compress method when defrag
In reply to Li ZefanUpdate defrag ioctl, so one can choose lzo or zlib when turning on compression in defrag operation.
Changelog:
v1 -> v2
- Add incompability flag.
- Fix to check invalid compress type.
--- fs/btrfs/ioctl.c | 24 +++++++++++++++++++++++- fs/btrfs/ioctl.h | 9 ++++++++- 2 files changed, 31 insertions(+), 2 deletions(-)
fs/btrfs/ioctl.c +23 -1
@@ -626,9 +626,11 @@ static int btrfs_defrag_file(struct file *file,struct extent_io_tree *io_tree = &BTRFS_I (inode)->io_tree;struct btrfs_ordered_extent *ordered;struct page *page; +struct btrfs_super_block *disk_super;unsigned long last_index;unsigned long ra_pages = root->fs_info->bdi.ra_pages;unsigned long total_read =0 ; +u64 features;u64 page_start;u64 page_end;u64 last_len =0 ;
@@ -636,6 +638,18 @@ static int btrfs_defrag_file(struct file *file,u64 defrag_end =0 ;unsigned long i;int ret; +int compress_type =BTRFS_COMPRESS_ZLIB ; + +if (range->compress_type >BTRFS_COMPRESS_TYPES ) +return -EINVAL ; + +/* +* Compression will be enabled only if BTRFS_DEFRAG_RANGE_COMPRESS +* is set, and the default is zlib if compression method is not +* specified. +*/ +if (range->compress_type) + compress_type = range->compress_type;if (inode->i_size ==0 )return 0 ;
@@ -671,7 +685,7 @@ static int btrfs_defrag_file(struct file *file, total_read++;mutex_lock (&inode->i_mutex);if (range->flags &BTRFS_DEFRAG_RANGE_COMPRESS ) -BTRFS_I (inode)->force_compress =BTRFS_COMPRESS_ZLIB ; +BTRFS_I (inode)->force_compress = compress_type; ret =btrfs_delalloc_reserve_space (inode,PAGE_CACHE_SIZE );if (ret)
@@ -773,6 +787,14 @@ loop_unlock:mutex_unlock (&inode->i_mutex); } + disk_super = &root->fs_info->super_copy; + features =btrfs_super_incompat_flags (disk_super); +if (range->flags &BTRFS_DEFRAG_RANGE_COMPRESS && + compress_type ==BTRFS_COMPRESS_LZO ) { + features |=BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO ; +btrfs_set_super_incompat_flags (disk_super, features); + } +return 0 ; err_reservations:
fs/btrfs/ioctl.h +8 -1
@@ -129,8 +129,15 @@ struct btrfs_ioctl_defrag_range_args {*/ __u32 extent_thresh; +/* +* which compression method to use if turning on compression +* for this defrag operation. If unspecified, zlib will +* be used +*/ +__u32 compress_type; +/* spare for later */ -__u32 unused[5 ]; +__u32 unused[4 ]; };struct btrfs_ioctl_space_info {
-- 1.7.3.1
PATCH v2 6/6 btrfs: Extract duplicate decompress code
In reply to Li ZefanAdd a common function to copy decompressed data from working buffer to bio pages.
--- fs/btrfs/compression.c | 92 +++++++++++++++++++++++++++++++++++++++ fs/btrfs/compression.h | 5 ++ fs/btrfs/lzo.c | 101 +++----------------------------------------- fs/btrfs/zlib.c | 111 +++++------------------------------------------ 4 files changed, 115 insertions(+), 194 deletions(-)
fs/btrfs/compression.c +92 -0
@@ -917,3 +917,95 @@ void __exit btrfs_exit_compress(void) {free_workspaces (); } + +/* +* Copy uncompressed data from working buffer to pages. +* +* buf_start is the byte offset we're of the start of our workspace buffer. +* +* total_out is the last byte of the buffer +*/ +int btrfs_decompress_buf2page (char *buf,unsigned long buf_start, +unsigned long total_out,u64 disk_start, +struct bio_vec *bvec,int vcnt, +unsigned long *page_index, +unsigned long *pg_offset) +{ +unsigned long buf_offset; +unsigned long current_buf_start; +unsigned long start_byte; +unsigned long working_bytes = total_out - buf_start; +unsigned long bytes; +char *kaddr; +struct page *page_out = bvec[*page_index].bv_page; + +/* +* start byte is the first byte of the page we're currently +* copying into relative to the start of the compressed data. +*/ + start_byte =page_offset (page_out) - disk_start; + +/* we haven't yet hit data corresponding to this page */ +if (total_out <= start_byte) +return 1 ; + +/* +* the start of the data we care about is offset into +* the middle of our working buffer +*/ +if (total_out > start_byte && buf_start < start_byte) { + buf_offset = start_byte - buf_start; + working_bytes -= buf_offset; + }else { + buf_offset =0 ; + } + current_buf_start = buf_start; + +/* copy bytes from the working buffer into the pages */ +while (working_bytes >0 ) { + bytes =min (PAGE_CACHE_SIZE - *pg_offset, +PAGE_CACHE_SIZE - buf_offset); + bytes =min (bytes, working_bytes); + kaddr =kmap_atomic (page_out,KM_USER0 ); +memcpy (kaddr + *pg_offset, buf + buf_offset, bytes); +kunmap_atomic (kaddr,KM_USER0 ); +flush_dcache_page (page_out); + + *pg_offset += bytes; + buf_offset += bytes; + working_bytes -= bytes; + current_buf_start += bytes; + +/* check if we need to pick another page */ +if (*pg_offset ==PAGE_CACHE_SIZE ) { + (*page_index)++; +if (*page_index >= vcnt) +return 0 ; + + page_out = bvec[*page_index].bv_page; + *pg_offset =0 ; + start_byte =page_offset (page_out) - disk_start; + +/* +* make sure our new page is covered by this +* working buffer +*/ +if (total_out <= start_byte) +return 1 ; + +/* +* the next page in the biovec might not be adjacent +* to the last page, but it might still be found +* inside this working buffer. bump our offset pointer +*/ +if (total_out > start_byte && + current_buf_start < start_byte) { + buf_offset = start_byte - buf_start; + working_bytes = total_out - start_byte; + current_buf_start = buf_start + buf_offset; + } + } + } + +return 1 ; +}
fs/btrfs/compression.h +5 -0
@@ -34,6 +34,11 @@ int btrfs_decompress_biovec(int type, struct page **pages_in, u64 disk_start, struct bio_vec *bvec, int vcnt, size_t srclen);int btrfs_decompress (int type,unsigned char *data_in,struct page *dest_page,unsigned long start_byte,size_t srclen,size_t destlen); +int btrfs_decompress_buf2page (char *buf,unsigned long buf_start, +unsigned long total_out,u64 disk_start, +struct bio_vec *bvec,int vcnt, +unsigned long *page_index, +unsigned long *pg_offset); int btrfs_submit_compressed_write(struct inode *inode, u64 start, unsigned long len, u64 disk_start,
fs/btrfs/lzo.c +6 -95
132 lines not on this page · raw message
fs/btrfs/zlib.c +12 -99
144 lines not on this page · raw message
-- 1.7.3.1
Quoted 70 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support
Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Usage:
# mount -t btrfs -o compress[=3D<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=3D<zlib,lzo>] dev /mnt"-o compress" without argument is still allowed for compatability.
Compatibility:
If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.
Performance:
The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.
(time in second)
=A0 =A0 =A0 =A0 =A0 lzo =A0 =A0 =A0 =A0zlib =A0 =A0 =A0 =A0nocompress
copy: =A0 =A0 =A010.6 =A0 =A0 =A0 21.7 =A0 =A0 =A0 =A014.9
extract: =A0 70.1 =A0 =A0 =A0 94.4 =A0 =A0 =A0 =A066.6(data size in MB)
=A0 =A0 =A0 =A0 =A0 lzo =A0 =A0 =A0 =A0zlib =A0 =A0 =A0 =A0nocompress
copy: =A0 =A0 =A0185.87 =A0 =A0 108.69 =A0 =A0 =A0394.49
extract: =A0 193.80 =A0 =A0 132.36 =A0 =A0 =A0381.21Test:
Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.
Other:
The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.
Main change from v1:
- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.---------------->
Li Zefan (6):
=A0 =A0 =A0btrfs: Fix bugs in zlib workspace
=A0 =A0 =A0btrfs: Fix error handling in zlib
=A0 =A0 =A0btrfs: Allow to add new compression algorithm =A0 =A0 =A0btrfs: Add lzo compression support
=A0 =A0 =A0btrfs: Allow to specify compress method when defrag =A0 =A0 =A0btrfs: Extract duplicate decompress code
Quoted 3 lines
=A0fs/btrfs/Makefile =A0 =A0 =A0 | =A0 =A02 +- =A0fs/btrfs/btrfs_inode.h =A0| =A0 =A02 +- =A0fs/btrfs/compression.c =A0| =A0329 +++++++++++++++++++++++++++++++=
++++++-
Quoted 11 lines
=A0fs/btrfs/compression.h =A0| =A0 72 ++++++--
=A0fs/btrfs/ctree.h =A0 =A0 =A0 =A0| =A0 11 +-
=A0fs/btrfs/extent_io.c =A0 =A0| =A0 =A05 +-
=A0fs/btrfs/extent_io.h =A0 =A0| =A0 17 ++-
=A0fs/btrfs/extent_map.c =A0 | =A0 =A02 +
=A0fs/btrfs/extent_map.h =A0 | =A0 =A03 +-=A0fs/btrfs/file.c =A0 =A0 =A0 =A0 | =A0 =A02 + =A0fs/btrfs/inode.c =A0 =A0 =A0 =A0| =A0 82 ++++++---- =A0fs/btrfs/ioctl.c =A0 =A0 =A0 =A0| =A0 10 +- =A0fs/btrfs/ioctl.h =A0 =A0 =A0 =A0| =A0 =A09 +-=A0fs/btrfs/lzo.c =A0 =A0 =A0 =A0 =A0| =A0409 +++++++++++++++++++++++=
++++++++++++++++++++++++
Quoted 4 lines
=A0fs/btrfs/ordered-data.c | =A0 18 ++- =A0fs/btrfs/ordered-data.h | =A0 =A08 +- =A0fs/btrfs/super.c =A0 =A0 =A0 =A0| =A0 47 +++++- =A0fs/btrfs/zlib.c =A0 =A0 =A0 =A0 | =A0361 +++++++------------------=
----------------
Quoted 1 line
=A018 files changed, 1013 insertions(+), 376 deletions(-)
is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.
what's the proper place (repo/branch) to see what is pending?
C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 91 lines · C Anthony Risinger
Quoted 81 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
git://repo.or.cz/linux-btrfs-devel.gitlzo-supportLzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Usage:
# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt"-o compress" without argument is still allowed for compatability.
Compatibility:
If we mount a filesystem with lzo compression, it will not be able be mounted in old kernels. One reason is, otherwise btrfs will directly dump compressed data, which sits in inline extent, to user.
Performance:
The test copied a linux source tarball (~400M) from an ext4 partition to the btrfs partition, and then extracted the tarball.
(time in second)
lzo zlib nocompresscopy: 10.6 21.7 14.9 extract: 70.1 94.4 66.6(data size in MB)
lzo zlib nocompresscopy: 185.87 108.69 394.49 extract: 193.80 132.36 381.21Test:
Mitch has tested the patchset, and provided some positive feedback. According to him, the patchset works as expected and nothing bad has he experienced.
Other:
The defrag ioctl is also updated, so one can choose lzo or zlib when turning on compression in defrag operation.
Main change from v1:
- Add incompat flag.
- Fix build issue by selecting kernel lzo module.
- Check compression type in defrag ioctl.---------------->
Li Zefan (6):
btrfs: Fix bugs in zlib workspace
btrfs: Fix error handling in zlib
btrfs: Allow to add new compression algorithm
btrfs: Add lzo compression support
btrfs: Allow to specify compress method when defrag
btrfs: Extract duplicate decompress codefs/btrfs/Makefile | 2 +- fs/btrfs/btrfs_inode.h | 2 +- fs/btrfs/compression.c | 329 +++++++++++++++++++++++++++++++++++++- fs/btrfs/compression.h | 72 ++++++-- fs/btrfs/ctree.h | 11 +- fs/btrfs/extent_io.c | 5 +- fs/btrfs/extent_io.h | 17 ++- fs/btrfs/extent_map.c | 2 + fs/btrfs/extent_map.h | 3 +- fs/btrfs/file.c | 2 + fs/btrfs/inode.c | 82 ++++++----Quoted 8 lines
fs/btrfs/ioctl.c | 10 +- fs/btrfs/ioctl.h | 9 +- fs/btrfs/lzo.c | 409 +++++++++++++++++++++++++++++++++++++++++++++++ fs/btrfs/ordered-data.c | 18 ++- fs/btrfs/ordered-data.h | 8 +- fs/btrfs/super.c | 47 +++++- fs/btrfs/zlib.c | 361 +++++++---------------------------------- 18 files changed, 1013 insertions(+), 376 deletions(-)
Quoted 5 lines
is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.
what's the proper place (repo/branch) to see what is pending?
As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.
Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.
Quoted 2 lines · Li Zefan
Quoted 1 line · C Anthony Risinger
On Wed, Nov 17, 2010 at 8:08 PM, Li Zefan <[email protected]> wrot=
e:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
You can pull from:
=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support
Lzo is a much faster compression algorithm than gzib, so would allo=
w
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
is this in a branch somewhere? =A0or for inclusion in .37/.38? =A0th=
is is
Quoted 6 lines
Quoted 3 lines
a very attractive feature.
what's the proper place (repo/branch) to see what is pending?
As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.
:-(
yeah i thought it was already too late for .37, but i'd love to see this for .38... should add even more kick to my eee s101 netbook.
Quoted 3 lines
Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.
i knew about Chris' tree, i probably should have been more clear; instead of "pending" i should have said "awaiting review", ie. is there a repo Chris or someone else keeps with all the potential patches (such as this, or the recent readonly ones, etc.)?
i keep a pretty good eye on the list, but i thought maybe someone already had a nice central point to see these "up and coming" patches that aren't necessarily "pending" for mainline yet.
thanks Li,
C Anthony
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 23 lines · C Anthony Risinger
Quoted 10 lines · Li Zefan
Quoted 9 lines · C Anthony Risinger
Quoted 8 lines · Li Zefan
You can pull from:
git://repo.or.cz/linux-btrfs-devel.gitlzo-supportLzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Quoted 6 lines
Quoted 4 lines
is this in a branch somewhere? or for inclusion in .37/.38? this is a very attractive feature.
what's the proper place (repo/branch) to see what is pending?
As a new feature, it's too late for .37. Hope to see it merged in .38 merge window.
:-(
yeah i thought it was already too late for .37, but i'd love to see this for .38... should add even more kick to my eee s101 netbook.
Quoted 9 lines
Quoted 3 lines
Chris' btrfs-unstable git tree is the "official" place to see what is pending, but just he hasn't picked up those patches, so for now it only sits in my own tree.
i knew about Chris' tree, i probably should have been more clear; instead of "pending" i should have said "awaiting review", ie. is there a repo Chris or someone else keeps with all the potential patches (such as this, or the recent readonly ones, etc.)?
Unfortunately no..
Quoted 4 lines
i keep a pretty good eye on the list, but i thought maybe someone already had a nice central point to see these "up and coming" patches that aren't necessarily "pending" for mainline yet.
But btrfs-unstable tree should have acted as this role. We put those patches that are ready for next release in different branches in that tree, and merge them into master branch later. And add that tree into linux-next for wider testing..
Quoted 15 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
=A0 =A0 =A0 =A0git://repo.or.cz/linux-btrfs-devel.git lzo-support
Lzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Quoted 7 lines
Usage:
# mount -t btrfs -o compress[=3D<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=3D<zlib,lzo>] dev /mnt"-o compress" without argument is still allowed for compatability.
This doesn't seem to work with -o remount. "mount / -o remount,compress=3Dlzo" fails with "[90192.345382] btrfs: unrecognized mount option 'compress=3Dlzo'". The -clzo option to defrag works fine. --
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" = in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 22 lines · cwillu
Quoted 21 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
git://repo.or.cz/linux-btrfs-devel.gitlzo-supportLzo is a much faster compression algorithm than gzib, so would allow more users to enable transparent compression, and some users can choose from compression ratio and compression speed.
Usage:
# mount -t btrfs -o compress[=<zlib,lzo>] dev /mnt or # mount -t btrfs -o compress-force[=<zlib,lzo>] dev /mnt
Quoted 6 lines
Quoted 1 line
"-o compress" without argument is still allowed for compatability.
This doesn't seem to work with -o remount. "mount / -o remount,compress=lzo" fails with "[90192.345382] btrfs: unrecognized mount option 'compress=lzo'". The -clzo option to defrag works fine.
It works for me:
# mount -t btrfs -o compress=zlib /dev/loop1 /mnt # mount -o remount,compress=lzo /dev/loop1 /mnt # dmesg ... device fsid c142fef10429f78f-540f99158993f7a2 devid 1 transid 31 /dev/loop1 btrfs: use zlib compression btrfs: use lzo compression
Excerpts from Li Zefan's message of 2010-12-24 01:55:53 -0500:
Quoted 11 lines
Quoted 10 lines · cwillu
Quoted 9 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
git://repo.or.cz/linux-btrfs-devel.gitlzo-support
I pulled this in last night and an all night run completed without troubles. I'm trying a few more this morning, but it looks really good.
I think I'll also bump the compressed extent size up to 512KB, which will cut down on the number of extents in files that compress well.
-chris
Excerpts from Chris Mason's message of 2010-12-24 07:32:16 -0500:
Quoted 11 lines
Excerpts from Li Zefan's message of 2010-12-24 01:55:53 -0500:
Quoted 10 lines
Quoted 9 lines · cwillu
Quoted 8 lines · Li Zefan
Hi Chris,
Here's the updated patchset. As I still haven't got a kernel.org account, I have set up a git tree in another public git repository, and I'll use it for now.
You can pull from:
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
git://repo.or.cz/linux-btrfs-devel.gitlzo-supportI pulled this in last night and an all night run completed without troubles. I'm trying a few more this morning, but it looks really good.
I think I'll also bump the compressed extent size up to 512KB, which will cut down on the number of extents in files that compress well.
The next branch of the btrfs unstable tree has lzo and the other fujitsu work so far. This is going to be the building block for our 2.6.38 pull request to Linus.
It might get rebased, but it seems good so far.
-chris
Quoted 1 line · Chris Mason
This is going to be the building block > for our 2.6.38 pull
request to Linus.
hooray christmas came early this year! what a nice gift, thanks!
heh, thanks for all the great work guys. I'm doing my best to spread the good btrfs word all over the net, correcting misconceptions, writing guides/hooks/scripts, and generally helping those that need it; the majority are most impressed, and looking forward to the onset.
Just thought I'd say that my travels have found many satisfied users, who are, including myself, very appreciative of the work being done here.
So thanks! Happy holidays! And to all a good night!
C Anthony [mobile]