staging: erofs: decompression inplace approach
v3 cover staging: erofs: decompression inplace approach
This is patch v3 of erofs decompression inplace approach, which is sent out by my personal email since I'm out of office to attend Open Source Summit China 2019 these days. No major change from PATCH v2 since no noticeable issue raised from landing to our products till now, mainly as a response to Chao's suggestions.
See the bottom lines which are taken from RFC PATCH v1 and describe the principle of these technologies.
The series is based on the latest staging-next since all dependencies have already been merged.
changelog from v2:
- wrap up some offsets as marcos;
- add some error handling for erofs_get_pcpubuf();
- move some decompression inplace stuffes from PATCH 5 -> 6.
changelog from v1:
- keep Z_EROFS_NR_INLINE_PAGEVECS in unzip_vle.h after switching to new decompression backend;
- add some DBG_BUGONs in new decompression backend to observe potential issues;
- minor code cleanup.
8<--------
Hi,
After working on for more than half a year, the detail of erofs decompression inplace is almost determined and ready for linux-next.
Currently, inplace IO is used if the whole compressed data is used in order to reduce compressed pages extra memory overhead and an extra memcpy (the only one memcpy) will be used for each inplace IO since temporary buffer is needed to keep decompressing safe for inplace IO.
However, most of lz-based decompression algorithms support decompression inplace by their algorithm designs, such as LZ4, LZO, etc.
If iend - oend margin is large enough, decompression inplace can be done in the same buffer safely, as illustrated below:
start of compressed logical extent
| end of this logical extent
| |
______v___________________________v________
... | page 6 | page 7 | page 8 | page 9 | ...
|__________|__________|__________|__________|
. ^ . ^
. |compressed|
. | data |
. . .
|< dstsize >|<margin>|
oend iendop ip
Fixed-size output compression can make the full use of this feature to reduce memory overhead and avoid extra memcpy compared with fixed-size input compression since iend is strictly not less than oend for fixed-size output compression with inplace IO to last pages.
In addition, erofs compression indexes have been improved as well by introducing compacted compression indexes.
These two techniques all benefit sequential read (on x86_64, 710.8MiB/s -> 755.4MiB/s; on Kirin980, 725MiB/s -> 812MiB/s) therefore erofs could have similar sequential read performance against ext4 in a larger CR range on high-spend SSD / NVMe devices as well.
However, note that it is _cpu vs storage device_ tradeoff, there is no absolute performance conclusion for all on-market combinations.
Test images:
name size CR enwik9 1000000000 1.00 enwik9_4k.squashfs.img 621211648 1.61 enwik9_4k.erofs.img 558133248 1.79 enwik9_8k.squashfs.img 556191744 1.80 enwik9_16k.squashfs.img 502661120 1.99 enwik9_128k.squashfs.img 398204928 2.51
Test Environment:
CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz (4 cores, 8 threads)
DDR: 8G
SSD: INTEL SSDPEKKF360G7H
Kernel: Linux 5.2-rc3+ (with lz4-1.8.3 algorithm)
Test configuration:
squashfs:CONFIG_SQUASHFS=yCONFIG_SQUASHFS_FILE_DIRECT=yCONFIG_SQUASHFS_DECOMP_MULTI_PERCPU=yCONFIG_SQUASHFS_LZ4=yCONFIG_SQUASHFS_4K_DEVBLK_SIZE=y
erofs:CONFIG_EROFS_FS_USE_VM_MAP_RAM=yCONFIG_EROFS_FS_ZIP=yCONFIG_EROFS_FS_CLUSTER_PAGE_LIMIT=1CONFIG_EROFS_FS_ZIP_CACHE_BIPOLAR=y
with intel_pstate=disable,
8 cpus on at 1801000 scaling_{min,max}_freq,
userspace scaling_governorSequential read results (MiB/s):
1 2 3 4 5 avg enwik9_4k.ext4.img 767 770 738 726 724 745 enwik9_4k.erofs.img 756 745 770 746 760 755.4 enwik9_4k.squashfs.img 90.3 83.0 94.3 90.7 92.6 90.18 enwik9_8k.squashfs.img 111 108 110 108 110 109.4 enwik9_16k.squashfs.img 158 163 146 165 174 161.2 enwik9_128k.squashfs.img 324 314 262 262 296 291.6
Thanks,
Gao Xiang
Gao Xiang (8):
staging: erofs: add compacted ondisk compression indexes
staging: erofs: add compacted compression indexes support
staging: erofs: move per-CPU buffers implementation to utils.c
staging: erofs: move stagingpage operations to compress.h
staging: erofs: introduce generic decompression backend
staging: erofs: introduce LZ4 decompression inplace
staging: erofs: switch to new decompression backend
staging: erofs: integrate decompression inplace
drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 62 ++++ drivers/staging/erofs/data.c | 4 +- drivers/staging/erofs/decompressor.c | 329 ++++++++++++++++++ drivers/staging/erofs/erofs_fs.h | 68 +++- drivers/staging/erofs/inode.c | 12 +- drivers/staging/erofs/internal.h | 52 ++- drivers/staging/erofs/unzip_vle.c | 368 ++------------------ drivers/staging/erofs/unzip_vle.h | 38 +-- drivers/staging/erofs/unzip_vle_lz4.c | 229 ------------- drivers/staging/erofs/utils.c | 12 + drivers/staging/erofs/zmap.c | 463 ++++++++++++++++++++++++++ 12 files changed, 1006 insertions(+), 633 deletions(-) create mode 100644 drivers/staging/erofs/compress.h create mode 100644 drivers/staging/erofs/decompressor.c delete mode 100644 drivers/staging/erofs/unzip_vle_lz4.c create mode 100644 drivers/staging/erofs/zmap.c
Signature · 1 line
2.17.1
PATCH v3 1/8 staging: erofs: add compacted ondisk compression indexes
From: Gao Xiang <[email protected]>
This patch introduces new compacted compression indexes.
In contract to legacy compression indexes that
each 4k logical cluster has an 8-byte index,
compacted ondisk compression indexes will have
amortized 2 bytes for each 4k logical cluster (compacted 2B) amortized 4 bytes for each 4k logical cluster (compacted 4B)
In detail, several continuous clusters will be encoded in a compacted pack with cluster types, offsets, and one blkaddr at the end of the pack to leave 4-byte margin for better decoding performance, as illustrated below:
_____________________________________________
|___ at _____ encoded bits __________|_ blkaddr _| 0 . amortized * vcnt . . . . amortized * vcnt - 4 . .
.___________________.
|_type_|_clusterofs_|
Note that compacted 2 / 4B should be aligned with 32 / 8 bytes in order to avoid each pack crossing page boundary.
---
change log v3:
- wrap up offset for better readability pointed out by Chao.
drivers/staging/erofs/data.c | 4 +- drivers/staging/erofs/erofs_fs.h | 65 ++++++++++++++++++++++++------- drivers/staging/erofs/inode.c | 5 +-- drivers/staging/erofs/internal.h | 11 ++---- drivers/staging/erofs/unzip_vle.c | 8 ++-- 5 files changed, 62 insertions(+), 31 deletions(-)
drivers/staging/erofs/data.c +2 -2
@@ -124,7 +124,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,trace_erofs_map_blocks_flatmode_enter (inode, map, flags); nblocks =DIV_ROUND_UP (inode->i_size,PAGE_SIZE ); - lastblk = nblocks -is_inode_layout_inline (inode); + lastblk = nblocks -is_inode_flat_inline (inode);if (unlikely (offset >= inode->i_size)) {/* leave out-of-bound access unmapped */
@@ -139,7 +139,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,if (offset <blknr_to_addr (lastblk)) { map->m_pa =blknr_to_addr (vi->raw_blkaddr) + map->m_la; map->m_plen =blknr_to_addr (lastblk) - offset; - }else if (is_inode_layout_inline (inode)) { + }else if (is_inode_flat_inline (inode)) {/* 2 - inode inline B: inode, [xattrs], inline last blk... */ struct erofs_sb_info *sbi =EROFS_SB (inode->i_sb);
drivers/staging/erofs/erofs_fs.h +52 -13
97 lines not on this page · raw message
drivers/staging/erofs/inode.c +1 -4
@@ -127,12 +127,9 @@ static int fill_inline_data(struct inode *inode, void *data, {struct erofs_vnode *vi =EROFS_V (inode);struct erofs_sb_info *sbi =EROFS_I_SB (inode); -const int mode = vi->datamode; - -DBG_BUGON (mode >=EROFS_INODE_LAYOUT_MAX );/* should be inode inline C */ -if (mode !=EROFS_INODE_LAYOUT_INLINE ) +if (!is_inode_flat_inline (inode))return 0 ;/* fast symlink (following ext4) */
drivers/staging/erofs/internal.h +3 -8
@@ -382,19 +382,14 @@ static inline unsigned long inode_datablocks(struct inode *inode)return DIV_ROUND_UP (inode->i_size,EROFS_BLKSIZ ); } -static inline bool is_inode_layout_plain (struct inode *inode) -{ -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_PLAIN ; -} -static inline bool is_inode_layout_compression (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_COMPRESSION ; +return erofs_inode_is_data_compressed (EROFS_V (inode)->datamode); } -static inline bool is_inode_layout_inline (struct inode *inode) +static inline bool is_inode_flat_inline (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_INLINE ; +return EROFS_V (inode)->datamode ==EROFS_INODE_FLAT_INLINE ; }extern const struct super_operations erofs_sops;
drivers/staging/erofs/unzip_vle.c +4 -4
@@ -1642,8 +1642,8 @@ vle_extent_blkaddr(struct inode *inode, pgoff_t index)struct erofs_sb_info *sbi =EROFS_I_SB (inode);struct erofs_vnode *vi =EROFS_V (inode); -unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + +unsigned int ofs =Z_EROFS_VLE_LEGACY_INDEX_ALIGN (vi->inode_isize + + vi->xattr_isize) + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blknr (iloc (sbi, vi->nid) + ofs);
@@ -1655,8 +1655,8 @@ vle_extent_blkoff(struct inode *inode, pgoff_t index)struct erofs_sb_info *sbi =EROFS_I_SB (inode);struct erofs_vnode *vi =EROFS_V (inode); -unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + +unsigned int ofs =Z_EROFS_VLE_LEGACY_INDEX_ALIGN (vi->inode_isize + + vi->xattr_isize) + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blkoff (iloc (sbi, vi->nid) + ofs);
Signature · 1 line
2.17.1
PATCH v3 2/8 staging: erofs: add compacted compression indexes support
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
This patch aims at compacted compression indexes:
1) cleanup z_erofs_map_blocks_iter and move into zmap.c;
2) add compacted 4/2B decoding support.
On kirin980 platform, sequential read is increased about 6% (725MiB/s -> 770MiB/s) on enwik9 dataset if compacted 2B feature is enabled.
---
change log v3:
- wrap up offset for better readability pointed out by Chao.
drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/inode.c | 7 +- drivers/staging/erofs/internal.h | 18 +- drivers/staging/erofs/unzip_vle.c | 286 ------------------ drivers/staging/erofs/zmap.c | 462 ++++++++++++++++++++++++++++++ 5 files changed, 480 insertions(+), 295 deletions(-) create mode 100644 drivers/staging/erofs/zmap.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o
drivers/staging/erofs/inode.c +1 -6
@@ -210,12 +210,7 @@ static int fill_inode(struct inode *inode, int isdir) }if (is_inode_layout_compression (inode)) { -#ifdef CONFIG_EROFS_FS_ZIP - inode->i_mapping->a_ops = - &z_erofs_vle_normalaccess_aops; -#else - err = -ENOTSUPP ; -#endif + err =z_erofs_fill_inode (inode); goto out_unlock; }
drivers/staging/erofs/internal.h +16 -2
@@ -339,9 +339,11 @@ static inline erofs_off_t iloc(struct erofs_sb_info *sbi, erofs_nid_t nid)/* atomic flag definitions */ #define EROFS_V_EA_INITED_BIT 0 +#define EROFS_V_Z_INITED_BIT 1/* bitlock definitions (arranged in reverse order) */ #define EROFS_V_BL_XATTR_BIT (BITS_PER_LONG - 1) +#define EROFS_V_BL_Z_BIT (BITS_PER_LONG - 2)struct erofs_vnode {erofs_nid_t nid;
@@ -356,8 +358,17 @@ struct erofs_vnode {unsigned xattr_shared_count;unsigned *xattr_shared_xattrs; -erofs_blk_t raw_blkaddr; - +union { +erofs_blk_t raw_blkaddr; +#ifdef CONFIG_EROFS_FS_ZIP +struct { +unsigned short z_advise; +unsigned char z_algorithmtype[2 ]; +unsigned char z_logical_clusterbits; +unsigned char z_physical_clusterbits[2 ]; + }; +#endif + };/* the corresponding vfs inode */ struct inode vfs_inode; };
@@ -447,11 +458,14 @@ struct erofs_map_blocks {/* Flags used by erofs_map_blocks() */ #define EROFS_GET_BLOCKS_RAW 0x0001 +/* zmap.c */ #ifdef CONFIG_EROFS_FS_ZIP +int z_erofs_fill_inode (struct inode *inode);int z_erofs_map_blocks_iter (struct inode *inode,struct erofs_map_blocks *map,int flags);#else +static inline int z_erofs_fill_inode (struct inode *inode) {return -ENOTSUPP ; }static inline int z_erofs_map_blocks_iter (struct inode *inode,struct erofs_map_blocks *map,int flags)
drivers/staging/erofs/unzip_vle.c +0 -286
289 lines not on this page · raw message
drivers/staging/erofs/zmap.c new file +462 -0
462 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 3/8 staging: erofs: move per-CPU buffers implementation to utils.c
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
This patch moves per-CPU buffers to utils.c in order for the upcoming generic decompression framework to use it.
Note that I tried to use generic per-CPU buffer or per-CPU page approaches to clean up further, but obvious performanace regression (about 2% for sequential read) was observed.
Therefore let's leave it as it is instead, just move to utils.c and I'll try to dig into the root cause later.
---
change log v3:
- handle error code if returned by erofs_get_pcpubuf() pointed out by Chao.
drivers/staging/erofs/internal.h | 26 +++++++++++++++++++ drivers/staging/erofs/unzip_vle.c | 5 ++-- drivers/staging/erofs/unzip_vle.h | 4 +-- drivers/staging/erofs/unzip_vle_lz4.c | 37 +++++++++++---------------- drivers/staging/erofs/utils.c | 12 +++++++++ 5 files changed, 56 insertions(+), 28 deletions(-)
drivers/staging/erofs/internal.h +26 -0
@@ -321,6 +321,16 @@ static inline void z_erofs_exit_zip_subsystem(void) {}/* page count of a compressed cluster */ #define erofs_clusterpages (sbi) ((1 << (sbi)->clusterbits) / PAGE_SIZE) +#define Z_EROFS_NR_INLINE_PAGEVECS 3 + +#if (Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_NR_INLINE_PAGEVECS ) +#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES +#else +#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_NR_INLINE_PAGEVECS +#endif + +#else +#define EROFS_PCPUBUF_NR_PAGES 0#endif typedef u64 erofs_off_t ;
@@ -608,6 +618,22 @@ static inline void erofs_vunmap(const void *mem, unsigned int count)extern struct shrinker erofs_shrinker_info;struct page *erofs_allocpage (struct list_head *pool,gfp_t gfp); + +#if (EROFS_PCPUBUF_NR_PAGES >0 ) +void *erofs_get_pcpubuf (unsigned int pagenr); +#define erofs_put_pcpubuf (buf) do { \ + (void)&(buf); \ + preempt_enable(); \ +} while (0) +#else +static inline void *erofs_get_pcpubuf (unsigned int pagenr) +{ +return ERR_PTR (-ENOTSUPP ); +} + +#define erofs_put_pcpubuf (buf) do {} while (0) +#endif +void erofs_register_super (struct super_block *sb);void erofs_unregister_super (struct super_block *sb);
drivers/staging/erofs/unzip_vle.c +2 -3
@@ -552,8 +552,7 @@ static int z_erofs_vle_work_iter_begin(struct z_erofs_vle_work_builder *builder,if (IS_ERR (work))return PTR_ERR (work); got_it: -z_erofs_pagevec_ctor_init (&builder->vector, -Z_EROFS_VLE_INLINE_PAGEVECS , +z_erofs_pagevec_ctor_init (&builder->vector,Z_EROFS_NR_INLINE_PAGEVECS , work->pagevec, work->vcnt);if (builder->role >=Z_EROFS_VLE_WORK_PRIMARY ) {
@@ -936,7 +935,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,for (i =0 ; i < nr_pages; ++i) pages[i] =NULL ; -z_erofs_pagevec_ctor_init (&ctor,Z_EROFS_VLE_INLINE_PAGEVECS , +z_erofs_pagevec_ctor_init (&ctor,Z_EROFS_NR_INLINE_PAGEVECS , work->pagevec,0 );for (i =0 ; i < work->vcnt; ++i) {
drivers/staging/erofs/unzip_vle.h +1 -3
@@ -44,8 +44,6 @@ static inline bool z_erofs_gather_if_stagingpage(struct list_head *page_pool,* */ -#define Z_EROFS_VLE_INLINE_PAGEVECS 3 -struct z_erofs_vle_work {struct mutex lock;
@@ -58,7 +56,7 @@ struct z_erofs_vle_work {union {/* L: pagevec */ -erofs_vtptr_t pagevec[Z_EROFS_VLE_INLINE_PAGEVECS ]; +erofs_vtptr_t pagevec[Z_EROFS_NR_INLINE_PAGEVECS ];struct rcu_head rcu; }; };
drivers/staging/erofs/unzip_vle_lz4.c +15 -22
@@ -34,16 +34,6 @@ static int z_erofs_unzip_lz4(void *in, void *out, size_t inlen, size_t outlen)return -EIO ; } -#if Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_VLE_INLINE_PAGEVECS -#define EROFS_PERCPU_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES -#else -#define EROFS_PERCPU_NR_PAGES Z_EROFS_VLE_INLINE_PAGEVECS -#endif - -static struct { -char data[PAGE_SIZE *EROFS_PERCPU_NR_PAGES ]; -} erofs_pcpubuf[NR_CPUS ]; - int z_erofs_vle_plain_copy(struct page **compressed_pages, unsigned int clusterpages, struct page **pages,
@@ -56,8 +46,9 @@ int z_erofs_vle_plain_copy(struct page **compressed_pages,char *percpu_data;bool mirrored[Z_EROFS_CLUSTER_MAX_PAGES ] = {0 }; -preempt_disable (); - percpu_data = erofs_pcpubuf[smp_processor_id ()].data; + percpu_data =erofs_get_pcpubuf (0 ); +if (IS_ERR (percpu_data)) +return PTR_ERR (percpu_data); j =0 ;for (i =0 ; i < nr_pages; j = i++) {
@@ -117,7 +108,7 @@ int z_erofs_vle_plain_copy(struct page **compressed_pages,if (src && !mirrored[j])kunmap_atomic (src); -preempt_enable (); +erofs_put_pcpubuf (percpu_data);return 0 ; }
@@ -131,7 +122,7 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,unsigned int nr_pages, i, j;int ret; -if (outlen + pageofs >EROFS_PERCPU_NR_PAGES *PAGE_SIZE ) +if (outlen + pageofs >EROFS_PCPUBUF_NR_PAGES *PAGE_SIZE )return -ENOTSUPP ; nr_pages =DIV_ROUND_UP (outlen + pageofs,PAGE_SIZE );
@@ -144,8 +135,9 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,return -ENOMEM ; } -preempt_disable (); - vout = erofs_pcpubuf[smp_processor_id ()].data; + vout =erofs_get_pcpubuf (0 ); +if (IS_ERR (vout)) +return PTR_ERR (vout); ret =z_erofs_unzip_lz4 (vin, vout + pageofs, clusterpages *PAGE_SIZE , outlen);
@@ -174,7 +166,7 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages, } out: -preempt_enable (); +erofs_put_pcpubuf (vout);if (clusterpages ==1 )kunmap_atomic (vin);
@@ -196,8 +188,9 @@ int z_erofs_vle_unzip_vmap(struct page **compressed_pages,int ret;if (overlapped) { -preempt_disable (); - vin = erofs_pcpubuf[smp_processor_id ()].data; + vin =erofs_get_pcpubuf (0 ); +if (IS_ERR (vin)) +return PTR_ERR (vin);for (i =0 ; i < clusterpages; ++i) {void *t =kmap_atomic (compressed_pages[i]);
@@ -216,13 +209,13 @@ int z_erofs_vle_unzip_vmap(struct page **compressed_pages,if (ret >0 ) ret =0 ; -if (!overlapped) { +if (overlapped) { +erofs_put_pcpubuf (vin); + }else {if (clusterpages ==1 )kunmap_atomic (vin);else erofs_vunmap (vin, clusterpages); - }else { -preempt_enable (); }return ret; }
drivers/staging/erofs/utils.c +12 -0
@@ -27,6 +27,18 @@ struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp)return page; } +#if (EROFS_PCPUBUF_NR_PAGES >0 ) +static struct { +u8 data[PAGE_SIZE *EROFS_PCPUBUF_NR_PAGES ]; +}____cacheline_aligned_in_smp erofs_pcpubuf[NR_CPUS ]; + +void *erofs_get_pcpubuf (unsigned int pagenr) +{ +preempt_disable (); +return &erofs_pcpubuf[smp_processor_id ()].data[pagenr *PAGE_SIZE ]; +} +#endif +/* global shrink count (for all mounted EROFS instances) */ static atomic_long_t erofs_global_shrink_cnt;
Signature · 1 line
2.17.1
PATCH v3 4/8 staging: erofs: move stagingpage operations to compress.h
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
stagingpages are behaved as bounce pages for temporary use. Move to compress.h since the upcoming decompressor will allocate stagingpages as well.
--- drivers/staging/erofs/compress.h | 40 +++++++++++++++++++++++++++++++ drivers/staging/erofs/unzip_vle.c | 11 +++++---- drivers/staging/erofs/unzip_vle.h | 20 ---------------- 3 files changed, 46 insertions(+), 25 deletions(-) create mode 100644 drivers/staging/erofs/compress.h
drivers/staging/erofs/compress.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* linux/drivers/staging/erofs/compress.h +* +* Copyright (C) 2019 HUAWEI, Inc. +* http://www.huawei.com/ +* Created by Gao Xiang <gaoxiang25 at huawei.com> +*/ +#ifndef __EROFS_FS_COMPRESS_H +#define __EROFS_FS_COMPRESS_H + +/* +* - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - +* used to mark temporary allocated pages from other +* file/cached pages and NULL mapping pages. +*/ +#define Z_EROFS_MAPPING_STAGING ((void *)0x5A110C8D) + +/* check if a page is marked as staging */ +static inline bool z_erofs_page_is_staging (struct page *page) +{ +return page->mapping ==Z_EROFS_MAPPING_STAGING ; +} + +static inline bool z_erofs_put_stagingpage (struct list_head *pagepool, +struct page *page) +{ +if (!z_erofs_page_is_staging (page)) +return false; + +/* staging pages should not be used by others at the same time */ +if (page_ref_count (page) >1 ) +put_page (page); +else +list_add (&page->lru, pagepool); +return true; +} + +#endif +
drivers/staging/erofs/unzip_vle.c +6 -5
@@ -11,6 +11,7 @@* distribution for more details. */ #include "unzip_vle.h" +#include "compress.h" #include <linux/prefetch.h> #include <trace/events/erofs.h>
@@ -855,7 +856,7 @@ static inline void z_erofs_vle_read_endio(struct bio *bio)DBG_BUGON (PageUptodate (page));DBG_BUGON (!page->mapping); -if (unlikely (!sbi && !z_erofs_is_stagingpage (page))) { +if (unlikely (!sbi && !z_erofs_page_is_staging (page))) { sbi =EROFS_SB (page->mapping->host->i_sb);if (time_to_inject (sbi,FAULT_READ_IO )) {
@@ -947,7 +948,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page);DBG_BUGON (!page->mapping); -if (z_erofs_gather_if_stagingpage (page_pool, page)) +if (z_erofs_put_stagingpage (page_pool, page))continue ;if (page_type ==Z_EROFS_VLE_PAGE_TYPE_HEAD )
@@ -977,7 +978,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page);DBG_BUGON (!page->mapping); -if (!z_erofs_is_stagingpage (page)) { +if (!z_erofs_page_is_staging (page)) {if (erofs_page_is_managed (sbi, page)) {if (unlikely (!PageUptodate (page))) err = -EIO ;
@@ -1055,7 +1056,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,continue ;/* recycle all individual staging pages */ - (void )z_erofs_gather_if_stagingpage (page_pool, page); + (void )z_erofs_put_stagingpage (page_pool, page);WRITE_ONCE (compressed_pages[i],NULL ); }
@@ -1068,7 +1069,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page->mapping);/* recycle all individual staging pages */ -if (z_erofs_gather_if_stagingpage (page_pool, page)) +if (z_erofs_put_stagingpage (page_pool, page))continue ;if (unlikely (err <0 ))
drivers/staging/erofs/unzip_vle.h +0 -20
@@ -16,26 +16,6 @@#include "internal.h" #include "unzip_pagevec.h" -/* -* - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - -* used for temporary allocated pages (via erofs_allocpage), -* in order to seperate those from NULL mapping (eg. truncated pages) -*/ -#define Z_EROFS_MAPPING_STAGING ((void *)0x5A110C8D) - -#define z_erofs_is_stagingpage (page) \ - ((page)->mapping == Z_EROFS_MAPPING_STAGING) - -static inline bool z_erofs_gather_if_stagingpage (struct list_head *page_pool, -struct page *page) -{ -if (z_erofs_is_stagingpage (page)) { -list_add (&page->lru, page_pool); -return true; - } -return false; -} -/* * Structure fields follow one of the following exclusion rules. *
Signature · 1 line
2.17.1
PATCH v3 5/8 staging: erofs: introduce generic decompression backend
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
This patch adds a new generic decompression framework in order to replace the old LZ4-specific decompression code.
Even though LZ4 is still the only supported algorithm, yet it is more cleaner and easy to integrate new algorithm than the old almost hard-coded decompression backend.
---
change log v3:
- move extra 0PADDING logic (used as a decompression inplace enable mark) to the following patch;
drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 21 ++ drivers/staging/erofs/decompressor.c | 301 +++++++++++++++++++++++++++ 3 files changed, 323 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/erofs/decompressor.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.o
drivers/staging/erofs/compress.h +21 -0
@@ -9,6 +9,24 @@#ifndef __EROFS_FS_COMPRESS_H#define __EROFS_FS_COMPRESS_H +#include "internal.h" + +enum { +Z_EROFS_COMPRESSION_SHIFTED =Z_EROFS_COMPRESSION_MAX , +Z_EROFS_COMPRESSION_RUNTIME_MAX +}; + +struct z_erofs_decompress_req { +struct page **in, **out; + +unsigned short pageofs_out; +unsigned int inputsize, outputsize; + +/* indicate the algorithm will be used for decompression */ +unsigned int alg; +bool inplace_io, partial_decoding; +}; +/* * - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - * used to mark temporary allocated pages from other
@@ -36,5 +54,8 @@ static inline bool z_erofs_put_stagingpage(struct list_head *pagepool, return true; } +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +#endif
drivers/staging/erofs/decompressor.c new file +301 -0
301 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 6/8 staging: erofs: introduce LZ4 decompression inplace
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
compressed data will be usually loaded into last pages of the extent (the last page for 4k) for in-place decompression (more specifically, in-place IO), as ilustration below,
start of compressed logical extent
| end of this logical extent
| |
______v___________________________v________
... | page 6 | page 7 | page 8 | page 9 | ...
|__________|__________|__________|__________|
. ^ . ^
. |compressed|
. | data |
. . .
|< dstsize >|<margin>|
oend iendop ip
Therefore, it's possible to do decompression inplace (thus no memcpy at all) if the margin is sufficient and safe enough [1], and it can be implemented only for fixed-size output compression compared with fixed-size input compression.
No memcpy for most of in-place IO (about 99% of enwik9) after decompression inplace is implemented and sequential read will be improved of course (see the following patches for test results).
[1] https://github.com/lz4/lz4/commit/b17f578a919b7e6b078cede2d52be29dd48c8e8c
https://github.com/lz4/lz4/commit/5997e139f53169fa3a1c1b4418d2452a90b01602
--- drivers/staging/erofs/compress.h | 1 + drivers/staging/erofs/decompressor.c | 36 ++++++++++++++++++++++++---- drivers/staging/erofs/erofs_fs.h | 3 ++- 3 files changed, 35 insertions(+), 5 deletions(-)
drivers/staging/erofs/compress.h +1 -0
@@ -17,6 +17,7 @@ enum { };struct z_erofs_decompress_req { +struct super_block *sb;struct page **in, **out;unsigned short pageofs_out;
drivers/staging/erofs/decompressor.c +32 -4
@@ -14,6 +14,9 @@#endif #define LZ4_MAX_DISTANCE_PAGES DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) +#ifndef LZ4_DECOMPRESS_INPLACE_MARGIN +#define LZ4_DECOMPRESS_INPLACE_MARGIN (srcsize) (((srcsize) >> 8) + 32) +#endif struct z_erofs_decompressor {/*
@@ -112,7 +115,7 @@ static int lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) {unsigned int inputmargin, inlen;u8 *src; -bool copied; +bool copied, support_0padding;int ret;if (rq->inputsize >PAGE_SIZE )
@@ -120,13 +123,38 @@ static int lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) src =kmap_atomic (*rq->in); inputmargin =0 ; + support_0padding = false; + +/* decompression inplace is only safe when 0padding is enabled */ +if (EROFS_SB (rq->sb)->requirements &EROFS_REQUIREMENT_LZ4_0PADDING ) { + support_0padding = true; + +while (!src[inputmargin & ~PAGE_MASK ]) +if (!(++inputmargin & ~PAGE_MASK )) +break ; + +if (inputmargin >= rq->inputsize) { +kunmap_atomic (src); +return -EIO ; + } + } copied = false; inlen = rq->inputsize - inputmargin;if (rq->inplace_io) { - src =generic_copy_inplace_data (rq, src, inputmargin); - inputmargin =0 ; - copied = true; +const uint oend = (rq->pageofs_out + + rq->outputsize) & ~PAGE_MASK ; +const uint nr =PAGE_ALIGN (rq->pageofs_out + + rq->outputsize) >>PAGE_SHIFT ; + +if (rq->partial_decoding || !support_0padding || + rq->out[nr -1 ] != rq->in[0 ] || + rq->inputsize - oend < +LZ4_DECOMPRESS_INPLACE_MARGIN (inlen)) { + src =generic_copy_inplace_data (rq, src, inputmargin); + inputmargin =0 ; + copied = true; + } } ret = LZ4_decompress_safe_partial(src + inputmargin, out,
drivers/staging/erofs/erofs_fs.h +2 -1
@@ -21,7 +21,8 @@* Any bits that aren't in EROFS_ALL_REQUIREMENTS should be * incompatible with this kernel version. */ -#define EROFS_ALL_REQUIREMENTS 0 +#define EROFS_REQUIREMENT_LZ4_0PADDING 0x00000001 +#define EROFS_ALL_REQUIREMENTS EROFS_REQUIREMENT_LZ4_0PADDINGstruct erofs_super_block {/* 0 */ __le32 magic;/* in the little endian */
Signature · 1 line
2.17.1
PATCH v3 7/8 staging: erofs: switch to new decompression backend
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
This patch integrates new decompression framework to erofs decompression path, and remove the old decompression implementation as well.
On kirin980 platform, sequential read is slightly
improved to 778MiB/s after the new decompression
backend is used.
--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/internal.h | 6 - drivers/staging/erofs/unzip_vle.c | 59 +++---- drivers/staging/erofs/unzip_vle.h | 15 +- drivers/staging/erofs/unzip_vle_lz4.c | 222 -------------------------- 5 files changed, 24 insertions(+), 280 deletions(-) delete mode 100644 drivers/staging/erofs/unzip_vle_lz4.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o zmap.o decompressor.o
drivers/staging/erofs/internal.h +0 -6
@@ -321,14 +321,8 @@ static inline void z_erofs_exit_zip_subsystem(void) {}/* page count of a compressed cluster */ #define erofs_clusterpages (sbi) ((1 << (sbi)->clusterbits) / PAGE_SIZE) -#define Z_EROFS_NR_INLINE_PAGEVECS 3 -#if (Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_NR_INLINE_PAGEVECS )#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES -#else -#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_NR_INLINE_PAGEVECS -#endif -#else #define EROFS_PCPUBUF_NR_PAGES 0#endif
drivers/staging/erofs/unzip_vle.c +21 -38
76 lines not on this page · raw message
drivers/staging/erofs/unzip_vle.h +2 -13
@@ -16,6 +16,8 @@#include "internal.h" #include "unzip_pagevec.h" +#define Z_EROFS_NR_INLINE_PAGEVECS 3 +/* * Structure fields follow one of the following exclusion rules. *
@@ -189,18 +191,5 @@ static inline void z_erofs_onlinepage_endio(struct page *page) min_t(unsigned int, THREAD_SIZE / 8 / sizeof(struct page *), 96U)#define Z_EROFS_VLE_VMAP_GLOBAL_PAGES 2048 -/* unzip_vle_lz4.c */ -int z_erofs_vle_plain_copy (struct page **compressed_pages, -unsigned int clusterpages,struct page **pages, -unsigned int nr_pages,unsigned short pageofs); -int z_erofs_vle_unzip_fast_percpu (struct page **compressed_pages, -unsigned int clusterpages, -struct page **pages,unsigned int outlen, -unsigned short pageofs); -int z_erofs_vle_unzip_vmap (struct page **compressed_pages, -unsigned int clusterpages, -void *vaddr,unsigned int llen, -unsigned short pageofs,bool overlapped); -#endif
drivers/staging/erofs/unzip_vle_lz4.c deleted +0 -222
222 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 8/8 staging: erofs: integrate decompression inplace
In reply to Gao XiangFrom: Gao Xiang <[email protected]>
Decompressor needs to know whether it's a partial or full decompression since only full decompression can be decompressed in-place.
On kirin980 platform, sequential read is finally increased to 812MiB/s after decompression inplace is enabled.
--- drivers/staging/erofs/internal.h | 3 +++ drivers/staging/erofs/unzip_vle.c | 15 +++++++++++---- drivers/staging/erofs/unzip_vle.h | 1 + drivers/staging/erofs/zmap.c | 1 + 4 files changed, 16 insertions(+), 4 deletions(-)
drivers/staging/erofs/internal.h +3 -0
@@ -441,6 +441,7 @@ extern const struct address_space_operations z_erofs_vle_normalaccess_aops;*/ enum { BH_Zipped = BH_PrivateStart, + BH_FullMapped, };/* Has a disk mapping */
@@ -449,6 +450,8 @@ enum {#define EROFS_MAP_META (1 << BH_Meta)/* The extent has been compressed */ #define EROFS_MAP_ZIPPED (1 << BH_Zipped) +/* The length of extent is full */ +#define EROFS_MAP_FULL_MAPPED (1 << BH_FullMapped)struct erofs_map_blocks {erofs_off_t m_pa, m_la;
drivers/staging/erofs/unzip_vle.c +11 -4
@@ -469,6 +469,9 @@ z_erofs_vle_work_register(const struct z_erofs_vle_work_finder *f, Z_EROFS_VLE_WORKGRP_FMT_LZ4 : Z_EROFS_VLE_WORKGRP_FMT_PLAIN); +if (map->m_flags &EROFS_MAP_FULL_MAPPED ) + grp->flags |=Z_EROFS_VLE_WORKGRP_FULL_LENGTH ; +/* new workgrps have been claimed as type 1 */ WRITE_ONCE (grp->next, *f->owned_head);/* primary and followed work for all new workgrps */
@@ -901,7 +904,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,unsigned int i, outputsize;enum z_erofs_page_type page_type; -bool overlapped; +bool overlapped, partial;struct z_erofs_vle_work *work;int err;
@@ -1009,10 +1012,13 @@ static int z_erofs_vle_unzip(struct super_block *sb,if (unlikely (err)) goto out; -if (nr_pages <<PAGE_SHIFT >= work->pageofs + grp->llen) +if (nr_pages <<PAGE_SHIFT >= work->pageofs + grp->llen) { outputsize = grp->llen; -else + partial = !(grp->flags &Z_EROFS_VLE_WORKGRP_FULL_LENGTH ); + }else { outputsize = (nr_pages <<PAGE_SHIFT ) - work->pageofs; + partial = true; + }if (z_erofs_vle_workgrp_fmt (grp) ==Z_EROFS_VLE_WORKGRP_FMT_PLAIN ) algorithm =Z_EROFS_COMPRESSION_SHIFTED ;
@@ -1028,7 +1034,8 @@ static int z_erofs_vle_unzip(struct super_block *sb, .outputsize = outputsize, .alg = algorithm, .inplace_io = overlapped, - .partial_decoding = true }, page_pool); + .partial_decoding = partial + }, page_pool); out:/* must handle all compressed pages before endding pages */
drivers/staging/erofs/unzip_vle.h +1 -0
@@ -46,6 +46,7 @@ struct z_erofs_vle_work {#define Z_EROFS_VLE_WORKGRP_FMT_PLAIN 0#define Z_EROFS_VLE_WORKGRP_FMT_LZ4 1#define Z_EROFS_VLE_WORKGRP_FMT_MASK 1 +#define Z_EROFS_VLE_WORKGRP_FULL_LENGTH 2typedef void *z_erofs_vle_owned_workgrp_t ;
drivers/staging/erofs/zmap.c +1 -0
@@ -424,6 +424,7 @@ int z_erofs_map_blocks_iter(struct inode *inode, goto unmap_out; } end = (m.lcn << lclusterbits) | m.clusterofs; + map->m_flags |=EROFS_MAP_FULL_MAPPED ; m.delta[0 ] =1 ;/* fallthrough */ case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD :
Signature · 1 line
2.17.1
Re: [PATCH v3 3/8] staging: erofs: move per-CPU buffers implementation to utils.c
In reply to Gao XiangQuoted 10 lines · Gao Xiang
From: Gao Xiang <gaoxiang25 at huawei.com>
This patch moves per-CPU buffers to
utils.cin order for the upcoming generic decompression framework to use it.Note that I tried to use generic per-CPU buffer or per-CPU page approaches to clean up further, but obvious performanace regression (about 2% for sequential read) was observed.
Quoted 4 lines
Therefore let's leave it as it is instead, just move to
utils.cand I'll try to dig into the root cause later.Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>
Thanks,
Quoted 9 lines · Gao Xiang
This is patch v3 of erofs decompression inplace approach, which is sent out by my personal email since I'm out of office to attend Open Source Summit China 2019 these days. No major change from PATCH v2 since no noticeable issue raised from landing to our products till now, mainly as a response to Chao's suggestions.
See the bottom lines which are taken from RFC PATCH v1 and describe the principle of these technologies.
Quoted 7 lines
The series is based on the latest staging-next since all dependencies have already been merged.
changelog from v2:
- wrap up some offsets as marcos;
- add some error handling forerofs_get_pcpubuf();
- move some decompression inplace stuffes from PATCH 5 -> 6.
Thanks for the update, those all changes look good to me. :)
Thanks,