staging: erofs: decompression inplace approach
v2 cover staging: erofs: decompression inplace approach
This is patch v2 of erofs decompression inplace approach, no major issues observed after v1 preliminarily applied to our products, and changes from v1 are minor, so I drop "RFC" tag from this version.
See the bottom lines which are taken from RFC PATCH v1 and describe the principle of these technologies.
The series is based on the latest staging-next since all dependencies have already been merged.
changelog from v1:
- keep Z_EROFS_NR_INLINE_PAGEVECS in unzip_vle.h after switching to new decompression backend;
- add some DBG_BUGONs in new decompression backend to observe potential issues;
- minor code cleanup.
8<--------
Hi,
After working on for more than half a year, the detail of erofs decompression inplace is almost determined and ready for linux-next.
Currently, inplace IO is used if the whole compressed data is used in order to reduce compressed pages extra memory overhead and an extra memcpy (the only one memcpy) will be used for each inplace IO since temporary buffer is needed to keep decompressing safe for inplace IO.
However, most of lz-based decompression algorithms support decompression inplace by their algorithm designs, such as LZ4, LZO, etc.
If iend - oend margin is large enough, decompression inplace can be done in the same buffer safely, as illustrated below:
start of compressed logical extent
| end of this logical extent
| |
______v___________________________v________
... | page 6 | page 7 | page 8 | page 9 | ...
|__________|__________|__________|__________|
. ^ . ^
. |compressed|
. | data |
. . .
|< dstsize >|<margin>|
oend iendop ip
Fixed-size output compression can make the full use of this feature to reduce memory overhead and avoid extra memcpy compared with fixed-size input compression since iend is strictly not less than oend for fixed-size output compression with inplace IO to last pages.
In addition, erofs compression indexes have been improved as well by introducing compacted compression indexes.
These two techniques all benefit sequential read (on x86_64, 710.8MiB/s -> 755.4MiB/s; on Kirin980, 725MiB/s -> 812MiB/s) therefore erofs could have similar sequential read performance against ext4 in a larger CR range on high-spend SSD / NVMe devices as well.
However, note that it is _cpu vs storage device_ tradeoff, there is no absolute performance conclusion for all on-market combinations.
Test images:
name size CR enwik9 1000000000 1.00 enwik9_4k.squashfs.img 621211648 1.61 enwik9_4k.erofs.img 558133248 1.79 enwik9_8k.squashfs.img 556191744 1.80 enwik9_16k.squashfs.img 502661120 1.99 enwik9_128k.squashfs.img 398204928 2.51
Test Environment:
CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz (4 cores, 8 threads)
DDR: 8G
SSD: INTEL SSDPEKKF360G7H
Kernel: Linux 5.2-rc3+ (with lz4-1.8.3 algorithm)
Test configuration:
squashfs:CONFIG_SQUASHFS=yCONFIG_SQUASHFS_FILE_DIRECT=yCONFIG_SQUASHFS_DECOMP_MULTI_PERCPU=yCONFIG_SQUASHFS_LZ4=yCONFIG_SQUASHFS_4K_DEVBLK_SIZE=y
erofs:CONFIG_EROFS_FS_USE_VM_MAP_RAM=yCONFIG_EROFS_FS_ZIP=yCONFIG_EROFS_FS_CLUSTER_PAGE_LIMIT=1CONFIG_EROFS_FS_ZIP_CACHE_BIPOLAR=y
with intel_pstate=disable,
8 cpus on at 1801000 scaling_{min,max}_freq,
userspace scaling_governorSequential read results (MiB/s):
1 2 3 4 5 avg enwik9_4k.ext4.img 767 770 738 726 724 745 enwik9_4k.erofs.img 756 745 770 746 760 755.4 enwik9_4k.squashfs.img 90.3 83.0 94.3 90.7 92.6 90.18 enwik9_8k.squashfs.img 111 108 110 108 110 109.4 enwik9_16k.squashfs.img 158 163 146 165 174 161.2 enwik9_128k.squashfs.img 324 314 262 262 296 291.6
Thanks,
Gao Xiang
Gao Xiang (8):
staging: erofs: add compacted ondisk compression indexes
staging: erofs: add compacted compression indexes support
staging: erofs: move per-CPU buffers implementation to utils.c
staging: erofs: move stagingpage operations to compress.h
staging: erofs: introduce generic decompression backend
staging: erofs: introduce LZ4 decompression inplace
staging: erofs: switch to new decompression backend
staging: erofs: integrate decompression inplace
drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 62 ++++ drivers/staging/erofs/data.c | 4 +- drivers/staging/erofs/decompressor.c | 322 ++++++++++++++++++ drivers/staging/erofs/erofs_fs.h | 60 +++- drivers/staging/erofs/inode.c | 12 +- drivers/staging/erofs/internal.h | 52 ++- drivers/staging/erofs/unzip_vle.c | 368 ++------------------ drivers/staging/erofs/unzip_vle.h | 38 +-- drivers/staging/erofs/unzip_vle_lz4.c | 229 ------------- drivers/staging/erofs/utils.c | 12 + drivers/staging/erofs/zmap.c | 463 ++++++++++++++++++++++++++ 12 files changed, 993 insertions(+), 631 deletions(-) create mode 100644 drivers/staging/erofs/compress.h create mode 100644 drivers/staging/erofs/decompressor.c delete mode 100644 drivers/staging/erofs/unzip_vle_lz4.c create mode 100644 drivers/staging/erofs/zmap.c
Signature · 1 line
2.17.1
PATCH v2 1/8 staging: erofs: add compacted ondisk compression indexes
This patch introduces new compacted compression indexes.
In contract to legacy compression indexes that
each 4k logical cluster has an 8-byte index,
compacted ondisk compression indexes will have
amortized 2 bytes for each 4k logical cluster (compacted 2B) amortized 4 bytes for each 4k logical cluster (compacted 4B)
In detail, several continuous clusters will be encoded in a compacted pack with cluster types, offsets, and one blkaddr at the end of the pack to leave 4-byte margin for better decoding performance, as illustrated below:
_____________________________________________
|___ at _____ encoded bits __________|_ blkaddr _| 0 . amortized * vcnt . . . . amortized * vcnt - 4 . .
.___________________.
|_type_|_clusterofs_|
Note that compacted 2 / 4B should be aligned with 32 / 8 bytes in order to avoid each pack crossing page boundary.
--- drivers/staging/erofs/data.c | 4 +-- drivers/staging/erofs/erofs_fs.h | 57 +++++++++++++++++++++++++------ drivers/staging/erofs/inode.c | 5 +-- drivers/staging/erofs/internal.h | 11 ++---- drivers/staging/erofs/unzip_vle.c | 8 ++--- 5 files changed, 56 insertions(+), 29 deletions(-)
drivers/staging/erofs/data.c +2 -2
@@ -124,7 +124,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,trace_erofs_map_blocks_flatmode_enter (inode, map, flags); nblocks =DIV_ROUND_UP (inode->i_size,PAGE_SIZE ); - lastblk = nblocks -is_inode_layout_inline (inode); + lastblk = nblocks -is_inode_flat_inline (inode);if (unlikely (offset >= inode->i_size)) {/* leave out-of-bound access unmapped */
@@ -139,7 +139,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,if (offset <blknr_to_addr (lastblk)) { map->m_pa =blknr_to_addr (vi->raw_blkaddr) + map->m_la; map->m_plen =blknr_to_addr (lastblk) - offset; - }else if (is_inode_layout_inline (inode)) { + }else if (is_inode_flat_inline (inode)) {/* 2 - inode inline B: inode, [xattrs], inline last blk... */ struct erofs_sb_info *sbi =EROFS_SB (inode->i_sb);
drivers/staging/erofs/erofs_fs.h +46 -11
83 lines not on this page · raw message
drivers/staging/erofs/inode.c +1 -4
@@ -127,12 +127,9 @@ static int fill_inline_data(struct inode *inode, void *data, {struct erofs_vnode *vi =EROFS_V (inode);struct erofs_sb_info *sbi =EROFS_I_SB (inode); -const int mode = vi->datamode; - -DBG_BUGON (mode >=EROFS_INODE_LAYOUT_MAX );/* should be inode inline C */ -if (mode !=EROFS_INODE_LAYOUT_INLINE ) +if (!is_inode_flat_inline (inode))return 0 ;/* fast symlink (following ext4) */
drivers/staging/erofs/internal.h +3 -8
@@ -382,19 +382,14 @@ static inline unsigned long inode_datablocks(struct inode *inode)return DIV_ROUND_UP (inode->i_size,EROFS_BLKSIZ ); } -static inline bool is_inode_layout_plain (struct inode *inode) -{ -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_PLAIN ; -} -static inline bool is_inode_layout_compression (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_COMPRESSION ; +return erofs_inode_is_data_compressed (EROFS_V (inode)->datamode); } -static inline bool is_inode_layout_inline (struct inode *inode) +static inline bool is_inode_flat_inline (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_INLINE ; +return EROFS_V (inode)->datamode ==EROFS_INODE_FLAT_INLINE ; }extern const struct super_operations erofs_sops;
drivers/staging/erofs/unzip_vle.c +4 -4
@@ -1643,8 +1643,8 @@ vle_extent_blkaddr(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blknr (iloc (sbi, vi->nid) + ofs); }
@@ -1656,8 +1656,8 @@ vle_extent_blkoff(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blkoff (iloc (sbi, vi->nid) + ofs); }
Signature · 1 line
2.17.1
Re: [PATCH v2 1/8] staging: erofs: add compacted ondisk compression indexes
Quoted 206 lines · Gao Xiang
This patch introduces new compacted compression indexes.
In contract to legacy compression indexes that
each 4k logical cluster has an 8-byte index,
compacted ondisk compression indexes will have
amortized 2 bytes for each 4k logical cluster (compacted 2B) amortized 4 bytes for each 4k logical cluster (compacted 4B)In detail, several continuous clusters will be encoded in a compacted pack with cluster types, offsets, and one blkaddr at the end of the pack to leave 4-byte margin for better decoding performance, as illustrated below:
_____________________________________________ |___ at _____ encoded bits __________|_ blkaddr _| 0 . amortized * vcnt . . . . amortized * vcnt - 4 . ..___________________.
|_type_|_clusterofs_|Note that compacted 2 / 4B should be aligned with 32 / 8 bytes in order to avoid each pack crossing page boundary.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>--- drivers/staging/erofs/data.c | 4 +-- drivers/staging/erofs/erofs_fs.h | 57 +++++++++++++++++++++++++------ drivers/staging/erofs/inode.c | 5 +-- drivers/staging/erofs/internal.h | 11 ++---- drivers/staging/erofs/unzip_vle.c | 8 ++--- 5 files changed, 56 insertions(+), 29 deletions(-)drivers/staging/erofs/data.c +2
-2@@ -124,7 +124,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,trace_erofs_map_blocks_flatmode_enter (inode, map, flags); nblocks =DIV_ROUND_UP (inode->i_size,PAGE_SIZE ); - lastblk = nblocks -is_inode_layout_inline (inode); + lastblk = nblocks -is_inode_flat_inline (inode);if (unlikely (offset >= inode->i_size)) {/* leave out-of-bound access unmapped */ @@ -139,7 +139,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,if (offset <blknr_to_addr (lastblk)) { map->m_pa =blknr_to_addr (vi->raw_blkaddr) + map->m_la; map->m_plen =blknr_to_addr (lastblk) - offset; - }else if (is_inode_layout_inline (inode)) { + }else if (is_inode_flat_inline (inode)) {/* 2 - inode inline B: inode, [xattrs], inline last blk... */ struct erofs_sb_info *sbi =EROFS_SB (inode->i_sb);drivers/staging/erofs/erofs_fs.h +46
-11@@ -49,19 +49,29 @@ struct erofs_super_block {* erofs inode data mapping: * 0 - inode plain without inline data A: * inode, [xattrs], ... | ... | no-holed data -* 1 - inode VLE compression B: +* 1 - inode VLE compression B (legacy): * inode, [xattrs], extents ... | ... * 2 - inode plain with inline data C: * inode, [xattrs], last_inline_data, ... | ... | no-holed data -* 3~7 - reserved +* 3 - inode compression D: +* inode, [xattrs], map_header, extents ... | ... +* 4~7 - reserved */ enum { -EROFS_INODE_LAYOUT_PLAIN , -EROFS_INODE_LAYOUT_COMPRESSION , -EROFS_INODE_LAYOUT_INLINE , +EROFS_INODE_FLAT_PLAIN , +EROFS_INODE_FLAT_COMPRESSION_LEGACY , +EROFS_INODE_FLAT_INLINE , +EROFS_INODE_FLAT_COMPRESSION ,EROFS_INODE_LAYOUT_MAX }; +static bool erofs_inode_is_data_compressed (unsigned int datamode) +{ +if (datamode ==EROFS_INODE_FLAT_COMPRESSION ) +return true; +return datamode ==EROFS_INODE_FLAT_COMPRESSION_LEGACY ; +} +/* bit definitions of inode i_advise */ #define EROFS_I_VERSION_BITS 1#define EROFS_I_DATA_MAPPING_BITS 3@@ -176,11 +186,37 @@ struct erofs_xattr_entry { sizeof(struct erofs_xattr_entry) + \ (entry)->e_name_len + le16_to_cpu((entry)->e_value_size)) -/* have to be aligned with 8 bytes on disk */ -struct erofs_extent_header { -__le32 eh_checksum; -__le32 eh_reserved[3 ]; -} __packed; +/* available compression algorithm types */ +enum { +Z_EROFS_COMPRESSION_LZ4 , +Z_EROFS_COMPRESSION_MAX +}; + +/* +* bit 0 : COMPACTED_2B indexes (0 - off; 1 - on) +* e.g. for 4k logical cluster size, 4B if compacted 2B is off; +* (4B) + 2B + (4B) if compacted 2B is on. +*/ +#define Z_EROFS_ADVISE_COMPACTED_2B_BIT 0 + +#define Z_EROFS_ADVISE_COMPACTED_2B (1 << Z_EROFS_ADVISE_COMPACTED_2B_BIT) + +struct z_erofs_map_header { +__le32 h_reserved1; +__le16 h_advise; +/* +* bit 0-3 : algorithm type of head 1 (logical cluster type 01); +* bit 4-7 : algorithm type of head 2 (logical cluster type 11). +*/ +__u8 h_algorithmtype; +/* +* bit 0-2 : logical cluster bits - 12, e.g. 0 for 4096; +* bit 3-4 : (physical - logical) cluster bits of head 1: +* For example, if logical clustersize = 4096, 1 for 8192. +* bit 5-7 : (physical - logical) cluster bits of head 2. +*/ +__u8 h_clusterbits; +};/* * Z_EROFS Variable-sized Logical Extent cluster type: @@ -270,7 +306,6 @@ static inline void erofs_check_ondisk_layout_definitions(void)BUILD_BUG_ON (sizeof (struct erofs_inode_v2 ) !=64 );BUILD_BUG_ON (sizeof (struct erofs_xattr_ibody_header ) !=12 );BUILD_BUG_ON (sizeof (struct erofs_xattr_entry ) !=4 ); -BUILD_BUG_ON (sizeof (struct erofs_extent_header ) !=16 );BUILD_BUG_ON (sizeof (struct z_erofs_vle_decompressed_index ) !=8 );BUILD_BUG_ON (sizeof (struct erofs_dirent ) !=12 );drivers/staging/erofs/inode.c +1
-4@@ -127,12 +127,9 @@ static int fill_inline_data(struct inode *inode, void *data, {struct erofs_vnode *vi =EROFS_V (inode);struct erofs_sb_info *sbi =EROFS_I_SB (inode); -const int mode = vi->datamode; - -DBG_BUGON (mode >=EROFS_INODE_LAYOUT_MAX );/* should be inode inline C */ -if (mode !=EROFS_INODE_LAYOUT_INLINE ) +if (!is_inode_flat_inline (inode))return 0 ;/* fast symlink (following ext4) */ drivers/staging/erofs/internal.h +3
-8@@ -382,19 +382,14 @@ static inline unsigned long inode_datablocks(struct inode *inode)return DIV_ROUND_UP (inode->i_size,EROFS_BLKSIZ ); } -static inline bool is_inode_layout_plain (struct inode *inode) -{ -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_PLAIN ; -} -static inline bool is_inode_layout_compression (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_COMPRESSION ; +return erofs_inode_is_data_compressed (EROFS_V (inode)->datamode); } -static inline bool is_inode_layout_inline (struct inode *inode) +static inline bool is_inode_flat_inline (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_INLINE ; +return EROFS_V (inode)->datamode ==EROFS_INODE_FLAT_INLINE ; }extern const struct super_operations erofs_sops;drivers/staging/erofs/unzip_vle.c +2
-2@@ -1643,8 +1643,8 @@ vle_extent_blkaddr(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blknr (iloc (sbi, vi->nid) + ofs); }
drivers/staging/erofs/unzip_vle.c vle_extent_blkoff(struct inode *inode, pgoff_t index) · line 1660
Quoted 7 lines
drivers/staging/erofs/unzip_vle.c +2
-2@@ -1656,8 +1656,8 @@ vle_extent_blkoff(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );
We can add one macro to wrap above offset (16) for better readability, anyway, this patch looks good to me. :)
Thanks,
Quoted 4 lines
drivers/staging/erofs/unzip_vle.c +0
-0return erofs_blkoff (iloc (sbi, vi->nid) + ofs); }
Re: [PATCH v2 1/8] staging: erofs: add compacted ondisk compression indexes
Hi Chao,
Quoted 209 lines · Chao Yu
Quoted 208 lines · Gao Xiang
This patch introduces new compacted compression indexes.
In contract to legacy compression indexes that
each 4k logical cluster has an 8-byte index,
compacted ondisk compression indexes will have
amortized 2 bytes for each 4k logical cluster (compacted 2B) amortized 4 bytes for each 4k logical cluster (compacted 4B)In detail, several continuous clusters will be encoded in a compacted pack with cluster types, offsets, and one blkaddr at the end of the pack to leave 4-byte margin for better decoding performance, as illustrated below:
_____________________________________________ |___ at _____ encoded bits __________|_ blkaddr _| 0 . amortized * vcnt . . . . amortized * vcnt - 4 . ..___________________.
|_type_|_clusterofs_|Note that compacted 2 / 4B should be aligned with 32 / 8 bytes in order to avoid each pack crossing page boundary.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>--- drivers/staging/erofs/data.c | 4 +-- drivers/staging/erofs/erofs_fs.h | 57 +++++++++++++++++++++++++------ drivers/staging/erofs/inode.c | 5 +-- drivers/staging/erofs/internal.h | 11 ++---- drivers/staging/erofs/unzip_vle.c | 8 ++--- 5 files changed, 56 insertions(+), 29 deletions(-)drivers/staging/erofs/data.c +2
-2@@ -124,7 +124,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,trace_erofs_map_blocks_flatmode_enter (inode, map, flags); nblocks =DIV_ROUND_UP (inode->i_size,PAGE_SIZE ); - lastblk = nblocks -is_inode_layout_inline (inode); + lastblk = nblocks -is_inode_flat_inline (inode);if (unlikely (offset >= inode->i_size)) {/* leave out-of-bound access unmapped */ @@ -139,7 +139,7 @@ static int erofs_map_blocks_flatmode(struct inode *inode,if (offset <blknr_to_addr (lastblk)) { map->m_pa =blknr_to_addr (vi->raw_blkaddr) + map->m_la; map->m_plen =blknr_to_addr (lastblk) - offset; - }else if (is_inode_layout_inline (inode)) { + }else if (is_inode_flat_inline (inode)) {/* 2 - inode inline B: inode, [xattrs], inline last blk... */ struct erofs_sb_info *sbi =EROFS_SB (inode->i_sb);drivers/staging/erofs/erofs_fs.h +46
-11@@ -49,19 +49,29 @@ struct erofs_super_block {* erofs inode data mapping: * 0 - inode plain without inline data A: * inode, [xattrs], ... | ... | no-holed data -* 1 - inode VLE compression B: +* 1 - inode VLE compression B (legacy): * inode, [xattrs], extents ... | ... * 2 - inode plain with inline data C: * inode, [xattrs], last_inline_data, ... | ... | no-holed data -* 3~7 - reserved +* 3 - inode compression D: +* inode, [xattrs], map_header, extents ... | ... +* 4~7 - reserved */ enum { -EROFS_INODE_LAYOUT_PLAIN , -EROFS_INODE_LAYOUT_COMPRESSION , -EROFS_INODE_LAYOUT_INLINE , +EROFS_INODE_FLAT_PLAIN , +EROFS_INODE_FLAT_COMPRESSION_LEGACY , +EROFS_INODE_FLAT_INLINE , +EROFS_INODE_FLAT_COMPRESSION ,EROFS_INODE_LAYOUT_MAX }; +static bool erofs_inode_is_data_compressed (unsigned int datamode) +{ +if (datamode ==EROFS_INODE_FLAT_COMPRESSION ) +return true; +return datamode ==EROFS_INODE_FLAT_COMPRESSION_LEGACY ; +} +/* bit definitions of inode i_advise */ #define EROFS_I_VERSION_BITS 1#define EROFS_I_DATA_MAPPING_BITS 3@@ -176,11 +186,37 @@ struct erofs_xattr_entry { sizeof(struct erofs_xattr_entry) + \ (entry)->e_name_len + le16_to_cpu((entry)->e_value_size)) -/* have to be aligned with 8 bytes on disk */ -struct erofs_extent_header { -__le32 eh_checksum; -__le32 eh_reserved[3 ]; -} __packed; +/* available compression algorithm types */ +enum { +Z_EROFS_COMPRESSION_LZ4 , +Z_EROFS_COMPRESSION_MAX +}; + +/* +* bit 0 : COMPACTED_2B indexes (0 - off; 1 - on) +* e.g. for 4k logical cluster size, 4B if compacted 2B is off; +* (4B) + 2B + (4B) if compacted 2B is on. +*/ +#define Z_EROFS_ADVISE_COMPACTED_2B_BIT 0 + +#define Z_EROFS_ADVISE_COMPACTED_2B (1 << Z_EROFS_ADVISE_COMPACTED_2B_BIT) + +struct z_erofs_map_header { +__le32 h_reserved1; +__le16 h_advise; +/* +* bit 0-3 : algorithm type of head 1 (logical cluster type 01); +* bit 4-7 : algorithm type of head 2 (logical cluster type 11). +*/ +__u8 h_algorithmtype; +/* +* bit 0-2 : logical cluster bits - 12, e.g. 0 for 4096; +* bit 3-4 : (physical - logical) cluster bits of head 1: +* For example, if logical clustersize = 4096, 1 for 8192. +* bit 5-7 : (physical - logical) cluster bits of head 2. +*/ +__u8 h_clusterbits; +};/* * Z_EROFS Variable-sized Logical Extent cluster type: @@ -270,7 +306,6 @@ static inline void erofs_check_ondisk_layout_definitions(void)BUILD_BUG_ON (sizeof (struct erofs_inode_v2 ) !=64 );BUILD_BUG_ON (sizeof (struct erofs_xattr_ibody_header ) !=12 );BUILD_BUG_ON (sizeof (struct erofs_xattr_entry ) !=4 ); -BUILD_BUG_ON (sizeof (struct erofs_extent_header ) !=16 );BUILD_BUG_ON (sizeof (struct z_erofs_vle_decompressed_index ) !=8 );BUILD_BUG_ON (sizeof (struct erofs_dirent ) !=12 );drivers/staging/erofs/inode.c +1
-4@@ -127,12 +127,9 @@ static int fill_inline_data(struct inode *inode, void *data, {struct erofs_vnode *vi =EROFS_V (inode);struct erofs_sb_info *sbi =EROFS_I_SB (inode); -const int mode = vi->datamode; - -DBG_BUGON (mode >=EROFS_INODE_LAYOUT_MAX );/* should be inode inline C */ -if (mode !=EROFS_INODE_LAYOUT_INLINE ) +if (!is_inode_flat_inline (inode))return 0 ;/* fast symlink (following ext4) */ drivers/staging/erofs/internal.h +3
-8@@ -382,19 +382,14 @@ static inline unsigned long inode_datablocks(struct inode *inode)return DIV_ROUND_UP (inode->i_size,EROFS_BLKSIZ ); } -static inline bool is_inode_layout_plain (struct inode *inode) -{ -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_PLAIN ; -} -static inline bool is_inode_layout_compression (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_COMPRESSION ; +return erofs_inode_is_data_compressed (EROFS_V (inode)->datamode); } -static inline bool is_inode_layout_inline (struct inode *inode) +static inline bool is_inode_flat_inline (struct inode *inode) { -return EROFS_V (inode)->datamode ==EROFS_INODE_LAYOUT_INLINE ; +return EROFS_V (inode)->datamode ==EROFS_INODE_FLAT_INLINE ; }extern const struct super_operations erofs_sops;drivers/staging/erofs/unzip_vle.c +2
-2@@ -1643,8 +1643,8 @@ vle_extent_blkaddr(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );return erofs_blknr (iloc (sbi, vi->nid) + ofs); }@@ -1656,8 +1656,8 @@ vle_extent_blkoff(struct inode *inode, pgoff_t index)struct erofs_vnode *vi =EROFS_V (inode);
Quoted 8 lines
drivers/staging/erofs/unzip_vle.c vle_extent_blkaddr(struct inode *inode, pgoff_t index) · line 1647
Quoted 5 lines
drivers/staging/erofs/unzip_vle.c +2
-2unsigned int ofs =Z_EROFS_VLE_EXTENT_ALIGN (vi->inode_isize + - vi->xattr_isize) +sizeof (struct erofs_extent_header ) + - index *sizeof (struct z_erofs_vle_decompressed_index ); + vi->xattr_isize) + +16 + index *sizeof (struct z_erofs_vle_decompressed_index );We can add one macro to wrap above offset (16) for better readability, anyway, this patch looks good to me. :)
OK, that is fine. I will add a marco to wrap up this number. Since currently we don't read or use the legacy extent header at all, I will make it obsoleted and remove the 16-byte legacy extent header directly.
However, in the future, I tend to add z_erofs_map_header support for the legacy index format as well in order to support large cluster size and more compression format, therefore it could be 8-byte z_erofs_map_header + 8-byte Z_EROFS_VLE_EXTENT_HEADER_PADDING and we need to add a requirement field for this expension. :)
Thanks,
Gao Xiang
Quoted 9 lines
Reviewed-by Chao Yu <yuchao0 at huawei.com>Thanks,
Quoted 4 lines
drivers/staging/erofs/unzip_vle.c +0
-0return erofs_blkoff (iloc (sbi, vi->nid) + ofs); }
PATCH v2 2/8 staging: erofs: add compacted compression indexes support
In reply to Gao XiangThis patch aims at compacted compression indexes:
1) cleanup z_erofs_map_blocks_iter and move into zmap.c;
2) add compacted 4/2B decoding support.
On kirin980 platform, sequential read is increased about 6% (725MiB/s -> 770MiB/s) on enwik9 dataset if compacted 2B feature is enabled.
--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/inode.c | 7 +- drivers/staging/erofs/internal.h | 18 +- drivers/staging/erofs/unzip_vle.c | 286 ------------------ drivers/staging/erofs/zmap.c | 462 ++++++++++++++++++++++++++++++ 5 files changed, 480 insertions(+), 295 deletions(-) create mode 100644 drivers/staging/erofs/zmap.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o
drivers/staging/erofs/inode.c +1 -6
@@ -210,12 +210,7 @@ static int fill_inode(struct inode *inode, int isdir) }if (is_inode_layout_compression (inode)) { -#ifdef CONFIG_EROFS_FS_ZIP - inode->i_mapping->a_ops = - &z_erofs_vle_normalaccess_aops; -#else - err = -ENOTSUPP ; -#endif + err =z_erofs_fill_inode (inode); goto out_unlock; }
drivers/staging/erofs/internal.h +16 -2
@@ -339,9 +339,11 @@ static inline erofs_off_t iloc(struct erofs_sb_info *sbi, erofs_nid_t nid)/* atomic flag definitions */ #define EROFS_V_EA_INITED_BIT 0 +#define EROFS_V_Z_INITED_BIT 1/* bitlock definitions (arranged in reverse order) */ #define EROFS_V_BL_XATTR_BIT (BITS_PER_LONG - 1) +#define EROFS_V_BL_Z_BIT (BITS_PER_LONG - 2)struct erofs_vnode {erofs_nid_t nid;
@@ -356,8 +358,17 @@ struct erofs_vnode {unsigned xattr_shared_count;unsigned *xattr_shared_xattrs; -erofs_blk_t raw_blkaddr; - +union { +erofs_blk_t raw_blkaddr; +#ifdef CONFIG_EROFS_FS_ZIP +struct { +unsigned short z_advise; +unsigned char z_algorithmtype[2 ]; +unsigned char z_logical_clusterbits; +unsigned char z_physical_clusterbits[2 ]; + }; +#endif + };/* the corresponding vfs inode */ struct inode vfs_inode; };
@@ -447,11 +458,14 @@ struct erofs_map_blocks {/* Flags used by erofs_map_blocks() */ #define EROFS_GET_BLOCKS_RAW 0x0001 +/* zmap.c */ #ifdef CONFIG_EROFS_FS_ZIP +int z_erofs_fill_inode (struct inode *inode);int z_erofs_map_blocks_iter (struct inode *inode,struct erofs_map_blocks *map,int flags);#else +static inline int z_erofs_fill_inode (struct inode *inode) {return -ENOTSUPP ; }static inline int z_erofs_map_blocks_iter (struct inode *inode,struct erofs_map_blocks *map,int flags)
drivers/staging/erofs/unzip_vle.c +0 -286
289 lines not on this page · raw message
drivers/staging/erofs/zmap.c new file +462 -0
462 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 2/8] staging: erofs: add compacted compression indexes support
Quoted 9 lines · Gao Xiang
This patch aims at compacted compression indexes:
1) cleanupz_erofs_map_blocks_iterand move intozmap.c;
2) add compacted 4/2B decoding support.On kirin980 platform, sequential read is increased about 6% (725MiB/s -> 770MiB/s) on enwik9 dataset if compacted 2B feature is enabled.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>
drivers/staging/erofs/zmap.c line 154
Quoted 10 lines
drivers/staging/erofs/zmap.c new file +10
-0+static int vle_legacy_load_cluster_from_disk (struct z_erofs_maprecorder *m, +unsigned long lcn) +{ +struct inode *const inode = m->inode; +struct erofs_vnode *const vi =EROFS_V (inode); +const erofs_off_t ibase =iloc (EROFS_I_SB (inode), vi->nid); +const erofs_off_t pos =Z_EROFS_VLE_EXTENT_ALIGN (ibase + + vi->inode_isize + + vi->xattr_isize) + +16 + lcn *sizeof (struct z_erofs_vle_decompressed_index );
use macro instead of raw number?
Thanks,
Re: [PATCH v2 2/8] staging: erofs: add compacted compression indexes support
Quoted 13 lines · Chao Yu
Quoted 9 lines · Gao Xiang
This patch aims at compacted compression indexes:
1) cleanupz_erofs_map_blocks_iterand move intozmap.c;
2) add compacted 4/2B decoding support.On kirin980 platform, sequential read is increased about 6% (725MiB/s -> 770MiB/s) on enwik9 dataset if compacted 2B feature is enabled.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>Reviewed-by Chao Yu <yuchao0 at huawei.com>
Quoted 12 lines
drivers/staging/erofs/zmap.c line 154
Quoted 10 lines
drivers/staging/erofs/zmap.c new file +10
-0+static int vle_legacy_load_cluster_from_disk (struct z_erofs_maprecorder *m, +unsigned long lcn) +{ +struct inode *const inode = m->inode; +struct erofs_vnode *const vi =EROFS_V (inode); +const erofs_off_t ibase =iloc (EROFS_I_SB (inode), vi->nid); +const erofs_off_t pos =Z_EROFS_VLE_EXTENT_ALIGN (ibase + + vi->inode_isize + + vi->xattr_isize) + +16 + lcn *sizeof (struct z_erofs_vle_decompressed_index );use macro instead of raw number?
OK, I will wrap it up in patch v3 too.
Thanks,
Gao Xiang
Quoted 3 lines
Thanks,
PATCH v2 3/8 staging: erofs: move per-CPU buffers implementation to utils.c
In reply to Gao XiangThis patch moves per-CPU buffers to utils.c in order for the upcoming generic decompression framework to use it.
Note that I tried to use generic per-CPU buffer or per-CPU page approaches to clean up further, but obvious performanace regression (about 2% for sequential read) was observed.
Therefore let's leave it as it is instead, just move to utils.c and I'll try to dig into the root cause later.
--- drivers/staging/erofs/internal.h | 26 ++++++++++++++++++++++ drivers/staging/erofs/unzip_vle.c | 5 ++--- drivers/staging/erofs/unzip_vle.h | 4 +--- drivers/staging/erofs/unzip_vle_lz4.c | 31 ++++++++------------------- drivers/staging/erofs/utils.c | 12 +++++++++++ 5 files changed, 50 insertions(+), 28 deletions(-)
drivers/staging/erofs/internal.h +26 -0
@@ -321,6 +321,16 @@ static inline void z_erofs_exit_zip_subsystem(void) {}/* page count of a compressed cluster */ #define erofs_clusterpages (sbi) ((1 << (sbi)->clusterbits) / PAGE_SIZE) +#define Z_EROFS_NR_INLINE_PAGEVECS 3 + +#if (Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_NR_INLINE_PAGEVECS ) +#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES +#else +#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_NR_INLINE_PAGEVECS +#endif + +#else +#define EROFS_PCPUBUF_NR_PAGES 0#endif typedef u64 erofs_off_t ;
@@ -608,6 +618,22 @@ static inline void erofs_vunmap(const void *mem, unsigned int count)extern struct shrinker erofs_shrinker_info;struct page *erofs_allocpage (struct list_head *pool,gfp_t gfp); + +#if (EROFS_PCPUBUF_NR_PAGES >0 ) +void *erofs_get_pcpubuf (unsigned int pagenr); +#define erofs_put_pcpubuf (buf) do { \ + (void)&(buf); \ + preempt_enable(); \ +} while (0) +#else +static inline void *erofs_get_pcpubuf (unsigned int pagenr) +{ +return ERR_PTR (-ENOTSUPP ); +} + +#define erofs_put_pcpubuf (buf) do {} while (0) +#endif +void erofs_register_super (struct super_block *sb);void erofs_unregister_super (struct super_block *sb);
drivers/staging/erofs/unzip_vle.c +2 -3
@@ -552,8 +552,7 @@ static int z_erofs_vle_work_iter_begin(struct z_erofs_vle_work_builder *builder,if (IS_ERR (work))return PTR_ERR (work); got_it: -z_erofs_pagevec_ctor_init (&builder->vector, -Z_EROFS_VLE_INLINE_PAGEVECS , +z_erofs_pagevec_ctor_init (&builder->vector,Z_EROFS_NR_INLINE_PAGEVECS , work->pagevec, work->vcnt);if (builder->role >=Z_EROFS_VLE_WORK_PRIMARY ) {
@@ -936,7 +935,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,for (i =0 ; i < nr_pages; ++i) pages[i] =NULL ; -z_erofs_pagevec_ctor_init (&ctor,Z_EROFS_VLE_INLINE_PAGEVECS , +z_erofs_pagevec_ctor_init (&ctor,Z_EROFS_NR_INLINE_PAGEVECS , work->pagevec,0 );for (i =0 ; i < work->vcnt; ++i) {
drivers/staging/erofs/unzip_vle.h +1 -3
@@ -44,8 +44,6 @@ static inline bool z_erofs_gather_if_stagingpage(struct list_head *page_pool,* */ -#define Z_EROFS_VLE_INLINE_PAGEVECS 3 -struct z_erofs_vle_work {struct mutex lock;
@@ -58,7 +56,7 @@ struct z_erofs_vle_work {union {/* L: pagevec */ -erofs_vtptr_t pagevec[Z_EROFS_VLE_INLINE_PAGEVECS ]; +erofs_vtptr_t pagevec[Z_EROFS_NR_INLINE_PAGEVECS ];struct rcu_head rcu; }; };
drivers/staging/erofs/unzip_vle_lz4.c +9 -22
@@ -34,16 +34,6 @@ static int z_erofs_unzip_lz4(void *in, void *out, size_t inlen, size_t outlen)return -EIO ; } -#if Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_VLE_INLINE_PAGEVECS -#define EROFS_PERCPU_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES -#else -#define EROFS_PERCPU_NR_PAGES Z_EROFS_VLE_INLINE_PAGEVECS -#endif - -static struct { -char data[PAGE_SIZE *EROFS_PERCPU_NR_PAGES ]; -} erofs_pcpubuf[NR_CPUS ]; - int z_erofs_vle_plain_copy(struct page **compressed_pages, unsigned int clusterpages, struct page **pages,
@@ -56,8 +46,7 @@ int z_erofs_vle_plain_copy(struct page **compressed_pages,char *percpu_data;bool mirrored[Z_EROFS_CLUSTER_MAX_PAGES ] = {0 }; -preempt_disable (); - percpu_data = erofs_pcpubuf[smp_processor_id ()].data; + percpu_data =erofs_get_pcpubuf (0 ); j =0 ;for (i =0 ; i < nr_pages; j = i++) {
@@ -117,7 +106,7 @@ int z_erofs_vle_plain_copy(struct page **compressed_pages,if (src && !mirrored[j])kunmap_atomic (src); -preempt_enable (); +erofs_put_pcpubuf (percpu_data);return 0 ; }
@@ -131,7 +120,7 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,unsigned int nr_pages, i, j;int ret; -if (outlen + pageofs >EROFS_PERCPU_NR_PAGES *PAGE_SIZE ) +if (outlen + pageofs >EROFS_PCPUBUF_NR_PAGES *PAGE_SIZE )return -ENOTSUPP ; nr_pages =DIV_ROUND_UP (outlen + pageofs,PAGE_SIZE );
@@ -144,8 +133,7 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages,return -ENOMEM ; } -preempt_disable (); - vout = erofs_pcpubuf[smp_processor_id ()].data; + vout =erofs_get_pcpubuf (0 ); ret =z_erofs_unzip_lz4 (vin, vout + pageofs, clusterpages *PAGE_SIZE , outlen);
@@ -174,7 +162,7 @@ int z_erofs_vle_unzip_fast_percpu(struct page **compressed_pages, } out: -preempt_enable (); +erofs_put_pcpubuf (vout);if (clusterpages ==1 )kunmap_atomic (vin);
@@ -196,8 +184,7 @@ int z_erofs_vle_unzip_vmap(struct page **compressed_pages,int ret;if (overlapped) { -preempt_disable (); - vin = erofs_pcpubuf[smp_processor_id ()].data; + vin =erofs_get_pcpubuf (0 );for (i =0 ; i < clusterpages; ++i) {void *t =kmap_atomic (compressed_pages[i]);
@@ -216,13 +203,13 @@ int z_erofs_vle_unzip_vmap(struct page **compressed_pages,if (ret >0 ) ret =0 ; -if (!overlapped) { +if (overlapped) { +erofs_put_pcpubuf (vin); + }else {if (clusterpages ==1 )kunmap_atomic (vin);else erofs_vunmap (vin, clusterpages); - }else { -preempt_enable (); }return ret; }
drivers/staging/erofs/utils.c +12 -0
@@ -27,6 +27,18 @@ struct page *erofs_allocpage(struct list_head *pool, gfp_t gfp)return page; } +#if (EROFS_PCPUBUF_NR_PAGES >0 ) +static struct { +u8 data[PAGE_SIZE *EROFS_PCPUBUF_NR_PAGES ]; +}____cacheline_aligned_in_smp erofs_pcpubuf[NR_CPUS ]; + +void *erofs_get_pcpubuf (unsigned int pagenr) +{ +preempt_disable (); +return &erofs_pcpubuf[smp_processor_id ()].data[pagenr *PAGE_SIZE ]; +} +#endif +/* global shrink count (for all mounted EROFS instances) */ static atomic_long_t erofs_global_shrink_cnt;
Signature · 1 line
2.17.1
Re: [PATCH v2 3/8] staging: erofs: move per-CPU buffers implementation to utils.c
Quoted 4 lines · Gao Xiang
drivers/staging/erofs/internal.h +4
-0+static inline void *erofs_get_pcpubuf (unsigned int pagenr) +{ +return ERR_PTR (-ENOTSUPP ); +}
[snip]
drivers/staging/erofs/unzip_vle_lz4.c int z_erofs_vle_plain_copy(struct page **compressed_pages, · line 49
Quoted 1 line
drivers/staging/erofs/unzip_vle_lz4.c +1
-0+ percpu_data =erofs_get_pcpubuf (0 );
If erofs_get_pcpubuf may return error once EROFS_PCPUBUF_NR_PAGES equals to zero, we'd better to check the error number here.
Thanks,
Re: [PATCH v2 3/8] staging: erofs: move per-CPU buffers implementation to utils.c
Hi Chao,
Quoted 8 lines · Chao Yu
Quoted 4 lines · Gao Xiang
drivers/staging/erofs/internal.h +4
-0+static inline void *erofs_get_pcpubuf (unsigned int pagenr) +{ +return ERR_PTR (-ENOTSUPP ); +}[snip]
Quoted 4 lines
drivers/staging/erofs/unzip_vle_lz4.c int z_erofs_vle_plain_copy(struct page **compressed_pages, · line 49
Quoted 1 line
drivers/staging/erofs/unzip_vle_lz4.c +1
-0+ percpu_data =erofs_get_pcpubuf (0 );If
erofs_get_pcpubufmay return error onceEROFS_PCPUBUF_NR_PAGESequals to zero, we'd better to check the error number here.
decompressor.c will be built-in only when decompression is enabled and if decompression is enabled EROFS_PCPUBUF_NR_PAGES is not zero.
But I think introducing some error handling logic is not bad as well. Will fix in the next version.
Thanks,
Gao Xiang
Quoted 3 lines
Thanks,
PATCH v2 4/8 staging: erofs: move stagingpage operations to compress.h
In reply to Gao Xiangstagingpages are behaved as bounce pages for temporary use. Move to compress.h since the upcoming decompressor will allocate stagingpages as well.
--- drivers/staging/erofs/compress.h | 40 +++++++++++++++++++++++++++++++ drivers/staging/erofs/unzip_vle.c | 11 +++++---- drivers/staging/erofs/unzip_vle.h | 20 ---------------- 3 files changed, 46 insertions(+), 25 deletions(-) create mode 100644 drivers/staging/erofs/compress.h
drivers/staging/erofs/compress.h new file +40 -0
@@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* linux/drivers/staging/erofs/compress.h +* +* Copyright (C) 2019 HUAWEI, Inc. +* http://www.huawei.com/ +* Created by Gao Xiang <gaoxiang25 at huawei.com> +*/ +#ifndef __EROFS_FS_COMPRESS_H +#define __EROFS_FS_COMPRESS_H + +/* +* - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - +* used to mark temporary allocated pages from other +* file/cached pages and NULL mapping pages. +*/ +#define Z_EROFS_MAPPING_STAGING ((void *)0x5A110C8D) + +/* check if a page is marked as staging */ +static inline bool z_erofs_page_is_staging (struct page *page) +{ +return page->mapping ==Z_EROFS_MAPPING_STAGING ; +} + +static inline bool z_erofs_put_stagingpage (struct list_head *pagepool, +struct page *page) +{ +if (!z_erofs_page_is_staging (page)) +return false; + +/* staging pages should not be used by others at the same time */ +if (page_ref_count (page) >1 ) +put_page (page); +else +list_add (&page->lru, pagepool); +return true; +} + +#endif +
drivers/staging/erofs/unzip_vle.c +6 -5
@@ -11,6 +11,7 @@* distribution for more details. */ #include "unzip_vle.h" +#include "compress.h" #include <linux/prefetch.h> #include <trace/events/erofs.h>
@@ -855,7 +856,7 @@ static inline void z_erofs_vle_read_endio(struct bio *bio)DBG_BUGON (PageUptodate (page));DBG_BUGON (!page->mapping); -if (unlikely (!sbi && !z_erofs_is_stagingpage (page))) { +if (unlikely (!sbi && !z_erofs_page_is_staging (page))) { sbi =EROFS_SB (page->mapping->host->i_sb);if (time_to_inject (sbi,FAULT_READ_IO )) {
@@ -947,7 +948,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page);DBG_BUGON (!page->mapping); -if (z_erofs_gather_if_stagingpage (page_pool, page)) +if (z_erofs_put_stagingpage (page_pool, page))continue ;if (page_type ==Z_EROFS_VLE_PAGE_TYPE_HEAD )
@@ -977,7 +978,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page);DBG_BUGON (!page->mapping); -if (!z_erofs_is_stagingpage (page)) { +if (!z_erofs_page_is_staging (page)) {if (erofs_page_is_managed (sbi, page)) {if (unlikely (!PageUptodate (page))) err = -EIO ;
@@ -1055,7 +1056,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,continue ;/* recycle all individual staging pages */ - (void )z_erofs_gather_if_stagingpage (page_pool, page); + (void )z_erofs_put_stagingpage (page_pool, page);WRITE_ONCE (compressed_pages[i],NULL ); }
@@ -1068,7 +1069,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,DBG_BUGON (!page->mapping);/* recycle all individual staging pages */ -if (z_erofs_gather_if_stagingpage (page_pool, page)) +if (z_erofs_put_stagingpage (page_pool, page))continue ;if (unlikely (err <0 ))
drivers/staging/erofs/unzip_vle.h +0 -20
@@ -16,26 +16,6 @@#include "internal.h" #include "unzip_pagevec.h" -/* -* - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - -* used for temporary allocated pages (via erofs_allocpage), -* in order to seperate those from NULL mapping (eg. truncated pages) -*/ -#define Z_EROFS_MAPPING_STAGING ((void *)0x5A110C8D) - -#define z_erofs_is_stagingpage (page) \ - ((page)->mapping == Z_EROFS_MAPPING_STAGING) - -static inline bool z_erofs_gather_if_stagingpage (struct list_head *page_pool, -struct page *page) -{ -if (z_erofs_is_stagingpage (page)) { -list_add (&page->lru, page_pool); -return true; - } -return false; -} -/* * Structure fields follow one of the following exclusion rules. *
Signature · 1 line
2.17.1
Re: [PATCH v2 4/8] staging: erofs: move stagingpage operations to compress.h
Quoted 5 lines · Gao Xiang
stagingpages are behaved as bounce pages for temporary use. Move to
compress.hsince the upcoming decompressor will allocate stagingpages as well.Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>
Thanks,
PATCH v2 5/8 staging: erofs: introduce generic decompression backend
In reply to Gao XiangThis patch adds a new generic decompression framework in order to replace the old LZ4-specific decompression code.
Even though LZ4 is still the only supported algorithm, yet it is more cleaner and easy to integrate new algorithm than the old almost hard-coded decompression backend.
--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 21 ++ drivers/staging/erofs/decompressor.c | 307 +++++++++++++++++++++++++++ 3 files changed, 329 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/erofs/decompressor.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.o
drivers/staging/erofs/compress.h +21 -0
@@ -9,6 +9,24 @@#ifndef __EROFS_FS_COMPRESS_H#define __EROFS_FS_COMPRESS_H +#include "internal.h" + +enum { +Z_EROFS_COMPRESSION_SHIFTED =Z_EROFS_COMPRESSION_MAX , +Z_EROFS_COMPRESSION_RUNTIME_MAX +}; + +struct z_erofs_decompress_req { +struct page **in, **out; + +unsigned short pageofs_out; +unsigned int inputsize, outputsize; + +/* indicate the algorithm will be used for decompression */ +unsigned int alg; +bool inplace_io, partial_decoding; +}; +/* * - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - * used to mark temporary allocated pages from other
@@ -36,5 +54,8 @@ static inline bool z_erofs_put_stagingpage(struct list_head *pagepool, return true; } +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +#endif
drivers/staging/erofs/decompressor.c new file +307 -0
307 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 5/8] staging: erofs: introduce generic decompression backend
Quoted 136 lines · Gao Xiang
This patch adds a new generic decompression framework in order to replace the old LZ4-specific decompression code.
Even though LZ4 is still the only supported algorithm, yet it is more cleaner and easy to integrate new algorithm than the old almost hard-coded decompression backend.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 21 ++ drivers/staging/erofs/decompressor.c | 307 +++++++++++++++++++++++++++ 3 files changed, 329 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/erofs/decompressor.cdrivers/staging/erofs/Makefile +1
-1@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.odrivers/staging/erofs/compress.h +21
-0@@ -9,6 +9,24 @@#ifndef __EROFS_FS_COMPRESS_H#define __EROFS_FS_COMPRESS_H +#include "internal.h" + +enum { +Z_EROFS_COMPRESSION_SHIFTED =Z_EROFS_COMPRESSION_MAX , +Z_EROFS_COMPRESSION_RUNTIME_MAX +}; + +struct z_erofs_decompress_req { +struct page **in, **out; + +unsigned short pageofs_out; +unsigned int inputsize, outputsize; + +/* indicate the algorithm will be used for decompression */ +unsigned int alg; +bool inplace_io, partial_decoding; +}; +/* * - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - * used to mark temporary allocated pages from other @@ -36,5 +54,8 @@ static inline bool z_erofs_put_stagingpage(struct list_head *pagepool, return true; } +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +#endif drivers/staging/erofs/decompressor.c new file +66
-0@@ -0,0 +1,307 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* linux/drivers/staging/erofs/decompressor.c +* +* Copyright (C) 2019 HUAWEI, Inc. +* http://www.huawei.com/ +* Created by Gao Xiang <gaoxiang25 at huawei.com> +*/ +#include "compress.h" +#include <linux/lz4.h> + +#ifndef LZ4_DISTANCE_MAX /* history window size */ +#define LZ4_DISTANCE_MAX 65535/* set to maximum value by default */ +#endif + +#define LZ4_MAX_DISTANCE_PAGES DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) + +struct z_erofs_decompressor { +/* +* if destpages have sparsed pages, fill them with bounce pages. +* it also check whether destpages indicate continuous physical memory. +*/ +int (*prepare_destpages)(struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +int (*decompress)(struct z_erofs_decompress_req *rq,u8 *out); +char *name; +}; + +static int lz4_prepare_destpages (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nr = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +struct page *availables[LZ4_MAX_DISTANCE_PAGES ] = {NULL }; +unsigned long unused[DIV_ROUND_UP (LZ4_MAX_DISTANCE_PAGES , +BITS_PER_LONG )] = {0 }; +void *kaddr =NULL ; +unsigned int i, j, k; + +for (i =0 ; i < nr; ++i) { +struct page *const page = rq->out[i]; + + j = i & (LZ4_MAX_DISTANCE_PAGES -1 ); +if (availables[j]) +__set_bit (j, unused); + +if (page) { +if (kaddr) { +if (kaddr +PAGE_SIZE ==page_address (page)) + kaddr +=PAGE_SIZE ; +else + kaddr =NULL ; + }else if (!i) { + kaddr =page_address (page); + } +continue ; + } + kaddr =NULL ; + + k =find_first_bit (unused,LZ4_MAX_DISTANCE_PAGES ); +if (k <LZ4_MAX_DISTANCE_PAGES ) { + j = k; +get_page (availables[j]); + }else { +DBG_BUGON (availables[j]); +
drivers/staging/erofs/decompressor.c line 76
Quoted 10 lines
drivers/staging/erofs/decompressor.c new file +10
-0+if (!list_empty (pagepool)) { + availables[j] =lru_to_page (pagepool); +list_del (&availables[j]->lru); +DBG_BUGON (page_ref_count (availables[j]) !=1 ); + }else { + availables[j] =alloc_pages (GFP_KERNEL ,0 ); +if (!availables[j]) +return -ENOMEM ; + } + availables[j]->mapping =Z_EROFS_MAPPING_STAGING ;
Could we use __stagingpage_alloc() instead opened codes, there is something different in between them though.
Thanks,
Quoted 232 lines
drivers/staging/erofs/decompressor.c new file +231
-0+ } + rq->out[i] = availables[j]; +__clear_bit (j, unused); + } +return kaddr ?1 :0 ; +} + +static void *generic_copy_inplace_data (struct z_erofs_decompress_req *rq, +u8 *src,unsigned int pageofs_in) +{ +/* +* if in-place decompression is ongoing, those decompressed +* pages should be copied in order to avoid being overlapped. +*/ +struct page **in = rq->in; +u8 *const tmp =erofs_get_pcpubuf (0 ); +u8 *tmpp = tmp; +unsigned int inlen = rq->inputsize - pageofs_in; +unsigned int count =min_t (uint, inlen,PAGE_SIZE - pageofs_in); + +while (tmpp < tmp + inlen) { +if (!src) + src =kmap_atomic (*in); +memcpy (tmpp, src + pageofs_in, count); +kunmap_atomic (src); + src =NULL ; + tmpp += count; + pageofs_in =0 ; + count =PAGE_SIZE ; + ++in; + } +return tmp; +} + +static int lz4_decompress (struct z_erofs_decompress_req *rq,u8 *out) +{ +unsigned int inputmargin, inlen; +u8 *src; +bool copied; +int ret; + +if (rq->inputsize >PAGE_SIZE ) +return -ENOTSUPP ; + + src =kmap_atomic (*rq->in); + inputmargin =0 ; +while (!src[inputmargin & ~PAGE_MASK ]) +if (!(++inputmargin & ~PAGE_MASK )) +break ; + +if (inputmargin >= rq->inputsize) { +kunmap_atomic (src); +return -EIO ; + } + + copied = false; + inlen = rq->inputsize - inputmargin; +if (rq->inplace_io) { + src =generic_copy_inplace_data (rq, src, inputmargin); + inputmargin =0 ; + copied = true; + } + + ret =LZ4_decompress_safe_partial (src + inputmargin, out, + inlen, rq->outputsize, + rq->outputsize); +if (ret <0 ) { +errln ("%s, failed to decompress, in[%p, %u, %u] out[%p, %u]" , + __func__, src + inputmargin, inlen, inputmargin, + out, rq->outputsize); +WARN_ON (1 ); +print_hex_dump (KERN_DEBUG ,"[ in]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , src + inputmargin, inlen, true); +print_hex_dump (KERN_DEBUG ,"[out]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , out, rq->outputsize, true); + ret = -EIO ; + } + +if (copied) +erofs_put_pcpubuf (src); +else +kunmap_atomic (src); +return ret; +} + +static struct z_erofs_decompressor decompressors[] = { + [Z_EROFS_COMPRESSION_SHIFTED ] = { + .name ="shifted" + }, + [Z_EROFS_COMPRESSION_LZ4 ] = { + .prepare_destpages = lz4_prepare_destpages, + .decompress = lz4_decompress, + .name ="lz4" + }, +}; + +static void copy_from_pcpubuf (struct page **out,const char *dst, +unsigned short pageofs_out, +unsigned int outputsize) +{ +const char *end = dst + outputsize; +const unsigned int righthalf =PAGE_SIZE - pageofs_out; +const char *cur = dst - pageofs_out; + +while (cur < end) { +struct page *const page = *out++; + +if (page) { +char *buf =kmap_atomic (page); + +if (cur >= dst) { +memcpy (buf, cur,min_t (uint,PAGE_SIZE , + end - cur)); + }else { +memcpy (buf + pageofs_out, cur + pageofs_out, +min_t (uint, righthalf, end - cur)); + } +kunmap_atomic (buf); + } + cur +=PAGE_SIZE ; + } +} + +static int decompress_generic (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const struct z_erofs_decompressor *alg = decompressors + rq->alg; +unsigned int dst_maptype; +void *dst; +int ret; + +if (nrpages_out ==1 && !rq->inplace_io) { +DBG_BUGON (!*rq->out); + dst =kmap_atomic (*rq->out); + dst_maptype =0 ; + goto dstmap_out; + } + +/* +* For the case of small output size (especially much less +* than PAGE_SIZE), memcpy the decompressed data rather than +* compressed data is preferred. +*/ +if (rq->outputsize <=PAGE_SIZE *7 /8 ) { + dst =erofs_get_pcpubuf (0 ); + + rq->inplace_io = false; + ret = alg->decompress (rq, dst); +if (!ret) +copy_from_pcpubuf (rq->out, dst, rq->pageofs_out, + rq->outputsize); + +erofs_put_pcpubuf (dst); +return ret; + } + + ret = alg->prepare_destpages (rq, pagepool); +if (ret <0 ) { +return ret; + }else if (ret) { + dst =page_address (*rq->out); + dst_maptype =1 ; + goto dstmap_out; + } + + dst =erofs_vmap (rq->out, nrpages_out); +if (!dst) +return -ENOMEM ; + dst_maptype =2 ; + +dstmap_out: + ret = alg->decompress (rq, dst + rq->pageofs_out); + +if (!dst_maptype) +kunmap_atomic (dst); +else if (dst_maptype ==2 ) +erofs_vunmap (dst, nrpages_out); +return ret; +} + +static int shifted_decompress (const struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const unsigned int righthalf =PAGE_SIZE - rq->pageofs_out; +unsigned char *src, *dst; + +if (nrpages_out >2 ) { +DBG_BUGON (1 ); +return -EIO ; + } + +if (rq->out[0 ] == *rq->in) { +DBG_BUGON (nrpages_out !=1 ); +return 0 ; + } + + src =kmap_atomic (*rq->in); +if (!rq->out[0 ]) { + dst =NULL ; + }else { + dst =kmap_atomic (rq->out[0 ]); +memcpy (dst + rq->pageofs_out, src, righthalf); + } + +if (rq->out[1 ] == *rq->in) { +memmove (src, src + righthalf, rq->pageofs_out); + }else if (nrpages_out ==2 ) { +if (dst) +kunmap_atomic (dst); +DBG_BUGON (!rq->out[1 ]); + dst =kmap_atomic (rq->out[1 ]); +memcpy (dst, src + righthalf, rq->pageofs_out); + } +if (dst) +kunmap_atomic (dst); +kunmap_atomic (src); +return 0 ; +} + +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +if (rq->alg ==Z_EROFS_COMPRESSION_SHIFTED ) +return shifted_decompress (rq, pagepool); +return decompress_generic (rq, pagepool); +} +
Re: [PATCH v2 5/8] staging: erofs: introduce generic decompression backend
Hi Chao,
Quoted 142 lines · Chao Yu
Quoted 141 lines · Gao Xiang
This patch adds a new generic decompression framework in order to replace the old LZ4-specific decompression code.
Even though LZ4 is still the only supported algorithm, yet it is more cleaner and easy to integrate new algorithm than the old almost hard-coded decompression backend.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 21 ++ drivers/staging/erofs/decompressor.c | 307 +++++++++++++++++++++++++++ 3 files changed, 329 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/erofs/decompressor.cdrivers/staging/erofs/Makefile +1
-1@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.odrivers/staging/erofs/compress.h +21
-0@@ -9,6 +9,24 @@#ifndef __EROFS_FS_COMPRESS_H#define __EROFS_FS_COMPRESS_H +#include "internal.h" + +enum { +Z_EROFS_COMPRESSION_SHIFTED =Z_EROFS_COMPRESSION_MAX , +Z_EROFS_COMPRESSION_RUNTIME_MAX +}; + +struct z_erofs_decompress_req { +struct page **in, **out; + +unsigned short pageofs_out; +unsigned int inputsize, outputsize; + +/* indicate the algorithm will be used for decompression */ +unsigned int alg; +bool inplace_io, partial_decoding; +}; +/* * - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - * used to mark temporary allocated pages from other @@ -36,5 +54,8 @@ static inline bool z_erofs_put_stagingpage(struct list_head *pagepool, return true; } +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +#endif drivers/staging/erofs/decompressor.c new file +71
-0@@ -0,0 +1,307 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* linux/drivers/staging/erofs/decompressor.c +* +* Copyright (C) 2019 HUAWEI, Inc. +* http://www.huawei.com/ +* Created by Gao Xiang <gaoxiang25 at huawei.com> +*/ +#include "compress.h" +#include <linux/lz4.h> + +#ifndef LZ4_DISTANCE_MAX /* history window size */ +#define LZ4_DISTANCE_MAX 65535/* set to maximum value by default */ +#endif + +#define LZ4_MAX_DISTANCE_PAGES DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) + +struct z_erofs_decompressor { +/* +* if destpages have sparsed pages, fill them with bounce pages. +* it also check whether destpages indicate continuous physical memory. +*/ +int (*prepare_destpages)(struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +int (*decompress)(struct z_erofs_decompress_req *rq,u8 *out); +char *name; +}; + +static int lz4_prepare_destpages (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nr = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +struct page *availables[LZ4_MAX_DISTANCE_PAGES ] = {NULL }; +unsigned long unused[DIV_ROUND_UP (LZ4_MAX_DISTANCE_PAGES , +BITS_PER_LONG )] = {0 }; +void *kaddr =NULL ; +unsigned int i, j, k; + +for (i =0 ; i < nr; ++i) { +struct page *const page = rq->out[i]; + + j = i & (LZ4_MAX_DISTANCE_PAGES -1 ); +if (availables[j]) +__set_bit (j, unused); + +if (page) { +if (kaddr) { +if (kaddr +PAGE_SIZE ==page_address (page)) + kaddr +=PAGE_SIZE ; +else + kaddr =NULL ; + }else if (!i) { + kaddr =page_address (page); + } +continue ; + } + kaddr =NULL ; + + k =find_first_bit (unused,LZ4_MAX_DISTANCE_PAGES ); +if (k <LZ4_MAX_DISTANCE_PAGES ) { + j = k; +get_page (availables[j]); + }else { +DBG_BUGON (availables[j]); + +if (!list_empty (pagepool)) { + availables[j] =lru_to_page (pagepool); +list_del (&availables[j]->lru); +DBG_BUGON (page_ref_count (availables[j]) !=1 ); + }else {
Quoted 8 lines
drivers/staging/erofs/decompressor.c line 76
Quoted 5 lines
drivers/staging/erofs/decompressor.c new file +5
-0+ availables[j] =alloc_pages (GFP_KERNEL ,0 ); +if (!availables[j]) +return -ENOMEM ; + } + availables[j]->mapping =Z_EROFS_MAPPING_STAGING ;Could we use
__stagingpage_alloc()instead opened codes, there is something different in between them though.
It was written "on propose" since the problem is that currently __stagingpage_alloc() will allocate all pages in GFP_NOFAIL case, but memory allocation failure is accepted here, therefore I open code to aim at introducing new decompression backend.
I will submit another patch after this series to clean up all staging page allocation cases later, which was already on my scheduling list. :)
Thanks,
Gao Xiang
Quoted 237 lines
Reviewed-by Chao Yu <yuchao0 at huawei.com>Thanks,
Quoted 232 lines
drivers/staging/erofs/decompressor.c new file +231
-0+ } + rq->out[i] = availables[j]; +__clear_bit (j, unused); + } +return kaddr ?1 :0 ; +} + +static void *generic_copy_inplace_data (struct z_erofs_decompress_req *rq, +u8 *src,unsigned int pageofs_in) +{ +/* +* if in-place decompression is ongoing, those decompressed +* pages should be copied in order to avoid being overlapped. +*/ +struct page **in = rq->in; +u8 *const tmp =erofs_get_pcpubuf (0 ); +u8 *tmpp = tmp; +unsigned int inlen = rq->inputsize - pageofs_in; +unsigned int count =min_t (uint, inlen,PAGE_SIZE - pageofs_in); + +while (tmpp < tmp + inlen) { +if (!src) + src =kmap_atomic (*in); +memcpy (tmpp, src + pageofs_in, count); +kunmap_atomic (src); + src =NULL ; + tmpp += count; + pageofs_in =0 ; + count =PAGE_SIZE ; + ++in; + } +return tmp; +} + +static int lz4_decompress (struct z_erofs_decompress_req *rq,u8 *out) +{ +unsigned int inputmargin, inlen; +u8 *src; +bool copied; +int ret; + +if (rq->inputsize >PAGE_SIZE ) +return -ENOTSUPP ; + + src =kmap_atomic (*rq->in); + inputmargin =0 ; +while (!src[inputmargin & ~PAGE_MASK ]) +if (!(++inputmargin & ~PAGE_MASK )) +break ; + +if (inputmargin >= rq->inputsize) { +kunmap_atomic (src); +return -EIO ; + } + + copied = false; + inlen = rq->inputsize - inputmargin; +if (rq->inplace_io) { + src =generic_copy_inplace_data (rq, src, inputmargin); + inputmargin =0 ; + copied = true; + } + + ret =LZ4_decompress_safe_partial (src + inputmargin, out, + inlen, rq->outputsize, + rq->outputsize); +if (ret <0 ) { +errln ("%s, failed to decompress, in[%p, %u, %u] out[%p, %u]" , + __func__, src + inputmargin, inlen, inputmargin, + out, rq->outputsize); +WARN_ON (1 ); +print_hex_dump (KERN_DEBUG ,"[ in]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , src + inputmargin, inlen, true); +print_hex_dump (KERN_DEBUG ,"[out]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , out, rq->outputsize, true); + ret = -EIO ; + } + +if (copied) +erofs_put_pcpubuf (src); +else +kunmap_atomic (src); +return ret; +} + +static struct z_erofs_decompressor decompressors[] = { + [Z_EROFS_COMPRESSION_SHIFTED ] = { + .name ="shifted" + }, + [Z_EROFS_COMPRESSION_LZ4 ] = { + .prepare_destpages = lz4_prepare_destpages, + .decompress = lz4_decompress, + .name ="lz4" + }, +}; + +static void copy_from_pcpubuf (struct page **out,const char *dst, +unsigned short pageofs_out, +unsigned int outputsize) +{ +const char *end = dst + outputsize; +const unsigned int righthalf =PAGE_SIZE - pageofs_out; +const char *cur = dst - pageofs_out; + +while (cur < end) { +struct page *const page = *out++; + +if (page) { +char *buf =kmap_atomic (page); + +if (cur >= dst) { +memcpy (buf, cur,min_t (uint,PAGE_SIZE , + end - cur)); + }else { +memcpy (buf + pageofs_out, cur + pageofs_out, +min_t (uint, righthalf, end - cur)); + } +kunmap_atomic (buf); + } + cur +=PAGE_SIZE ; + } +} + +static int decompress_generic (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const struct z_erofs_decompressor *alg = decompressors + rq->alg; +unsigned int dst_maptype; +void *dst; +int ret; + +if (nrpages_out ==1 && !rq->inplace_io) { +DBG_BUGON (!*rq->out); + dst =kmap_atomic (*rq->out); + dst_maptype =0 ; + goto dstmap_out; + } + +/* +* For the case of small output size (especially much less +* than PAGE_SIZE), memcpy the decompressed data rather than +* compressed data is preferred. +*/ +if (rq->outputsize <=PAGE_SIZE *7 /8 ) { + dst =erofs_get_pcpubuf (0 ); + + rq->inplace_io = false; + ret = alg->decompress (rq, dst); +if (!ret) +copy_from_pcpubuf (rq->out, dst, rq->pageofs_out, + rq->outputsize); + +erofs_put_pcpubuf (dst); +return ret; + } + + ret = alg->prepare_destpages (rq, pagepool); +if (ret <0 ) { +return ret; + }else if (ret) { + dst =page_address (*rq->out); + dst_maptype =1 ; + goto dstmap_out; + } + + dst =erofs_vmap (rq->out, nrpages_out); +if (!dst) +return -ENOMEM ; + dst_maptype =2 ; + +dstmap_out: + ret = alg->decompress (rq, dst + rq->pageofs_out); + +if (!dst_maptype) +kunmap_atomic (dst); +else if (dst_maptype ==2 ) +erofs_vunmap (dst, nrpages_out); +return ret; +} + +static int shifted_decompress (const struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const unsigned int righthalf =PAGE_SIZE - rq->pageofs_out; +unsigned char *src, *dst; + +if (nrpages_out >2 ) { +DBG_BUGON (1 ); +return -EIO ; + } + +if (rq->out[0 ] == *rq->in) { +DBG_BUGON (nrpages_out !=1 ); +return 0 ; + } + + src =kmap_atomic (*rq->in); +if (!rq->out[0 ]) { + dst =NULL ; + }else { + dst =kmap_atomic (rq->out[0 ]); +memcpy (dst + rq->pageofs_out, src, righthalf); + } + +if (rq->out[1 ] == *rq->in) { +memmove (src, src + righthalf, rq->pageofs_out); + }else if (nrpages_out ==2 ) { +if (dst) +kunmap_atomic (dst); +DBG_BUGON (!rq->out[1 ]); + dst =kmap_atomic (rq->out[1 ]); +memcpy (dst, src + righthalf, rq->pageofs_out); + } +if (dst) +kunmap_atomic (dst); +kunmap_atomic (src); +return 0 ; +} + +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +if (rq->alg ==Z_EROFS_COMPRESSION_SHIFTED ) +return shifted_decompress (rq, pagepool); +return decompress_generic (rq, pagepool); +} +
Re: [PATCH v2 5/8] staging: erofs: introduce generic decompression backend
Quoted 154 lines · Gao Xiang
Hi Chao,
Quoted 142 lines · Chao Yu
Quoted 141 lines · Gao Xiang
This patch adds a new generic decompression framework in order to replace the old LZ4-specific decompression code.
Even though LZ4 is still the only supported algorithm, yet it is more cleaner and easy to integrate new algorithm than the old almost hard-coded decompression backend.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/compress.h | 21 ++ drivers/staging/erofs/decompressor.c | 307 +++++++++++++++++++++++++++ 3 files changed, 329 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/erofs/decompressor.cdrivers/staging/erofs/Makefile +1
-1@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.odrivers/staging/erofs/compress.h +21
-0@@ -9,6 +9,24 @@#ifndef __EROFS_FS_COMPRESS_H#define __EROFS_FS_COMPRESS_H +#include "internal.h" + +enum { +Z_EROFS_COMPRESSION_SHIFTED =Z_EROFS_COMPRESSION_MAX , +Z_EROFS_COMPRESSION_RUNTIME_MAX +}; + +struct z_erofs_decompress_req { +struct page **in, **out; + +unsigned short pageofs_out; +unsigned int inputsize, outputsize; + +/* indicate the algorithm will be used for decompression */ +unsigned int alg; +bool inplace_io, partial_decoding; +}; +/* * - 0x5A110C8D ('sallocated', Z_EROFS_MAPPING_STAGING) - * used to mark temporary allocated pages from other @@ -36,5 +54,8 @@ static inline bool z_erofs_put_stagingpage(struct list_head *pagepool, return true; } +int z_erofs_decompress (struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +#endif drivers/staging/erofs/decompressor.c new file +71
-0@@ -0,0 +1,307 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* linux/drivers/staging/erofs/decompressor.c +* +* Copyright (C) 2019 HUAWEI, Inc. +* http://www.huawei.com/ +* Created by Gao Xiang <gaoxiang25 at huawei.com> +*/ +#include "compress.h" +#include <linux/lz4.h> + +#ifndef LZ4_DISTANCE_MAX /* history window size */ +#define LZ4_DISTANCE_MAX 65535/* set to maximum value by default */ +#endif + +#define LZ4_MAX_DISTANCE_PAGES DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) + +struct z_erofs_decompressor { +/* +* if destpages have sparsed pages, fill them with bounce pages. +* it also check whether destpages indicate continuous physical memory. +*/ +int (*prepare_destpages)(struct z_erofs_decompress_req *rq, +struct list_head *pagepool); +int (*decompress)(struct z_erofs_decompress_req *rq,u8 *out); +char *name; +}; + +static int lz4_prepare_destpages (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nr = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +struct page *availables[LZ4_MAX_DISTANCE_PAGES ] = {NULL }; +unsigned long unused[DIV_ROUND_UP (LZ4_MAX_DISTANCE_PAGES , +BITS_PER_LONG )] = {0 }; +void *kaddr =NULL ; +unsigned int i, j, k; + +for (i =0 ; i < nr; ++i) { +struct page *const page = rq->out[i]; + + j = i & (LZ4_MAX_DISTANCE_PAGES -1 ); +if (availables[j]) +__set_bit (j, unused); + +if (page) { +if (kaddr) { +if (kaddr +PAGE_SIZE ==page_address (page)) + kaddr +=PAGE_SIZE ; +else + kaddr =NULL ; + }else if (!i) { + kaddr =page_address (page); + } +continue ; + } + kaddr =NULL ; + + k =find_first_bit (unused,LZ4_MAX_DISTANCE_PAGES ); +if (k <LZ4_MAX_DISTANCE_PAGES ) { + j = k; +get_page (availables[j]); + }else { +DBG_BUGON (availables[j]); + +if (!list_empty (pagepool)) { + availables[j] =lru_to_page (pagepool); +list_del (&availables[j]->lru); +DBG_BUGON (page_ref_count (availables[j]) !=1 ); + }else {Quoted 8 lines
drivers/staging/erofs/decompressor.c line 76
Quoted 5 lines
drivers/staging/erofs/decompressor.c new file +5
-0+ availables[j] =alloc_pages (GFP_KERNEL ,0 ); +if (!availables[j]) +return -ENOMEM ; + } + availables[j]->mapping =Z_EROFS_MAPPING_STAGING ;Could we use
__stagingpage_alloc()instead opened codes, there is something different in between them though.
Quoted 6 lines
It was written "on propose" since the problem is that currently
__stagingpage_alloc()will allocate all pages inGFP_NOFAILcase, but memory allocation failure is accepted here, therefore I open code to aim at introducing new decompression backend.I will submit another patch after this series to clean up all staging page allocation cases later, which was already on my scheduling list. :)
Hi Xiang,
Yes, I noticed that there is something different in between __stagingpage_alloc and opened codes, I think we can change __stagingpage_alloc to accept extra parameters for different requirement.
But, anyway, I agreed with you that we can clean up with additional patch later. :)
Thanks,
Quoted 243 lines
Thanks,
Gao XiangQuoted 229 lines
Reviewed-by Chao Yu <yuchao0 at huawei.com>Thanks,
Quoted 224 lines
drivers/staging/erofs/decompressor.c new file +224
-0+ } + rq->out[i] = availables[j]; +__clear_bit (j, unused); + } +return kaddr ?1 :0 ; +} + +static void *generic_copy_inplace_data (struct z_erofs_decompress_req *rq, +u8 *src,unsigned int pageofs_in) +{ +/* +* if in-place decompression is ongoing, those decompressed +* pages should be copied in order to avoid being overlapped. +*/ +struct page **in = rq->in; +u8 *const tmp =erofs_get_pcpubuf (0 ); +u8 *tmpp = tmp; +unsigned int inlen = rq->inputsize - pageofs_in; +unsigned int count =min_t (uint, inlen,PAGE_SIZE - pageofs_in); + +while (tmpp < tmp + inlen) { +if (!src) + src =kmap_atomic (*in); +memcpy (tmpp, src + pageofs_in, count); +kunmap_atomic (src); + src =NULL ; + tmpp += count; + pageofs_in =0 ; + count =PAGE_SIZE ; + ++in; + } +return tmp; +} + +static int lz4_decompress (struct z_erofs_decompress_req *rq,u8 *out) +{ +unsigned int inputmargin, inlen; +u8 *src; +bool copied; +int ret; + +if (rq->inputsize >PAGE_SIZE ) +return -ENOTSUPP ; + + src =kmap_atomic (*rq->in); + inputmargin =0 ; +while (!src[inputmargin & ~PAGE_MASK ]) +if (!(++inputmargin & ~PAGE_MASK )) +break ; + +if (inputmargin >= rq->inputsize) { +kunmap_atomic (src); +return -EIO ; + } + + copied = false; + inlen = rq->inputsize - inputmargin; +if (rq->inplace_io) { + src =generic_copy_inplace_data (rq, src, inputmargin); + inputmargin =0 ; + copied = true; + } + + ret =LZ4_decompress_safe_partial (src + inputmargin, out, + inlen, rq->outputsize, + rq->outputsize); +if (ret <0 ) { +errln ("%s, failed to decompress, in[%p, %u, %u] out[%p, %u]" , + __func__, src + inputmargin, inlen, inputmargin, + out, rq->outputsize); +WARN_ON (1 ); +print_hex_dump (KERN_DEBUG ,"[ in]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , src + inputmargin, inlen, true); +print_hex_dump (KERN_DEBUG ,"[out]: " ,DUMP_PREFIX_OFFSET , +16 ,1 , out, rq->outputsize, true); + ret = -EIO ; + } + +if (copied) +erofs_put_pcpubuf (src); +else +kunmap_atomic (src); +return ret; +} + +static struct z_erofs_decompressor decompressors[] = { + [Z_EROFS_COMPRESSION_SHIFTED ] = { + .name ="shifted" + }, + [Z_EROFS_COMPRESSION_LZ4 ] = { + .prepare_destpages = lz4_prepare_destpages, + .decompress = lz4_decompress, + .name ="lz4" + }, +}; + +static void copy_from_pcpubuf (struct page **out,const char *dst, +unsigned short pageofs_out, +unsigned int outputsize) +{ +const char *end = dst + outputsize; +const unsigned int righthalf =PAGE_SIZE - pageofs_out; +const char *cur = dst - pageofs_out; + +while (cur < end) { +struct page *const page = *out++; + +if (page) { +char *buf =kmap_atomic (page); + +if (cur >= dst) { +memcpy (buf, cur,min_t (uint,PAGE_SIZE , + end - cur)); + }else { +memcpy (buf + pageofs_out, cur + pageofs_out, +min_t (uint, righthalf, end - cur)); + } +kunmap_atomic (buf); + } + cur +=PAGE_SIZE ; + } +} + +static int decompress_generic (struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const struct z_erofs_decompressor *alg = decompressors + rq->alg; +unsigned int dst_maptype; +void *dst; +int ret; + +if (nrpages_out ==1 && !rq->inplace_io) { +DBG_BUGON (!*rq->out); + dst =kmap_atomic (*rq->out); + dst_maptype =0 ; + goto dstmap_out; + } + +/* +* For the case of small output size (especially much less +* than PAGE_SIZE), memcpy the decompressed data rather than +* compressed data is preferred. +*/ +if (rq->outputsize <=PAGE_SIZE *7 /8 ) { + dst =erofs_get_pcpubuf (0 ); + + rq->inplace_io = false; + ret = alg->decompress (rq, dst); +if (!ret) +copy_from_pcpubuf (rq->out, dst, rq->pageofs_out, + rq->outputsize); + +erofs_put_pcpubuf (dst); +return ret; + } + + ret = alg->prepare_destpages (rq, pagepool); +if (ret <0 ) { +return ret; + }else if (ret) { + dst =page_address (*rq->out); + dst_maptype =1 ; + goto dstmap_out; + } + + dst =erofs_vmap (rq->out, nrpages_out); +if (!dst) +return -ENOMEM ; + dst_maptype =2 ; + +dstmap_out: + ret = alg->decompress (rq, dst + rq->pageofs_out); + +if (!dst_maptype) +kunmap_atomic (dst); +else if (dst_maptype ==2 ) +erofs_vunmap (dst, nrpages_out); +return ret; +} + +static int shifted_decompress (const struct z_erofs_decompress_req *rq, +struct list_head *pagepool) +{ +const unsigned int nrpages_out = +PAGE_ALIGN (rq->pageofs_out + rq->outputsize) >>PAGE_SHIFT ; +const unsigned int righthalf =PAGE_SIZE - rq->pageofs_out; +unsigned char *src, *dst; + +if (nrpages_out >2 ) { +DBG_BUGON (1 ); +return -EIO ; + } + +if (rq->out[0 ] == *rq->in) { +DBG_BUGON (nrpages_out !=1 ); +return 0 ; + } + + src =kmap_atomic (*rq->in); +if (!rq->out[0 ]) { + dst =NULL ; + }else { + dst =kmap_atomic (rq->out[0 ]); +memcpy (dst + rq->pageofs_out, src, righthalf); + } + +if (rq->out[1 ] == *rq->in) { +memmove (src, src + righthalf, rq->pageofs_out); + }else if (nrpages_out ==2 ) { +if (dst) +kunmap_atomic (dst); +DBG_BUGON (!rq->out[1 ]); + dst =kmap_atomic (rq->out[1 ]); +memcpy (dst, src + righthalf, rq->pageofs_out); + } +if (dst) +kunmap_atomic (dst); +kunmap_atomic (src); +return 0 ; +} + +int z_erofs_decompress(struct z_erofs_decompress_req *rq,Quoted 8 lines
drivers/staging/erofs/decompressor.c line 307
Quoted 8 lines
drivers/staging/erofs/decompressor.c new file +7
-0+ struct list_head *pagepool) +{ +if (rq->alg ==Z_EROFS_COMPRESSION_SHIFTED ) +return shifted_decompress (rq, pagepool); +return decompress_generic (rq, pagepool); +} +.
PATCH v2 6/8 staging: erofs: introduce LZ4 decompression inplace
In reply to Gao Xiangcompressed data will be usually loaded into last pages of the extent (the last page for 4k) for in-place decompression (more specifically, in-place IO), as ilustration below,
start of compressed logical extent
| end of this logical extent
| |
______v___________________________v________
... | page 6 | page 7 | page 8 | page 9 | ...
|__________|__________|__________|__________|
. ^ . ^
. |compressed|
. | data |
. . .
|< dstsize >|<margin>|
oend iendop ip
Therefore, it's possible to do decompression inplace (thus no memcpy at all) if the margin is sufficient and safe enough [1], and it can be implemented only for fixed-size output compression compared with fixed-size input compression.
No memcpy for most of in-place IO (about 99% of enwik9) after decompression inplace is implemented and sequential read will be improved of course (see the following patches for test results).
[1] https://github.com/lz4/lz4/commit/b17f578a919b7e6b078cede2d52be29dd48c8e8c
https://github.com/lz4/lz4/commit/5997e139f53169fa3a1c1b4418d2452a90b01602
--- drivers/staging/erofs/compress.h | 1 + drivers/staging/erofs/decompressor.c | 21 ++++++++++++++++++--- drivers/staging/erofs/erofs_fs.h | 3 ++- 3 files changed, 21 insertions(+), 4 deletions(-)
drivers/staging/erofs/compress.h +1 -0
@@ -17,6 +17,7 @@ enum { };struct z_erofs_decompress_req { +struct super_block *sb;struct page **in, **out;unsigned short pageofs_out;
drivers/staging/erofs/decompressor.c +18 -3
@@ -14,6 +14,9 @@#endif #define LZ4_MAX_DISTANCE_PAGES DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) +#ifndef LZ4_DECOMPRESS_INPLACE_MARGIN +#define LZ4_DECOMPRESS_INPLACE_MARGIN (srcsize) (((srcsize) >> 8) + 32) +#endif struct z_erofs_decompressor {/*
@@ -132,9 +135,21 @@ static int lz4_decompress(struct z_erofs_decompress_req *rq, u8 *out) copied = false; inlen = rq->inputsize - inputmargin;if (rq->inplace_io) { - src =generic_copy_inplace_data (rq, src, inputmargin); - inputmargin =0 ; - copied = true; +const uint oend = (rq->pageofs_out + + rq->outputsize) & ~PAGE_MASK ; +const uint nr =PAGE_ALIGN (rq->pageofs_out + + rq->outputsize) >>PAGE_SHIFT ; + +if (rq->partial_decoding || + !(EROFS_SB (rq->sb)->requirements & +EROFS_REQUIREMENT_LZ4_0PADDING ) || + rq->out[nr -1 ] != rq->in[0 ] || + rq->inputsize - oend < +LZ4_DECOMPRESS_INPLACE_MARGIN (inlen)) { + src =generic_copy_inplace_data (rq, src, inputmargin); + inputmargin =0 ; + copied = true; + } } ret = LZ4_decompress_safe_partial(src + inputmargin, out,
drivers/staging/erofs/erofs_fs.h +2 -1
@@ -21,7 +21,8 @@* Any bits that aren't in EROFS_ALL_REQUIREMENTS should be * incompatible with this kernel version. */ -#define EROFS_ALL_REQUIREMENTS 0 +#define EROFS_REQUIREMENT_LZ4_0PADDING 0x00000001 +#define EROFS_ALL_REQUIREMENTS EROFS_REQUIREMENT_LZ4_0PADDINGstruct erofs_super_block {/* 0 */ __le32 magic;/* in the little endian */
Signature · 1 line
2.17.1
Re: [PATCH v2 6/8] staging: erofs: introduce LZ4 decompression inplace
Quoted 23 lines · Gao Xiang
compressed data will be usually loaded into last pages of the extent (the last page for 4k) for in-place decompression (more specifically, in-place IO), as ilustration below,
start of compressed logical extent
| end of this logical extent
| |
______v___________________________v________... | page 6 | page 7 | page 8 | page 9 | ... |__________|__________|__________|__________| . ^ . ^ . |compressed| . | data | . . . |< dstsize >|<margin>| oend iendop ip
Therefore, it's possible to do decompression inplace (thus no memcpy at all) if the margin is sufficient and safe enough [1], and it can be implemented only for fixed-size output compression compared with fixed-size input compression.
Quoted 8 lines
No memcpy for most of in-place IO (about 99% of enwik9) after decompression inplace is implemented and sequential read will be improved of course (see the following patches for test results).
[1] https://github.com/lz4/lz4/commit/b17f578a919b7e6b078cede2d52be29dd48c8e8c
https://github.com/lz4/lz4/commit/5997e139f53169fa3a1c1b4418d2452a90b01602Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>
PATCH v2 7/8 staging: erofs: switch to new decompression backend
In reply to Gao XiangThis patch integrates new decompression framework to erofs decompression path, and remove the old decompression implementation as well.
On kirin980 platform, sequential read is slightly
improved to 778MiB/s after the new decompression
backend is used.
--- drivers/staging/erofs/Makefile | 2 +- drivers/staging/erofs/internal.h | 6 - drivers/staging/erofs/unzip_vle.c | 59 +++---- drivers/staging/erofs/unzip_vle.h | 15 +- drivers/staging/erofs/unzip_vle_lz4.c | 216 -------------------------- 5 files changed, 24 insertions(+), 274 deletions(-) delete mode 100644 drivers/staging/erofs/unzip_vle_lz4.c
drivers/staging/erofs/Makefile +1 -1
@@ -9,5 +9,5 @@ obj-$(CONFIG_EROFS_FS) += erofs.o ccflags-y += -I $(srctree)/$(src)/include erofs-objs := super.o inode.o data.o namei.o dir.o utils.o erofs-$(CONFIG_EROFS_FS_XATTR) += xattr.o -erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o unzip_vle_lz4.o zmap.o decompressor.o +erofs-$(CONFIG_EROFS_FS_ZIP) += unzip_vle.o zmap.o decompressor.o
drivers/staging/erofs/internal.h +0 -6
@@ -321,14 +321,8 @@ static inline void z_erofs_exit_zip_subsystem(void) {}/* page count of a compressed cluster */ #define erofs_clusterpages (sbi) ((1 << (sbi)->clusterbits) / PAGE_SIZE) -#define Z_EROFS_NR_INLINE_PAGEVECS 3 -#if (Z_EROFS_CLUSTER_MAX_PAGES >Z_EROFS_NR_INLINE_PAGEVECS )#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_CLUSTER_MAX_PAGES -#else -#define EROFS_PCPUBUF_NR_PAGES Z_EROFS_NR_INLINE_PAGEVECS -#endif -#else #define EROFS_PCPUBUF_NR_PAGES 0#endif
drivers/staging/erofs/unzip_vle.c +21 -38
76 lines not on this page · raw message
drivers/staging/erofs/unzip_vle.h +2 -13
@@ -16,6 +16,8 @@#include "internal.h" #include "unzip_pagevec.h" +#define Z_EROFS_NR_INLINE_PAGEVECS 3 +/* * Structure fields follow one of the following exclusion rules. *
@@ -189,18 +191,5 @@ static inline void z_erofs_onlinepage_endio(struct page *page) min_t(unsigned int, THREAD_SIZE / 8 / sizeof(struct page *), 96U)#define Z_EROFS_VLE_VMAP_GLOBAL_PAGES 2048 -/* unzip_vle_lz4.c */ -int z_erofs_vle_plain_copy (struct page **compressed_pages, -unsigned int clusterpages,struct page **pages, -unsigned int nr_pages,unsigned short pageofs); -int z_erofs_vle_unzip_fast_percpu (struct page **compressed_pages, -unsigned int clusterpages, -struct page **pages,unsigned int outlen, -unsigned short pageofs); -int z_erofs_vle_unzip_vmap (struct page **compressed_pages, -unsigned int clusterpages, -void *vaddr,unsigned int llen, -unsigned short pageofs,bool overlapped); -#endif
drivers/staging/erofs/unzip_vle_lz4.c deleted +0 -216
216 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [PATCH v2 7/8] staging: erofs: switch to new decompression backend
Quoted 9 lines · Gao Xiang
This patch integrates new decompression framework to erofs decompression path, and remove the old decompression implementation as well.
On kirin980 platform, sequential read is slightly improved to 778MiB/s after the new decompression backend is used.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>
PATCH v2 8/8 staging: erofs: integrate decompression inplace
In reply to Gao XiangDecompressor needs to know whether it's a partial or full decompression since only full decompression can be decompressed in-place.
On kirin980 platform, sequential read is finally increased to 812MiB/s after decompression inplace is enabled.
--- drivers/staging/erofs/internal.h | 3 +++ drivers/staging/erofs/unzip_vle.c | 15 +++++++++++---- drivers/staging/erofs/unzip_vle.h | 1 + drivers/staging/erofs/zmap.c | 1 + 4 files changed, 16 insertions(+), 4 deletions(-)
drivers/staging/erofs/internal.h +3 -0
@@ -441,6 +441,7 @@ extern const struct address_space_operations z_erofs_vle_normalaccess_aops;*/ enum { BH_Zipped = BH_PrivateStart, + BH_FullMapped, };/* Has a disk mapping */
@@ -449,6 +450,8 @@ enum {#define EROFS_MAP_META (1 << BH_Meta)/* The extent has been compressed */ #define EROFS_MAP_ZIPPED (1 << BH_Zipped) +/* The length of extent is full */ +#define EROFS_MAP_FULL_MAPPED (1 << BH_FullMapped)struct erofs_map_blocks {erofs_off_t m_pa, m_la;
drivers/staging/erofs/unzip_vle.c +11 -4
@@ -469,6 +469,9 @@ z_erofs_vle_work_register(const struct z_erofs_vle_work_finder *f, Z_EROFS_VLE_WORKGRP_FMT_LZ4 : Z_EROFS_VLE_WORKGRP_FMT_PLAIN); +if (map->m_flags &EROFS_MAP_FULL_MAPPED ) + grp->flags |=Z_EROFS_VLE_WORKGRP_FULL_LENGTH ; +/* new workgrps have been claimed as type 1 */ WRITE_ONCE (grp->next, *f->owned_head);/* primary and followed work for all new workgrps */
@@ -901,7 +904,7 @@ static int z_erofs_vle_unzip(struct super_block *sb,unsigned int i, outputsize;enum z_erofs_page_type page_type; -bool overlapped; +bool overlapped, partial;struct z_erofs_vle_work *work;int err;
@@ -1009,10 +1012,13 @@ static int z_erofs_vle_unzip(struct super_block *sb,if (unlikely (err)) goto out; -if (nr_pages <<PAGE_SHIFT >= work->pageofs + grp->llen) +if (nr_pages <<PAGE_SHIFT >= work->pageofs + grp->llen) { outputsize = grp->llen; -else + partial = !(grp->flags &Z_EROFS_VLE_WORKGRP_FULL_LENGTH ); + }else { outputsize = (nr_pages <<PAGE_SHIFT ) - work->pageofs; + partial = true; + }if (z_erofs_vle_workgrp_fmt (grp) ==Z_EROFS_VLE_WORKGRP_FMT_PLAIN ) algorithm =Z_EROFS_COMPRESSION_SHIFTED ;
@@ -1028,7 +1034,8 @@ static int z_erofs_vle_unzip(struct super_block *sb, .outputsize = outputsize, .alg = algorithm, .inplace_io = overlapped, - .partial_decoding = true }, page_pool); + .partial_decoding = partial + }, page_pool); out:/* must handle all compressed pages before endding pages */
drivers/staging/erofs/unzip_vle.h +1 -0
@@ -46,6 +46,7 @@ struct z_erofs_vle_work {#define Z_EROFS_VLE_WORKGRP_FMT_PLAIN 0#define Z_EROFS_VLE_WORKGRP_FMT_LZ4 1#define Z_EROFS_VLE_WORKGRP_FMT_MASK 1 +#define Z_EROFS_VLE_WORKGRP_FULL_LENGTH 2typedef void *z_erofs_vle_owned_workgrp_t ;
drivers/staging/erofs/zmap.c +1 -0
@@ -424,6 +424,7 @@ int z_erofs_map_blocks_iter(struct inode *inode, goto unmap_out; } end = (m.lcn << lclusterbits) | m.clusterofs; + map->m_flags |=EROFS_MAP_FULL_MAPPED ; m.delta[0 ] =1 ;/* fallthrough */ case Z_EROFS_VLE_CLUSTER_TYPE_NONHEAD :
Signature · 1 line
2.17.1
Re: [PATCH v2 8/8] staging: erofs: integrate decompression inplace
Quoted 9 lines · Gao Xiang
Decompressor needs to know whether it's a partial or full decompression since only full decompression can be decompressed in-place.
On kirin980 platform, sequential read is finally increased to 812MiB/s after decompression inplace is enabled.
Signed-off-by Gao Xiang <gaoxiang25 at huawei.com>