erofs-utils: support multiple block compression
A message that is not in the archive (sent off-list or not yet received): [email protected]
cover erofs-utils: support multiple block compression
From: Gao Xiang <[email protected]>
Hi folks,
This is the first RFC patch of multiple block compression (including erofsfuse) after I carefully think over the on-disk design to support multiblock in-place decompression.
Compression ratio results (POC, lz4hc, lz4-1.9.3, not final result):
1000000000 enwik9 621211648 enwik9_4k.squashfs.img 557858816 enwik9_4k.erofs.img 556191744 enwik9_8k.squashfs.img 502661120 enwik9_16k.squashfs.img 500723712 enwik9_8k.erofs.img 458784768 enwik9_32k.squashfs.img 453971968 enwik9_16k.erofs.img 422318080 enwik9_64k.squashfs.img 416686080 enwik9_32k.erofs.img 398204928 enwik9_128k.squashfs.img 395276288 enwik9_64k.erofs.img
TODO:
- support compact indexes for multiple block compression **;
- support multithread compression (keep compressed data in memory);
- carefully design kernel optimized paths to maximize runtime performance;
- widely testing.
If you think that'd be useful for your products and you also have interest in development, feel free to follow that as well since I don't have abundant free time so the progress might be somewhat slow (I tend to finish them all before the next LTS).
Thanks,
Gao Xiang
Gao Xiang (3):
erofs-utils: add -C# for the maximum size of pclusters
erofs-utils: mkfs: support multiple block compression
erofs-utils: fuse: support multiple block compression
include/erofs/config.h | 2 ++ include/erofs/internal.h | 1 + include/erofs_fs.h | 19 ++++++++--- lib/compress.c | 70 ++++++++++++++++++++++++-------------- lib/config.c | 1 + lib/data.c | 4 +-- lib/zmap.c | 72 ++++++++++++++++++++++++++++++++++++---- mkfs/main.c | 14 +++++++- 8 files changed, 146 insertions(+), 37 deletions(-)
Signature · 1 line
2.24.0
RFC PATCH 1/3 erofs-utils: add -C# for the maximum size of pclusters
From: Gao Xiang <[email protected]>
Set up -C >= EROFS_BLKSIZ (more specifically, >= lclustersize) to enable big pcluster feature.
--- include/erofs/config.h | 2 ++ lib/config.c | 1 + mkfs/main.c | 14 +++++++++++++- 3 files changed, 16 insertions(+), 1 deletion(-)
include/erofs/config.h +2 -0
@@ -53,6 +53,8 @@ struct erofs_configure {int c_force_inodeversion;/* < 0, xattr disabled and INT_MAX, always use inline xattrs */ int c_inline_xattr_tolerance; + +u32 c_physical_clusterblks;u64 c_unix_timestamp;#ifdef WITH_ANDROID char *mount_point;
lib/config.c +1 -0
@@ -24,6 +24,7 @@ void erofs_init_configure(void) cfg.c_force_inodeversion =0 ; cfg.c_inline_xattr_tolerance =2 ; cfg.c_unix_timestamp =-1 ; + cfg.c_physical_clusterblks =1 ; }void erofs_show_config (void )
mkfs/main.c +13 -1
@@ -62,6 +62,7 @@ static void usage(void)fputs ("usage: [options] FILE DIRECTORY\n\n" "Generate erofs image from DIRECTORY to FILE, and [options] are:\n" " -zX[,Y] X=compressor (Y=compression level, optional)\n" +" -C# specify the size of compress physical cluster in bytes\n" " -d# set output message level to # (maximum 9)\n" " -x# set xattr tolerance to # (< 0, disable xattrs; default 2)\n" " -EX[,...] X=extended options\n"
@@ -152,7 +153,7 @@ static int mkfs_parse_options_cfg(int argc, char *argv[])char *endptr;int opt, i; -while ((opt =getopt_long (argc, argv,"d:x:z:E:T:U:" , +while ((opt =getopt_long (argc, argv,"d:x:z:E:T:U:C:" , long_options,NULL )) !=-1 ) {switch (opt) {case 'z' :
@@ -248,6 +249,17 @@ static int mkfs_parse_options_cfg(int argc, char *argv[]) cfg.fs_config_file = optarg;break ;#endif +case 'C' : + i =strtoull (optarg, &endptr,0 ); +if (*endptr !='\0' || + i <EROFS_BLKSIZ || i %EROFS_BLKSIZ ) { +erofs_err ("invalid physical clustersize %s" , + optarg); +return -EINVAL ; + } + cfg.c_physical_clusterblks = i /EROFS_BLKSIZ ; +break ; +case 1 :usage ();exit (0 );
Signature · 1 line
2.24.0
RFC PATCH 2/3 erofs-utils: mkfs: support multiple block compression
In reply to Gao Xiang via Linux-erofsFrom: Gao Xiang <[email protected]>
Store compressed block count to the compressed index so that EROFS can compress from variable-sized input to variable-sized compressed blocks and make the in-place decompression possible as well.
TODO: support storing compressed block count for compact indexes.
--- include/erofs/internal.h | 1 + include/erofs_fs.h | 19 ++++++++--- lib/compress.c | 70 ++++++++++++++++++++++++++-------------- 3 files changed, 62 insertions(+), 28 deletions(-)
include/erofs/internal.h +1 -0
@@ -104,6 +104,7 @@ static inline void erofs_sb_clear_##name(void) \ } EROFS_FEATURE_FUNCS(lz4_0padding, incompat, INCOMPAT_LZ4_0PADDING) +EROFS_FEATURE_FUNCS(big_pcluster, incompat, INCOMPAT_BIG_PCLUSTER) EROFS_FEATURE_FUNCS(sb_chksum, compat, COMPAT_SB_CHKSUM)#define EROFS_I_EA_INITED (1 << 0)
include/erofs_fs.h +15 -4
@@ -20,7 +20,10 @@* be incompatible with this kernel version. */ #define EROFS_FEATURE_INCOMPAT_LZ4_0PADDING 0x00000001 -#define EROFS_ALL_FEATURE_INCOMPAT EROFS_FEATURE_INCOMPAT_LZ4_0PADDING +#define EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER 0x00000002 +#define EROFS_ALL_FEATURE_INCOMPAT \ + (EROFS_FEATURE_INCOMPAT_LZ4_0PADDING | \ + EROFS_FEATURE_INCOMPAT_BIG_PCLUSTER)/* 128-byte erofs on-disk super block */ struct erofs_super_block {
@@ -201,10 +204,11 @@ enum {* bit 0 : COMPACTED_2B indexes (0 - off; 1 - on) * e.g. for 4k logical cluster size, 4B if compacted 2B is off; * (4B) + 2B + (4B) if compacted 2B is on. +* bit 1 : HEAD1 big pcluster (0 - off; 1 - on) +* bit 2 : (reserved now) HEAD2 big pcluster (0 - off; 1 - on) */ -#define Z_EROFS_ADVISE_COMPACTED_2B_BIT 0 - -#define Z_EROFS_ADVISE_COMPACTED_2B (1 << Z_EROFS_ADVISE_COMPACTED_2B_BIT) +#define Z_EROFS_ADVISE_COMPACTED_2B 0x0001 +#define Z_EROFS_ADVISE_BIG_PCLUSTER_1 0x0002struct z_erofs_map_header {__le32 h_reserved1;
@@ -261,6 +265,13 @@ enum {#define Z_EROFS_VLE_DI_CLUSTER_TYPE_BITS 2#define Z_EROFS_VLE_DI_CLUSTER_TYPE_BIT 0 +/* +* D0_CBLKCNT will be marked _only_ for the 1st non-head lcluster to +* store the compressed block count of a compressed extent (aka. block +* count of a pcluster). +*/ +#define Z_EROFS_VLE_DI_D0_CBLKCNT 0x8000 +struct z_erofs_vle_decompressed_index {__le16 di_advise;/* where to decompress in the head cluster */
lib/compress.c +46 -24
154 lines not on this page · raw message
Signature · 1 line
2.24.0
RFC PATCH 3/3 erofs-utils: fuse: support multiple block compression
In reply to Gao Xiang via Linux-erofsFrom: Gao Xiang <[email protected]>
Add multiple block compression runtime support for erofsfuse.
--- lib/data.c | 4 +-- lib/zmap.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 8 deletions(-)
lib/data.c +2 -2
@@ -127,7 +127,7 @@ static int z_erofs_read_data(struct erofs_inode *inode, char *buffer, };bool partial;unsigned int algorithmformat; -char raw[EROFS_BLKSIZ ]; +char raw[1024 *EROFS_BLKSIZ ]; end = offset + size;while (end > offset) {
@@ -142,7 +142,7 @@ static int z_erofs_read_data(struct erofs_inode *inode, char *buffer,continue ; } - ret =dev_read (raw, map.m_pa,EROFS_BLKSIZ ); + ret =dev_read (raw, map.m_pa, map.m_plen);if (ret <0 )return -EIO ;
lib/zmap.c +66 -6
123 lines not on this page · raw message
Signature · 1 line
2.24.0
Quoted 9 lines · Gao Xiang via Linux-erofs
From: Gao Xiang <[email protected]>
Hi folks,
This is the first RFC patch of multiple block compression (including erofsfuse) after I carefully think over the on-disk design to support multiblock in-place decompression.
Compression ratio results (POC, lz4hc, lz4-1.9.3, not final result):
Quoted 12 lines
1000000000 enwik9 621211648 enwik9_4k.squashfs.img 557858816 enwik9_4k.erofs.img 556191744 enwik9_8k.squashfs.img 502661120 enwik9_16k.squashfs.img 500723712 enwik9_8k.erofs.img 458784768 enwik9_32k.squashfs.img 453971968 enwik9_16k.erofs.img 422318080 enwik9_64k.squashfs.img 416686080 enwik9_32k.erofs.img 398204928 enwik9_128k.squashfs.img 395276288 enwik9_64k.erofs.img
I can also think out several compress strategies to control read amplification but maintain a given C/R due to EROFS can compress variable-sized input data to arbitary compressed block count for each pcluster, FYI.
Thanks,
Gao Xiang