erofs-utils: mkfs: support fragment deduplication
PATCH v7 1/1 erofs-utils: mkfs: support fragment deduplication
From: Yue Hu <[email protected]>
Previously, there's no fragment deduplication when this feature is introduced. Let's support it now.
Fragments are deduplicated before compression, so that duplicated parts will not be written into the packed inode again.
Dataset: linux 5.10 + 5.10.50 + 5.10.100 Compression algorithm: lz4hc,12
Additional options: -T0 --force-uid=1000 --force-gid=1000
(in order to force 32-byte inodes to match squashfs)
4k pcluster + fragment + dedupe 397168640
8k pcluster + fragment + dedupe 364224512
16k pcluster + fragment + dedupe 341921792
32k pcluster + fragment + dedupe 328298496
64k pcluster + fragment + dedupe 324694016
128k pcluster + fragment + dedupe 323674112
256k pcluster + fragment + dedupe 322011136squashfs-tools 4.5.1 test results (which uses level 12 by default for lz4hc):
16k block 428785664
32k block 382894080
64k block 350179328
128k block 327073792
128k block + noI 334327808
256k block 315441152
256k block + noI 322707456
1m block 307425280
1m block + noI 314712064If the compressed block size is large, the image size of squashfs will benefit from its simple _unseekable_ data block indexes:
location = block_start
for i = 0; i < index; i++
location += block_sizes[i] & 0x00FFFFFF
and its fragment lookup table. EROFS can improve such cases later.
---
v6 submit -> v7:
- move tailused in if condition for fixing and recover original code related in vle_compress_one, otherwise decompression may fail when only ztailpacking is enabled
include/erofs/compress.h | 3 +- include/erofs/fragments.h | 4 +- lib/compress.c | 126 +++++++++++++++++++++---- lib/fragments.c | 190 ++++++++++++++++++++++++++++++++++++-- 4 files changed, 295 insertions(+), 28 deletions(-)
include/erofs/compress.h +1 -2
@@ -14,8 +14,7 @@ extern "C"#include "internal.h" -#define EROFS_CONFIG_COMPR_MAX_SZ (3000 * 1024) -#define EROFS_CONFIG_COMPR_MIN_SZ (32 * 1024) +#define EROFS_CONFIG_COMPR_MAX_SZ (4000 * 1024)void z_erofs_drop_inline_pcluster (struct erofs_inode *inode);int erofs_write_compressed_file (struct erofs_inode *inode,int fd);
include/erofs/fragments.h +3 -1
@@ -15,8 +15,10 @@ extern "C"extern const char *frags_packedname;#define EROFS_PACKED_INODE frags_packedname +int z_erofs_fragments_dedupe (struct erofs_inode *inode,int fd,u32 *tofcrc);int z_erofs_pack_fragments (struct erofs_inode *inode,void *data, -unsigned int len); +unsigned int len,u32 tofcrc); +void z_erofs_fragments_commit (struct erofs_inode *inode);struct erofs_inode *erofs_mkfs_build_fragments (void );int erofs_fragments_init (void );void erofs_fragments_exit (void );
lib/compress.c +108 -18
236 lines not on this page · raw message
lib/fragments.c +183 -7
224 lines not on this page · raw message
Signature · 1 line
2.17.1