erofs-utils: mkfs: support fragment deduplication
PATCH v5 1/1 erofs-utils: mkfs: support fragment deduplication
From: Yue Hu <[email protected]>
Previously, there's no fragment deduplication when this feature is introduced. Let's support it now.
We intend to dedupe the fragments before compression, so that duplicate parts will not be written into packed inode.
I have tested some types with dataset of Linux 5.10.1 + 5.10.87 source code. The results of image size in MiB are:
32k pcluster + lz4hc,12 + T0 + fragment 311
64k pcluster + lz4hc,12 + T0 + fragment 295
128k pcluster + lz4hc,12 + T0 + fragment 287
32k pcluster + lz4hc,12 + T0 + fragment + dedupe 286
64k pcluster + lz4hc,12 + T0 + fragment + dedupe 281
128k pcluster + lz4hc,12 + T0 + fragment + dedupe 278
Before the patch, they were:
32k pcluster + lz4hc,12 + T0 + fragment 450
64k pcluster + lz4hc,12 + T0 + fragment 434
128k pcluster + lz4hc,12 + T0 + fragment 426
32k pcluster + lz4hc,12 + T0 + fragment + dedupe 368
64k pcluster + lz4hc,12 + T0 + fragment + dedupe 380
128k pcluster + lz4hc,12 + T0 + fragment + dedupe 395
Test results on squashfs (which uses level 12 by default for lz4hc):
32k block + lz4hc 332
64k block + lz4hc 304
128k block + lz4hc 283
256k block + lz4hc 273
256k block + lz4hc + noI 278
---
v5:
- fix the issue that decompression may fail when ztailpacking is enabled as well
- fix the issue that decompression may fail when dedupe is enabled as well in v4
- minor commit message update
v4:
- renaming include tofcrc/new_fragmentsize
- move fixup into ctx
- use may_fixing to check packing fragment or not
- move sb/inode flag + 64bits case from erofs_pack_fragments() to new helper erofs_fragments_commit()
- move recompress ahead of may_inline case when compressing succeeds
- update commit message/code comments
- change to generate a ctx for duplicate fragment in compression
- note that decompress will fail when enable ztailpacking at the same time, need some time to debug
v3:
- modify acrroding to Xiang's comments in v2
- simplify the logic in vle_compress_one
- fix the crash for 1MB pcluster
v2: mainly improve the logic in compression
include/erofs/fragments.h | 4 +- lib/compress.c | 140 ++++++++++++++++++++++----- lib/fragments.c | 197 ++++++++++++++++++++++++++++++++++++-- 3 files changed, 309 insertions(+), 32 deletions(-)
include/erofs/fragments.h +3 -1
@@ -15,8 +15,10 @@ extern "C"extern const char *frags_packedname;#define EROFS_PACKED_INODE frags_packedname +int z_erofs_fragments_dedupe (struct erofs_inode *inode,int fd,u32 *tofcrc_r);int z_erofs_pack_fragments (struct erofs_inode *inode,void *data, -unsigned int len); +unsigned int len,u32 tofcrc); +void z_erofs_fragments_commit (struct erofs_inode *inode);struct erofs_inode *erofs_mkfs_build_fragments (void );int erofs_fragments_init (void );void erofs_fragments_exit (void );
lib/compress.c +116 -24
246 lines not on this page · raw message
lib/fragments.c +190 -7
231 lines not on this page · raw message
Signature · 1 line
2.17.1