Skip to contentSkip to conversation

erofs-utils: mkfs: support fragment deduplication

PATCH Superseded Yue Hu 1 participant 1 message Dec 3, 2022 Library code

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Yue Hu

author
#0 lore raw

PATCH v5 1/1 erofs-utils: mkfs: support fragment deduplication

From: Yue Hu <[email protected]>

Previously, there's no fragment deduplication when this feature is introduced. Let's support it now.

We intend to dedupe the fragments before compression, so that duplicate parts will not be written into packed inode.

I have tested some types with dataset of Linux 5.10.1 + 5.10.87 source code. The results of image size in MiB are:

32k pcluster + lz4hc,12 + T0 + fragment 311
64k pcluster + lz4hc,12 + T0 + fragment 295
128k pcluster + lz4hc,12 + T0 + fragment 287
32k pcluster + lz4hc,12 + T0 + fragment + dedupe 286
64k pcluster + lz4hc,12 + T0 + fragment + dedupe 281
128k pcluster + lz4hc,12 + T0 + fragment + dedupe 278

Before the patch, they were:

32k pcluster + lz4hc,12 + T0 + fragment 450
64k pcluster + lz4hc,12 + T0 + fragment 434
128k pcluster + lz4hc,12 + T0 + fragment 426
32k pcluster + lz4hc,12 + T0 + fragment + dedupe 368
64k pcluster + lz4hc,12 + T0 + fragment + dedupe 380
128k pcluster + lz4hc,12 + T0 + fragment + dedupe 395

Test results on squashfs (which uses level 12 by default for lz4hc):

32k block + lz4hc 332
64k block + lz4hc 304
128k block + lz4hc 283
256k block + lz4hc 273
256k block + lz4hc + noI 278

---
v5:
- fix the issue that decompression may fail when ztailpacking is enabled as well
- fix the issue that decompression may fail when dedupe is enabled as well in v4
- minor commit message update

v4:
- renaming include tofcrc/new_fragmentsize
- move fixup into ctx
- use may_fixing to check packing fragment or not
- move sb/inode flag + 64bits case from erofs_pack_fragments() to new helper erofs_fragments_commit()
- move recompress ahead of may_inline case when compressing succeeds
- update commit message/code comments
- change to generate a ctx for duplicate fragment in compression
- note that decompress will fail when enable ztailpacking at the same time, need some time to debug

v3:
- modify acrroding to Xiang's comments in v2
- simplify the logic in vle_compress_one
- fix the crash for 1MB pcluster

v2: mainly improve the logic in compression

 include/erofs/fragments.h |   4 +-
 lib/compress.c            | 140 ++++++++++++++++++++++-----
 lib/fragments.c           | 197 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 309 insertions(+), 32 deletions(-)
include/erofs/fragments.h +3 -1
index 5444384..e91fb31 100644
@@ -15,8 +15,10 @@ extern "C"
 extern const char *frags_packedname;
 #define EROFS_PACKED_INODE	frags_packedname
 
+int z_erofs_fragments_dedupe(struct erofs_inode *inode, int fd, u32 *tofcrc_r);
 int z_erofs_pack_fragments(struct erofs_inode *inode, void *data,
-			   unsigned int len);
+			   unsigned int len, u32 tofcrc);
+void z_erofs_fragments_commit(struct erofs_inode *inode);
 struct erofs_inode *erofs_mkfs_build_fragments(void);
 int erofs_fragments_init(void);
 void erofs_fragments_exit(void);
lib/compress.c +116 -24

246 lines not on this page · raw message

lib/fragments.c +190 -7

231 lines not on this page · raw message

Signature · 1 line
2.17.1