Skip to contentSkip to conversation

erofs-utils: mkfs: support fragment deduplication

PATCH New Yue Hu 1 participant 1 message Dec 8, 2022 Library code

lore
Share & tools
lore

Yue Hu

author
#0 lore raw

PATCH v7 1/1 erofs-utils: mkfs: support fragment deduplication

From: Yue Hu <[email protected]>

Previously, there's no fragment deduplication when this feature is introduced. Let's support it now.

Fragments are deduplicated before compression, so that duplicated parts will not be written into the packed inode again.

Dataset:               linux 5.10 + 5.10.50 + 5.10.100
Compression algorithm: lz4hc,12

Additional options: -T0 --force-uid=1000 --force-gid=1000

              (in order to force 32-byte inodes to match squashfs)

      4k  pcluster + fragment + dedupe  397168640
      8k  pcluster + fragment + dedupe  364224512
     16k  pcluster + fragment + dedupe  341921792
     32k  pcluster + fragment + dedupe  328298496
     64k  pcluster + fragment + dedupe  324694016
    128k  pcluster + fragment + dedupe  323674112
    256k  pcluster + fragment + dedupe  322011136

squashfs-tools 4.5.1 test results (which uses level 12 by default for lz4hc):

     16k  block                         428785664
     32k  block                         382894080
     64k  block                         350179328
    128k  block                         327073792
    128k  block + noI                   334327808
    256k  block                         315441152
    256k  block + noI                   322707456
      1m  block                         307425280
      1m  block + noI                   314712064

If the compressed block size is large, the image size of squashfs will benefit from its simple _unseekable_ data block indexes:

location = block_start
for i = 0; i < index; i++

      location += block_sizes[i] & 0x00FFFFFF

and its fragment lookup table. EROFS can improve such cases later.

---
v6 submit -> v7:
- move tailused in if condition for fixing and recover original code related in vle_compress_one, otherwise decompression may fail when only ztailpacking is enabled

 include/erofs/compress.h  |   3 +-
 include/erofs/fragments.h |   4 +-
 lib/compress.c            | 126 +++++++++++++++++++++----
 lib/fragments.c           | 190 ++++++++++++++++++++++++++++++++++++--
 4 files changed, 295 insertions(+), 28 deletions(-)
include/erofs/compress.h +1 -2
index e9dfaf2..08af9e3 100644
@@ -14,8 +14,7 @@ extern "C"
 
 #include "internal.h"
 
-#define EROFS_CONFIG_COMPR_MAX_SZ           (3000 * 1024)
-#define EROFS_CONFIG_COMPR_MIN_SZ           (32   * 1024)
+#define EROFS_CONFIG_COMPR_MAX_SZ           (4000 * 1024)
 
 void z_erofs_drop_inline_pcluster(struct erofs_inode *inode);
 int erofs_write_compressed_file(struct erofs_inode *inode, int fd);
include/erofs/fragments.h +3 -1
index 5444384..4caaf6b 100644
@@ -15,8 +15,10 @@ extern "C"
 extern const char *frags_packedname;
 #define EROFS_PACKED_INODE	frags_packedname
 
+int z_erofs_fragments_dedupe(struct erofs_inode *inode, int fd, u32 *tofcrc);
 int z_erofs_pack_fragments(struct erofs_inode *inode, void *data,
-			   unsigned int len);
+			   unsigned int len, u32 tofcrc);
+void z_erofs_fragments_commit(struct erofs_inode *inode);
 struct erofs_inode *erofs_mkfs_build_fragments(void);
 int erofs_fragments_init(void);
 void erofs_fragments_exit(void);
lib/compress.c +108 -18

236 lines not on this page · raw message

lib/fragments.c +183 -7

224 lines not on this page · raw message

Signature · 1 line
2.17.1