Skip to contentSkip to conversation

erofs-utils: lib: treat data blocks filled with 0s as a hole

PATCH In review Sandeep Dhavale via Linux-erofs 2 participants 2 messages Apr 17, 2024 – Apr 18, 2024 Library code

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 erofs-utils: lib: treat data blocks filled with 0s as a hole

Add optimization to treat data blocks filled with 0s as a hole. Even though diskspace savings are comparable to chunk based or dedupe, having no block assigned saves us redundant disk IOs during read.

To detect blocks filled with zeros during chunking, we insert block filled with zeros (zerochunk) in the hashmap. If we detect a possible dedupe, we map it to the hole so there is no physical block assigned.

---
Changes since v2:
- Fix memory leak for zeros buffer
- Add and use helper erofs_blob_can_merge() as suggested by Gao
- Return erofs_holechunk directly from erofs_blob_getchunk()
Changes since v1:
- Instead of checking every block for 0s word by word, add a zerochunk in blobs during init. So we effectively detect the zero blocks by comparing the hash.

 include/erofs/blobchunk.h |  2 +-
 lib/blobchunk.c           | 74 +++++++++++++++++++++++++++++++++------
 mkfs/main.c               |  2 +-
 3 files changed, 65 insertions(+), 13 deletions(-)
include/erofs/blobchunk.h +1 -1
index a674640..ebe2efe 100644
@@ -23,7 +23,7 @@ int erofs_write_zero_inode(struct erofs_inode *inode);
 int tarerofs_write_chunkes(struct erofs_inode *inode, erofs_off_t data_offset);
 int erofs_mkfs_dump_blobs(struct erofs_sb_info *sbi);
 void erofs_blob_exit(void);
-int erofs_blob_init(const char *blobfile_path);
+int erofs_blob_init(const char *blobfile_path, erofs_off_t chunksize);
 int erofs_mkfs_init_devices(struct erofs_sb_info *sbi, unsigned int devices);
 
 #ifdef __cplusplus
lib/blobchunk.c +63 -11

115 lines not on this page · raw message

mkfs/main.c +1 -1
index 2fb4a57..d632f74 100644
@@ -1255,7 +1255,7 @@ int main(int argc, char **argv)
 	}
 
 	if (cfg.c_chunkbits) {
-		err = erofs_blob_init(cfg.c_blobdev_path);
+		err = erofs_blob_init(cfg.c_blobdev_path, 1 << cfg.c_chunkbits);
 		if (err)
 			return 1;
 	}
Signature · 1 line
2.44.0.683.g7961c838ac-goog
Reviewed-by on v3 1/1
#1 lore raw
Quoted 9 lines · Sandeep Dhavale

Add optimization to treat data blocks filled with 0s as a hole. Even though diskspace savings are comparable to chunk based or dedupe, having no block assigned saves us redundant disk IOs during read.

To detect blocks filled with zeros during chunking, we insert block filled with zeros (zerochunk) in the hashmap. If we detect a possible dedupe, we map it to the hole so there is no physical block assigned.

Thanks,
Gao Xiang