Skip to content

RFC erofs-utils: support multiple block compression

Gao Xiang via Linux-erofs linux-erofs 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v0 RFC Latest3 patches 1 replyIn review

Patches in v0 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/3erofs-utils: add -C# for the maximum size of pclustersinclude/erofs/config.h lib/config.c mkfs/main.c+16 -10 replies
2/3erofs-utils: mkfs: support multiple block compressioninclude/erofs/internal.h include/erofs_fs.h lib/compress.c+62 -280 replies
3/3erofs-utils: fuse: support multiple block compressionlib/data.c lib/zmap.c+68 -80 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v0 yet.

Read the v0 thread

Cover letter [RFC PATCH v0 0/3] erofs-utils: support multiple block compression

From: Gao Xiang <[email protected]>

Hi folks,

This is the first RFC patch of multiple block compression (including erofsfuse) after I carefully think over the on-disk design to support multiblock in-place decompression.

Compression ratio results (POC, lz4hc, lz4-1.9.3, not final result):

	1000000000		enwik9
		621211648	enwik9_4k.squashfs.img
	 557858816      	enwik9_4k.erofs.img
		556191744	enwik9_8k.squashfs.img
		502661120	enwik9_16k.squashfs.img
	 500723712		enwik9_8k.erofs.img
		458784768	enwik9_32k.squashfs.img
	 453971968		enwik9_16k.erofs.img
		422318080	enwik9_64k.squashfs.img
	 416686080		enwik9_32k.erofs.img
		398204928	enwik9_128k.squashfs.img
	 395276288		enwik9_64k.erofs.img

TODO:
- support compact indexes for multiple block compression **;
- support multithread compression (keep compressed data in memory);
- carefully design kernel optimized paths to maximize runtime performance;
- widely testing.

If you think that'd be useful for your products and you also have interest in development, feel free to follow that as well since I don't have abundant free time so the progress might be somewhat slow (I tend to finish them all before the next LTS).

Thanks,
Gao Xiang

Gao Xiang (3):
erofs-utils: add -C# for the maximum size of pclusters
erofs-utils: mkfs: support multiple block compression
erofs-utils: fuse: support multiple block compression

 include/erofs/config.h   |  2 ++
 include/erofs/internal.h |  1 +
 include/erofs_fs.h       | 19 ++++++++---
 lib/compress.c           | 70 ++++++++++++++++++++++++--------------
 lib/config.c             |  1 +
 lib/data.c               |  4 +--
 lib/zmap.c               | 72 ++++++++++++++++++++++++++++++++++++----
 mkfs/main.c              | 14 +++++++-
 8 files changed, 146 insertions(+), 37 deletions(-)
Signature · 1 line
2.24.0