Skip to contentSkip to conversation

erofs-utils: add clusterofs zero check to write_uncompressed_extent()

PATCH In review Yue Hu 2 participants 7 messages Aug 17, 2021 – Aug 18, 2021 Library code

lore
Share & tools
lore

Yue Hu

author
#0 lore raw

PATCH 1/1 erofs-utils: add clusterofs zero check to write_uncompressed_extent()

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +1 -1
index 40723a1..a8ebbc1 100644
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&
 	    ctx->head >= ctx->clusterofs) {
 		ctx->head -= ctx->clusterofs;
 		*len += ctx->clusterofs;
Signature · 1 line
1.9.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 6 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

(btw, also, if compression performance is important for some cases, I'd
suggest if someone could take some time on multi-threaded mkfs... That includes:
- paralleled compression for different files;
- paralleled segment compression for one file (the basic principle is to deal with a file with sub-files with the same segment size, such as 16M, like what I described in the following thread:

https://lore.kernel.org/r/20200705182049.GA20632@hsiangkao-HP-ZHAN-66-Pro-G1

That may be quite also useful for LZMA compression later.)
Thanks,
Gao Xiang

#2 lore raw
In reply to Yue Hu

Hi Yue,

Quoted 15 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +0 -0
index 40723a1..a8ebbc1 100644

lib/compress.c static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx, · line 133

Quoted 5 lines
lib/compress.c +1 -1
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&

Also out of curiosity, which means erofs is used without lz4 0padding? That way is not recommended now anyway, since it forbids erofs in-place decompression (only inplace I/O works instead.)

Actually we could make clusterofs aligned with 0 for 0padding cases as well, but in that cases, we need to recompress the previous pcluster with new trimmed size rather than just like this.

Thanks,
Gao Xiang

Quoted 5 lines
lib/compress.c +0 -0
 	    ctx->head >= ctx->clusterofs) {
 		ctx->head -= ctx->clusterofs;
 		*len += ctx->clusterofs;
Signature · 1 line
1.9.1
author
#3 lore raw

Hi Xiang,

Quoted 20 lines · On Tue, 17 Aug 2021 21:13:14 +0800 Gao Xiang <[email protected]>…

Hi Yue,

Quoted 17 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +0 -0
index 40723a1..a8ebbc1 100644
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
Quoted 5 lines

lib/compress.c static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx, · line 133

Quoted 3 lines
lib/compress.c +1 -1
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&  

Also out of curiosity, which means erofs is used without lz4 0padding?

Yes. We are using legacy compression layout now.

Quoted 2 lines

That way is not recommended now anyway, since it forbids erofs in-place decompression (only inplace I/O works instead.)

Thanks for the reminding.

Quoted 4 lines

Actually we could make clusterofs aligned with 0 for 0padding cases as well, but in that cases, we need to recompress the previous pcluster with new trimmed size rather than just like this.

Got it.
BTW, I think the information should be useful for me to understand the code further.

Thanks.

Quoted 9 lines

Thanks,
Gao Xiang

Quoted 5 lines
lib/compress.c +0 -0
 	    ctx->head >= ctx->clusterofs) {
 		ctx->head -= ctx->clusterofs;
 		*len += ctx->clusterofs;
Signature · 1 line
1.9.1  
#4 lore raw
Quoted 25 lines · Yue Hu

Hi Xiang,

Quoted 20 lines · On Tue, 17 Aug 2021 21:13:14 +0800 Gao Xiang <[email protected]>…

Hi Yue,

Quoted 17 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +0 -0
index 40723a1..a8ebbc1 100644
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
Quoted 7 lines
Quoted 5 lines

lib/compress.c static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx, · line 133

Quoted 3 lines
lib/compress.c +1 -1
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&  

Also out of curiosity, which means erofs is used without lz4 0padding?

Yes. We are using legacy compression layout now.

Ok, I think sometime I will seperate it into (non)compact compress indexes, and (non)0padding. It was once named as legacy just before Linux < 5.3 didn't support them.

By default, the preferred option now I think is compact and 0padding.

0padding will enable in-place decompression (and LZMA will always enable it.)
compact will reduce metadata even further as long as the compressed data is consecutive (e.g. 2 bytes each lcluster for compact indexes rather than 8 bytes each lcluster for noncompact indexes, but anyway, either (non)compact metadata can support effective data random access.)

Ok, if it's somewhat kernel 4.19 based code, I'd suggest to upgrade the codebase at least to 5.4, not because 4.19 doesn't work, but really for some performance concern....

Thanks,
Gao Xiang

author
#5 lore raw

Hi Xiang,

Quoted 26 lines · On Tue, 17 Aug 2021 22:38:04 +0800 Gao Xiang <[email protected]>…
Quoted 25 lines · Yue Hu

Hi Xiang,

Quoted 20 lines · On Tue, 17 Aug 2021 21:13:14 +0800 Gao Xiang <[email protected]>…

Hi Yue,

Quoted 17 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +0 -0
index 40723a1..a8ebbc1 100644
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/compress.c static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx, · line 133

Quoted 3 lines
lib/compress.c +1 -1
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&    

Also out of curiosity, which means erofs is used without lz4 0padding?

Moreover, seems 'ctx->head' will always >= 'ctx->clusterofs' here since there are only 2 sites to increase it. One is 'ctx->head += count' after compression, another is 'ctx->head = qh_after' ( which should be same as ->clusterofs, rt?).

I understand there are part of duplication data when writing uncompressed data to disk due to 'ctx->head' decreased?

Thanks.

Quoted 22 lines
Quoted 2 lines

Yes. We are using legacy compression layout now.

Ok, I think sometime I will seperate it into (non)compact compress indexes, and (non)0padding. It was once named as legacy just before Linux < 5.3 didn't support them.

By default, the preferred option now I think is compact and 0padding.

0padding will enable in-place decompression (and LZMA will always enable it.)
compact will reduce metadata even further as long as the compressed data is consecutive (e.g. 2 bytes each lcluster for compact indexes rather than 8 bytes each lcluster for noncompact indexes, but anyway, either (non)compact metadata can support effective data random access.)

Ok, if it's somewhat kernel 4.19 based code, I'd suggest to upgrade the codebase at least to 5.4, not because 4.19 doesn't work, but really for some performance concern....

Thanks,
Gao Xiang

#6 lore raw

Hi Yue,

Quoted 37 lines · Yue Hu

Hi Xiang,

Quoted 26 lines · On Tue, 17 Aug 2021 22:38:04 +0800 Gao Xiang <[email protected]>…
Quoted 25 lines · Yue Hu

Hi Xiang,

Quoted 20 lines · On Tue, 17 Aug 2021 21:13:14 +0800 Gao Xiang <[email protected]>…

Hi Yue,

Quoted 17 lines · Yue Hu

From: Yue Hu <[email protected]>

No need to reset clusterofs to 0 if it's already 0. Acturally, we also observed that case frequently. Let's avoid it.

---
 lib/compress.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
lib/compress.c +0 -0
index 40723a1..a8ebbc1 100644
@@ -130,7 +130,7 @@ static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx,
 	unsigned int count;
 
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

lib/compress.c static int write_uncompressed_extent(struct z_erofs_vle_compress_ctx *ctx, · line 133

Quoted 3 lines
lib/compress.c +1 -1
 	/* reset clusterofs to 0 if permitted */
-	if (!erofs_sb_has_lz4_0padding() &&
+	if (!erofs_sb_has_lz4_0padding() && ctx->clusterofs &&    

Also out of curiosity, which means erofs is used without lz4 0padding?

Quoted 3 lines

Moreover, seems 'ctx->head' will always >= 'ctx->clusterofs' here since there are only 2 sites to increase it. One is 'ctx->head += count' after compression, another is 'ctx->head = qh_after' ( which should be same as ->clusterofs, rt?).

Sorry for some delay (since I shouldn't take working time about this.)

The design of this part is that

Due to most compression algorithms can only accept a one-shot consecutive buffer for each lz4 block and we limit the maximal input (uncompressed) data for each pcluster is EROFS_CONFIG_COMPR_MAX_SZ, so EROFS allocates a buffer with the size of EROFS_CONFIG_COMPR_MAX_SZ * 2, as below:

2 * EROFS_CONFIG_COMPR_MAX_SZ
_____________________________________________________________

 |_____________________________________________________________|
      ^                                     ^
      head                                  tail

head is the pending data for compression, and tail is for the next read. As you can see, if head < EROFS_CONFIG_COMPR_MAX_SZ, we can always get `EROFS_CONFIG_COMPR_MAX_SZ consecutive data' in this buffer.

So after head >= EROFS_CONFIG_COMPR_MAX_SZ, we needs to move head ~ tail to the start of this buffer to keep it work.

Quoted 3 lines

I understand there are part of duplication data when writing uncompressed data to disk due to 'ctx->head' decreased?

Yeah, qh_aligned is mainly used for !0padding alignment of uncompressed data. It's not quite useful for 0padding, since head pointer won't be decreased.

But I don't think I like omiting ">= 'ctx->clusterofs'" (and I suspect if it's correct or just because the special EROFS_CONFIG_COMPR_MAX_SZ) since it involves many assumption and deep condition, and could cause maintain burden if we change logic like this.

Thanks,
Gao Xiang

Quoted 25 lines

Thanks.

Quoted 22 lines
Quoted 2 lines

Yes. We are using legacy compression layout now.

Ok, I think sometime I will seperate it into (non)compact compress indexes, and (non)0padding. It was once named as legacy just before Linux < 5.3 didn't support them.

By default, the preferred option now I think is compact and 0padding.

0padding will enable in-place decompression (and LZMA will always enable it.)
compact will reduce metadata even further as long as the compressed data is consecutive (e.g. 2 bytes each lcluster for compact indexes rather than 8 bytes each lcluster for noncompact indexes, but anyway, either (non)compact metadata can support effective data random access.)

Ok, if it's somewhat kernel 4.19 based code, I'd suggest to upgrade the codebase at least to 5.4, not because 4.19 doesn't work, but really for some performance concern....

Thanks,
Gao Xiang