Skip to contentSkip to conversation

f2fs: change maximum zstd compression buffer size

PATCH In review Jaegeuk Kim 3 participants 8 messages May 4, 2020 – May 7, 2020 F2FS file system

lore
Share & tools
lore

Jaegeuk Kim

author maintainer
#0 lore raw

PATCH 1/1 f2fs: change maximum zstd compression buffer size

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +2 -2
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
 	inbuf.pos = 0;
Signature · 1 line
2.26.2.526.g744177e7f7-goog
maintainer
#1 lore raw
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 11 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 5 lines
fs/f2fs/compress.c +1 -1
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Quoted 13 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
 	inbuf.pos = 0;
maintainer
#2 lore raw
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 14 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
     cc->private = workspace;
     cc->private2 = stream;
 
Quoted 2 lines

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 2 lines
fs/f2fs/compress.c +1 -1
-    cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+    cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

Max size of intermediate buffer has been set as below?

- zstd_init_compress_ctx
- params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0)
- ZSTD_CStreamWorkspaceBound(params.cParams)
- outBuffSize = ZSTD_compressBound(blockSize) + 1;
- workspace_size = ... + outBuffSize + ...
- workspace = f2fs_kvmalloc(workspace_size)

Quoted 25 lines

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336

Quoted 13 lines
fs/f2fs/compress.c +1 -1
     return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
     ZSTD_inBuffer inbuf;
     ZSTD_outBuffer outbuf;
     int src_size = cc->rlen;
-    int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+    int dst_size = cc->clen;
     int ret;
 
     inbuf.pos = 0;

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

author maintainer
#3 lore raw
In reply to Chao Yu
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 14 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
Quoted 7 lines

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 2 lines
fs/f2fs/compress.c +1 -1
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Could you give it a try compress_log_size=2 and check decompression works?

Quoted 14 lines
Quoted 14 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
 	inbuf.pos = 0;
#4 lore raw
Quoted 34 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 14 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
Quoted 7 lines

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 2 lines
fs/f2fs/compress.c +1 -1
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Quoted 1 line

Could you give it a try compress_log_size=2 and check decompression works?

I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.

Thanks,

Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
Quoted 2 lines

fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336

Quoted 2 lines
fs/f2fs/compress.c +0 -0
 	inbuf.pos = 0;

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.

author maintainer
#5 lore raw
Quoted 35 lines · Chao Yu
Quoted 34 lines · Jaegeuk Kim
Quoted 32 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 11 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 5 lines
fs/f2fs/compress.c +1 -1
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Quoted 4 lines
Quoted 1 line

Could you give it a try compress_log_size=2 and check decompression works?

I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.

Daeho reported:
1. cp -a src_file comp_dir/dst_file (comp_dir is a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 > /proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).

Quoted 26 lines

Thanks,

Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
Quoted 2 lines

fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336

Quoted 2 lines
fs/f2fs/compress.c +0 -0
 	inbuf.pos = 0;

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.

#6 lore raw
Quoted 38 lines · Jaegeuk Kim
Quoted 37 lines · Chao Yu
Quoted 35 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 14 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
Quoted 7 lines

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 2 lines
fs/f2fs/compress.c +1 -1
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Could you give it a try compress_log_size=2 and check decompression works?

Quoted 8 lines
Quoted 2 lines

I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.

Daeho reported:
1. cp -a src_file comp_dir/dst_file (comp_dir is a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 > /proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).

Let me check this issue.

Thanks,

Quoted 29 lines
Quoted 27 lines

Thanks,

Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
Quoted 2 lines

fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336

Quoted 2 lines
fs/f2fs/compress.c +0 -0
 	inbuf.pos = 0;

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.

.

#7 lore raw
In reply to Jaegeuk Kim
Quoted 38 lines · Jaegeuk Kim
Quoted 37 lines · Chao Yu
Quoted 35 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim

From: Daeho Jeong <[email protected]>

Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.

What's the root cause of this issue? I didn't get it.

Quoted 14 lines
---
 fs/f2fs/compress.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +0 -0
index 4c7eaeee52336..a9fa8049b295f 100644
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc)
 	cc->private = workspace;
 	cc->private2 = stream;
 
Quoted 7 lines

fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316

Quoted 2 lines
fs/f2fs/compress.c +1 -1
-	cc->clen = cc->rlen - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	cc->clen = ZSTD_compressBound(PAGE_SIZE << cc->log_cluster_size);

In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.

Could you give it a try compress_log_size=2 and check decompression works?

Quoted 8 lines
Quoted 2 lines

I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.

Daeho reported:
1. cp -a src_file comp_dir/dst_file (comp_dir is a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 > /proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).

* ZSTD_endStream() to complete the flush. It returns the number of bytes left
* in the internal buffer and must be called until it returns 0.

It looks we need to check return value of ZSTD_endStream() to see whether dst buffer has enough space to store all compressed data.

Thanks,

Quoted 29 lines
Quoted 27 lines

Thanks,

Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1 -1
 	return 0;
 }
 
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)
 	ZSTD_inBuffer inbuf;
 	ZSTD_outBuffer outbuf;
 	int src_size = cc->rlen;
-	int dst_size = src_size - PAGE_SIZE - COMPRESS_HEADER_SIZE;
+	int dst_size = cc->clen;
 	int ret;
 
Quoted 2 lines

fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336

Quoted 2 lines
fs/f2fs/compress.c +0 -0
 	inbuf.pos = 0;

_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.

.