f2fs: change maximum zstd compression buffer size
Jaegeuk Kim
maintainerPATCH 1/1 f2fs: change maximum zstd compression buffer size
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/f2fs/compress.c +2 -2
@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream; - cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);return 0 ; }
@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret; inbuf.pos =0 ;
Signature · 1 line
2.26.2.526.g744177e7f7-goog
Chao Yu
maintainerQuoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 11 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 5 lines
fs/f2fs/compress.c +1
-1@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream; - cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);
In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret; inbuf.pos =0 ;
Chao Yu
maintainerQuoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 14 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream;
Quoted 2 lines
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 2 lines
fs/f2fs/compress.c +1
-1- cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);
Max size of intermediate buffer has been set as below?
- zstd_init_compress_ctx
- params = ZSTD_getParams(F2FS_ZSTD_DEFAULT_CLEVEL, cc->rlen, 0)
- ZSTD_CStreamWorkspaceBound(params.cParams)
- outBuffSize = ZSTD_compressBound(blockSize) + 1;
- workspace_size = ... + outBuffSize + ...
- workspace = f2fs_kvmalloc(workspace_size)
Quoted 25 lines
In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret; inbuf.pos =0 ;_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Jaegeuk Kim
maintainerQuoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 14 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream;
Quoted 7 lines
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 2 lines
fs/f2fs/compress.c +1
-1- cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Could you give it a try compress_log_size=2 and check decompression works?
Quoted 14 lines
Quoted 14 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret; inbuf.pos =0 ;
Quoted 34 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 14 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream;Quoted 7 lines
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 2 lines
fs/f2fs/compress.c +1
-1- cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Quoted 1 line
Could you give it a try compress_log_size=2 and check decompression works?
I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.
Thanks,
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret;Quoted 2 lines
fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336
Quoted 2 lines
fs/f2fs/compress.c +0
-0inbuf.pos =0 ;_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.
Jaegeuk Kim
maintainerQuoted 35 lines · Chao Yu
Quoted 34 lines · Jaegeuk Kim
Quoted 32 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 11 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 5 lines
fs/f2fs/compress.c +1
-1@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream; - cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Quoted 4 lines
Quoted 1 line
Could you give it a try compress_log_size=2 and check decompression works?
I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.
Daeho reported:
1. cp -a src_file comp_dir/dst_file (comp_dir is a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 > /proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).
Quoted 26 lines
Thanks,
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret;Quoted 2 lines
fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336
Quoted 2 lines
fs/f2fs/compress.c +0
-0inbuf.pos =0 ;_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
.
Quoted 38 lines · Jaegeuk Kim
Quoted 37 lines · Chao Yu
Quoted 35 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 14 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream;Quoted 7 lines
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 2 lines
fs/f2fs/compress.c +1
-1- cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Could you give it a try compress_log_size=2 and check decompression works?
Quoted 8 lines
Quoted 2 lines
I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.
Daeho reported:
1. cp -asrc_filecomp_dir/dst_file (comp_diris a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 >/proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).
Let me check this issue.
Thanks,
Quoted 29 lines
Quoted 27 lines
Thanks,
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret;Quoted 2 lines
fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336
Quoted 2 lines
fs/f2fs/compress.c +0
-0inbuf.pos =0 ;_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
..
Quoted 38 lines · Jaegeuk Kim
Quoted 37 lines · Chao Yu
Quoted 35 lines · Jaegeuk Kim
Quoted 25 lines · Chao Yu
Quoted 7 lines · Jaegeuk Kim
From: Daeho Jeong <[email protected]>
Current zstd compression buffer size is one page and header size less than cluster size. By this, zstd compression always succeeds even if the real compression data is failed to fit into the buffer size, and eventually reading the cluster returns I/O error with the corrupted compression data.
What's the root cause of this issue? I didn't get it.
Quoted 14 lines
Signed-off-by Daeho Jeong <[email protected]>--- fs/f2fs/compress.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)fs/f2fs/compress.c +0
-0@@ -313,7 +313,7 @@ static int zstd_init_compress_ctx(struct compress_ctx *cc) cc->private = workspace; cc->private2 = stream;Quoted 7 lines
fs/f2fs/compress.c static int zstd_init_compress_ctx(struct compress_ctx *cc) · line 316
Quoted 2 lines
fs/f2fs/compress.c +1
-1- cc->clen = cc->rlen -PAGE_SIZE -COMPRESS_HEADER_SIZE ; + cc->clen =ZSTD_compressBound (PAGE_SIZE << cc->log_cluster_size);In my machine, the value is 66572 which is much larger than size of dst buffer, so, in where we can tell the real size of dst buffer to zstd compressor? Otherwise, if compressed data size is larger than dst buffer size, when we flush compressed data into dst buffer, we may suffer overflow.
Could you give it a try compress_log_size=2 and check decompression works?
Quoted 8 lines
Quoted 2 lines
I tried some samples before submitting the patch, did you encounter app's data corruption when using zstd algorithm? If so, let me check this issue.
Daeho reported:
1. cp -asrc_filecomp_dir/dst_file (comp_diris a directory for compression)
2. sync comp_dir/dst_file
3. echo 3 >/proc/sys/vm/drop_caches
4. cat comp_dir/dst_file > dump (it returns I/O error depending on the file).
* ZSTD_endStream() to complete the flush. It returns the number of bytes left
* in the internal buffer and must be called until it returns 0.
It looks we need to check return value of ZSTD_endStream() to see whether dst buffer has enough space to store all compressed data.
Thanks,
Quoted 29 lines
Quoted 27 lines
Thanks,
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
fs/f2fs/compress.c +1
-1return 0 ; }@@ -330,7 +330,7 @@ static int zstd_compress_pages(struct compress_ctx *cc)ZSTD_inBuffer inbuf;ZSTD_outBuffer outbuf;int src_size = cc->rlen; -int dst_size = src_size -PAGE_SIZE -COMPRESS_HEADER_SIZE ; +int dst_size = cc->clen;int ret;Quoted 2 lines
fs/f2fs/compress.c static int zstd_compress_pages(struct compress_ctx *cc) · line 336
Quoted 2 lines
fs/f2fs/compress.c +0
-0inbuf.pos =0 ;_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
..