Update to zstd-1.4.6
Subsystems (7)
v7 cover Update to zstd-1.4.6
From: Nick Terrell <[email protected]>
Please pull from
[email protected]:terrelln/linux.git tags/v7-zstd-1.4.6
to get these changes. Alternatively the patchset is included.
This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upstream zstd-1.3.1. At the time it was integrated, zstd wasn't ready to be used in the kernel as-is. But, it is now possible to use upstream zstd directly in the kernel.
I have not yet released zstd-1.4.6 upstream. I want the zstd version in the kernel to match up with a known upstream release, so we know exactly what code is running. Whenever this patchset is ready for merge, I will cut a release at the upstream commit that gets merged. This should not be necessary for future releases.
The kernel zstd library is automatically generated from upstream zstd. A script makes the necessary changes and imports it into the kernel. The changes are:
1. Replace all libc dependencies with kernel replacements and rewrite includes.
2. Remove unncessary portability macros like: #if defined(_MSC_VER).
3. Use the kernel xxhash instead of bundling it.
This automation gets tested every commit by upstream's continuous integration. When we cut a new zstd release, we will submit a patch to the kernel to update the zstd version in the kernel.
I've updated zstd to upstream with one big patch because every commit must build, so that precludes partial updates. Since the commit is 100% generated, I hope the review burden is lightened. I considered replaying upstream commits, but that is not possible because there have been ~3500 upstream commits since the last zstd import, and the commits don't all build individually. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug. After this big switch in how the kernel consumes zstd, future patches will be smaller, because they will only have one upstream release worth of changes each.
This patchset adds a new kernel-style wrapper around zstd. This wrapper API is functionally equivalent to the subset of the current zstd API that is currently used. The wrapper API changes to be kernel style so that the symbols don't collide with zstd's symbols. The update to zstd-1.4.6 maintains the same API and preserves the semantics, so that none of the callers need to be updated.
This patchset comes in 2 parts:
1. The first 2 patches prepare for the zstd upgrade. The first patch adds the new kernel style API so zstd can be upgraded without modifying any callers. The second patch adds an indirection for the lib/decompress_unzstd.c including of all decompression source files.
2. Import zstd-1.4.6. This patch is completely generated from upstream using automated tooling.
I tested every caller of zstd on x86_64. I tested both after the 1.4.6 upgrade using the compatibility wrapper, and after the final patch in this series.
I tested kernel and initramfs decompression in i386 and arm.
I ran benchmarks to compare the current zstd in the kernel with zstd-1.4.6. I benchmarked on x86_64 using QEMU with KVM enabled on an Intel i9-9900k. I found:
* BtrFS zstd compression at levels 1 and 3 is 5% faster
* BtrFS zstd decompression+read is 15% faster
* SquashFS zstd decompression+read is 15% faster
* F2FS zstd compression+write at level 3 is 8% faster
* F2FS zstd decompression+read is 20% faster
* ZRAM decompression+read is 30% faster
* Kernel zstd decompression is 35% faster
* Initramfs zstd decompression+build is 5% faster
The latest zstd also offers bug fixes and a 1 KB reduction in stack uage during compression. For example the recent problem with large kernel decompression has been fixed upstream for over 2 years https://lkml.org/lkml/2020/9/29/27.
Please let me know if there is anything that I can do to ease the way for these patches. I think it is important because it gets large performance improvements, contains bug fixes, and is switching to a more maintainable model of consuming upstream zstd directly, making it easy to keep up to date.
Best,
Nick Terrell
v1 -> v2:
* Successfully tested F2FS with help from Chao Yu to fix my test.
* (1/9) Fix ZSTD_initCStream() wrapper to handle pledged_src_size=0 means unknown. This fixes F2FS with the zstd-1.4.6 compatibility wrapper, exposed by the test.
v2 -> v3:
* (3/9) Silence warnings by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2324
Stack size warnings remain, but these aren't new, and the functions it warns on are either unused or not in the maximum stack path. This patchset reduces zstd compression stack usage by 1 KB overall. I've gotten the low hanging fruit, and more stack reduction would require significant changes that have the potential to introduce new bugs. However, I do hope to continue to reduce zstd stack usage in future versions.
v3 -> v4:
* (3/9) Fix errors and warnings reported by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2326
- Replace mem.h with a custom kernel implementation that matches the current lib/zstd/mem.h in the kernel. This avoids calls to __builtin_bswap*() which don't work on certain architectures, as exposed by the Kernel Test Robot.
- Remove ASAN/MSAN (un)poisoning code which doesn't work in the kernel, as exposed by the Kernel Test Robot.
- I've fixed all of the valid cppcheck warnings reported, but there were many false positives, where cppcheck was incorrectly analyzing the situation, which I did not fix. I don't believe it is reasonable to expect that upstream zstd silences all the static analyzer false positives. Upstream zstd uses clang scan-build for its static analysis. We find that supporting multiple static analysis tools multiplies the burden of silencing false positives, without providing enough marginal value over running a single static analysis tool.
v4 -> v5:
* Rebase onto v5.10-rc2
* (6/9) Merge with other F2FS changes (no functional change in patch).
v5 -> v6:
* Rebase onto v5.10-rc6.
* Switch to using a kernel style wrapper API as suggested by Cristoph.
v6 -> v7:
* Expose the upstream library header as include/linux/zstd_lib.h. Instead of creating new structs mirroring the upstream zstd structs use upstream's structs directly with a typedef to get a kernel style name. This removes the memcpy cruft.
* (1/3) Undo ZSTD_WINDOWLOG_MAX and handle_zstd_error changes.
* (3/3) Expose zstd_errors.h as include/linux/zstd_errors.h because it is needed by the kernel wrapper API.
Nick Terrell (3):
lib: zstd: Add kernel-specific API
lib: zstd: Add decompress_sources.h for decompress_unzstd
lib: zstd: Upgrade to latest upstream zstd version 1.4.6
crypto/zstd.c | 28 +- fs/btrfs/zstd.c | 68 +- fs/f2fs/compress.c | 56 +- fs/pstore/platform.c | 2 +- fs/squashfs/zstd_wrapper.c | 16 +- include/linux/zstd.h | 1225 +---- include/linux/zstd_errors.h | 76 + include/linux/zstd_lib.h | 2104 +++++++++ lib/decompress_unzstd.c | 48 +- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 150 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 258 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 124 + lib/zstd/common/zstd_internal.h | 438 ++ lib/zstd/compress.c | 3485 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4157 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 465 ++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2531 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 19 + lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 103 + lib/zstd/zstd_decompress_module.c | 96 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 73 files changed, 24800 insertions(+), 12929 deletions(-) create mode 100644 include/linux/zstd_errors.h create mode 100644 include/linux/zstd_lib.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/compress.c create mode 100644 lib/zstd/compress/fse_compress.c create mode 100644 lib/zstd/compress/hist.c create mode 100644 lib/zstd/compress/hist.h create mode 100644 lib/zstd/compress/huf_compress.c create mode 100644 lib/zstd/compress/zstd_compress.c create mode 100644 lib/zstd/compress/zstd_compress_internal.h create mode 100644 lib/zstd/compress/zstd_compress_literals.c create mode 100644 lib/zstd/compress/zstd_compress_literals.h create mode 100644 lib/zstd/compress/zstd_compress_sequences.c create mode 100644 lib/zstd/compress/zstd_compress_sequences.h create mode 100644 lib/zstd/compress/zstd_compress_superblock.c create mode 100644 lib/zstd/compress/zstd_compress_superblock.h create mode 100644 lib/zstd/compress/zstd_cwksp.h create mode 100644 lib/zstd/compress/zstd_double_fast.c create mode 100644 lib/zstd/compress/zstd_double_fast.h create mode 100644 lib/zstd/compress/zstd_fast.c create mode 100644 lib/zstd/compress/zstd_fast.h create mode 100644 lib/zstd/compress/zstd_lazy.c create mode 100644 lib/zstd/compress/zstd_lazy.h create mode 100644 lib/zstd/compress/zstd_ldm.c create mode 100644 lib/zstd/compress/zstd_ldm.h create mode 100644 lib/zstd/compress/zstd_opt.c create mode 100644 lib/zstd/compress/zstd_opt.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_compress.c delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_compress.c delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_compress_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h
Signature · 8 lines
2.29.2 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
PATCH v7 1/3 lib: zstd: Add kernel-specific API
From: Nick Terrell <[email protected]>
This patch:
- Moves include/linux/zstd.h -> include/linux/zstd_lib.h
- Adds a new API in include/linux/zstd.h that is functionally equivalent to the in-use subset of the current API. Functions are renamed to avoid symbol collisions with zstd, to make it clear it is not the upstream zstd API, and to follow the kernel style guide.
- Updates all callers to use the new API.
There are no functional changes in this patch. Since there are no functional change, I felt it was okay to update all the callers in a single patch, since once the API is approved, the callers are mechanically changed.
This patch is preparing the next patch in the series, which updates zstd to version 1.4.6. Since the upstream zstd API is no longer exposed to callers, the update can happen transparently.
--- crypto/zstd.c | 28 +- fs/btrfs/zstd.c | 68 +- fs/f2fs/compress.c | 56 +- fs/pstore/platform.c | 2 +- fs/squashfs/zstd_wrapper.c | 16 +- include/linux/zstd.h | 1218 ++++++++---------------------------- include/linux/zstd_lib.h | 1157 ++++++++++++++++++++++++++++++++++ lib/decompress_unzstd.c | 42 +- lib/zstd/compress.c | 107 ++-- lib/zstd/decompress.c | 112 ++-- 10 files changed, 1657 insertions(+), 1149 deletions(-) create mode 100644 include/linux/zstd_lib.h
crypto/zstd.c +14 -14
@@ -18,22 +18,22 @@#define ZSTD_DEF_LEVEL 3struct zstd_ctx { -ZSTD_CCtx *cctx; -ZSTD_DCtx *dctx; +zstd_cctx *cctx; +zstd_dctx *dctx;void *cwksp;void *dwksp; }; -static ZSTD_parameters zstd_params (void ) +static zstd_parameters zstd_params (void ) { -return ZSTD_getParams (ZSTD_DEF_LEVEL ,0 ,0 ); +return zstd_get_params (ZSTD_DEF_LEVEL ,0 ); }static int zstd_comp_init (struct zstd_ctx *ctx) {int ret =0 ; -const ZSTD_parameters params =zstd_params (); -const size_t wksp_size =ZSTD_CCtxWorkspaceBound (params.cParams); +const zstd_parameters params =zstd_params (); +const size_t wksp_size =zstd_cctx_workspace_bound (¶ms.cParams); ctx->cwksp =vzalloc (wksp_size);if (!ctx->cwksp) {
@@ -41,7 +41,7 @@ static int zstd_comp_init(struct zstd_ctx *ctx) goto out; } - ctx->cctx =ZSTD_initCCtx (ctx->cwksp, wksp_size); + ctx->cctx =zstd_init_cctx (ctx->cwksp, wksp_size);if (!ctx->cctx) { ret = -EINVAL ; goto out_free;
@@ -56,7 +56,7 @@ static int zstd_comp_init(struct zstd_ctx *ctx)static int zstd_decomp_init (struct zstd_ctx *ctx) {int ret =0 ; -const size_t wksp_size =ZSTD_DCtxWorkspaceBound (); +const size_t wksp_size =zstd_dctx_workspace_bound (); ctx->dwksp =vzalloc (wksp_size);if (!ctx->dwksp) {
@@ -64,7 +64,7 @@ static int zstd_decomp_init(struct zstd_ctx *ctx) goto out; } - ctx->dctx =ZSTD_initDCtx (ctx->dwksp, wksp_size); + ctx->dctx =zstd_init_dctx (ctx->dwksp, wksp_size);if (!ctx->dctx) { ret = -EINVAL ; goto out_free;
@@ -152,10 +152,10 @@ static int __zstd_compress(const u8 *src, unsigned int slen, {size_t out_len;struct zstd_ctx *zctx = ctx; -const ZSTD_parameters params =zstd_params (); +const zstd_parameters params =zstd_params (); - out_len =ZSTD_compressCCtx (zctx->cctx, dst, *dlen, src, slen, params); -if (ZSTD_isError (out_len)) + out_len =zstd_compress_cctx (zctx->cctx, dst, *dlen, src, slen, ¶ms); +if (zstd_is_error (out_len))return -EINVAL ; *dlen = out_len;return 0 ;
@@ -182,8 +182,8 @@ static int __zstd_decompress(const u8 *src, unsigned int slen,size_t out_len;struct zstd_ctx *zctx = ctx; - out_len =ZSTD_decompressDCtx (zctx->dctx, dst, *dlen, src, slen); -if (ZSTD_isError (out_len)) + out_len =zstd_decompress_dctx (zctx->dctx, dst, *dlen, src, slen); +if (zstd_is_error (out_len))return -EINVAL ; *dlen = out_len;return 0 ;
fs/btrfs/zstd.c +34 -34
169 lines not on this page · raw message
fs/f2fs/compress.c +28 -28
120 lines not on this page · raw message
fs/pstore/platform.c +1 -1
@@ -218,7 +218,7 @@ static int zbufsize_842(size_t size)#if IS_ENABLED (CONFIG_PSTORE_ZSTD_COMPRESS )static int zbufsize_zstd (size_t size) { -return ZSTD_compressBound (size); +return zstd_compress_bound (size); }#endif
fs/squashfs/zstd_wrapper.c +8 -8
@@ -34,7 +34,7 @@ static void *zstd_init(struct squashfs_sb_info *msblk, void *buff) goto failed; wksp->window_size =max_t (size_t, msblk->block_size,SQUASHFS_METADATA_SIZE ); - wksp->mem_size =ZSTD_DStreamWorkspaceBound (wksp->window_size); + wksp->mem_size =zstd_dstream_workspace_bound (wksp->window_size); wksp->mem =vmalloc (wksp->mem_size);if (wksp->mem ==NULL ) goto failed;
@@ -63,15 +63,15 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, struct squashfs_page_actor *output) {struct workspace *wksp = strm; -ZSTD_DStream *stream; +zstd_dstream *stream;size_t total_out =0 ;int error =0 ; -ZSTD_inBuffer in_buf = {NULL ,0 ,0 }; -ZSTD_outBuffer out_buf = {NULL ,0 ,0 }; +zstd_in_buffer in_buf = {NULL ,0 ,0 }; +zstd_out_buffer out_buf = {NULL ,0 ,0 };struct bvec_iter_all iter_all = {};struct bio_vec *bvec =bvec_init_iter_all (&iter_all); - stream =ZSTD_initDStream (wksp->window_size, wksp->mem, wksp->mem_size); + stream =zstd_init_dstream (wksp->window_size, wksp->mem, wksp->mem_size);if (!stream) {ERROR ("Failed to initialize zstd decompressor\n" );
@@ -116,14 +116,14 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, } total_out -= out_buf.pos; - zstd_err =ZSTD_decompressStream (stream, &out_buf, &in_buf); + zstd_err =zstd_decompress_stream (stream, &out_buf, &in_buf); total_out += out_buf.pos;/* add the additional data produced */ if (zstd_err ==0 )break ; -if (ZSTD_isError (zstd_err)) { +if (zstd_is_error (zstd_err)) {ERROR ("zstd decompression error: %d\n" , - (int )ZSTD_getErrorCode (zstd_err)); + (int )zstd_get_error_code (zstd_err)); error = -EIO ;break ; }
include/linux/zstd.h +246 -972
1,391 lines not on this page · raw message
include/linux/zstd_lib.h new file +1157 -0
1,157 lines not on this page · raw message
lib/decompress_unzstd.c +23 -19
36 lines not on this page · raw message
lib/zstd/compress.c +70 -37
107 lines not on this page · raw message
lib/zstd/decompress.c +76 -36
112 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 8 lines
2.29.2 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Part of the diff is not shown here. View the raw message.
PATCH v7 2/3 lib: zstd: Add decompress_sources.h for decompress_unzstd
In reply to Nick TerrellFrom: Nick Terrell <[email protected]>
Adds decompress_sources.h which includes every .c file necessary for zstd decompression. This is used in decompress_unzstd.c so the internal structure of the library isn't exposed.
This allows us to upgrade the zstd library version without modifying any callers. Instead we just need to update decompress_sources.h.
--- lib/decompress_unzstd.c | 6 +----- lib/zstd/decompress_sources.h | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 5 deletions(-) create mode 100644 lib/zstd/decompress_sources.h
lib/decompress_unzstd.c +1 -5
@@ -68,11 +68,7 @@#ifdef STATIC # define UNZSTD_PREBOOT # include "xxhash.c" -# include "zstd/entropy_common.c" -# include "zstd/fse_decompress.c" -# include "zstd/huf_decompress.c" -# include "zstd/zstd_common.c" -# include "zstd/decompress.c" +# include "zstd/decompress_sources.h" #endif #include <linux/decompress/mm.h>
lib/zstd/decompress_sources.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +/* +* This file includes every .c file needed for decompression. +* It is used by lib/decompress_unzstd.c to include the decompression +* source into the translation-unit, so it can be used for kernel +* decompression. +*/ + +#include "entropy_common.c" +#include "fse_decompress.c" +#include "huf_decompress.c" +#include "zstd_common.c" +#include "decompress.c"
Signature · 8 lines
2.29.2 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
PATCH v7 3/3 lib: zstd: Upgrade to latest upstream zstd version 1.4.6
In reply to Nick TerrellFrom: Nick Terrell <[email protected]>
Upgrade to the latest upstream zstd version 1.4.6.
This patch is 100% generated from upstream zstd commit f45fa724fef1 [0].
This patch is very large because it is transitioning from the custom kernel zstd to using upstream directly. The new zstd follows upstreams file structure which is different. Future update patches will be much smaller because they will only contain the changes from one upstream zstd release.
As an aid for review I've created a commit [1] that shows the diff between upstream zstd as-is (which doesn't compile), and the zstd code imported in this patch. The verion of zstd in this patch is generated from upstream with changes applied by automation to replace upstreams libc dependencies, remove unnecessary portability macros, and use the kernel's xxhash instead of bundling it.
The benefits of this patch are as follows:
1. Using upstream directly with automated script to generate kernel code. This allows us to update the kernel every upstream release, so the kernel gets the latest bug fixes and performance improvements, and doesn't get 3 years out of date again. The automation and the translated code are tested every upstream commit to ensure it continues to work.
2. Upgrades from a custom zstd based on 1.3.1 to 1.4.6, getting 3 years of performance improvements and bug fixes. On x86_64 I've measured 15% faster BtrFS and SquashFS decompression+read speeds, 35% faster kernel decompression, and 30% faster ZRAM decompression+read speeds. Additionally, the latest zstd uses ~1 KB less stack space for compression.
3. Zstd-1.4.6 supports negative compression levels, which allow zstd to match or subsume lzo's performance.
4. Maintains the same kernel-specific wrapper API, so no callers have to be modified with zstd version updates.
I chose the bulk update instead of replaying upstream commits because there have been ~3500 upstream commits since the 1.3.1 release, zstd wasn't ready to be used in the kernel as-is before a month ago, and not all upstream zstd commits build. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug.
Note that upstream zstd release 1.4.6 doesn't exist yet. I have cut a staging branch at f45fa724fef1 [0] and will apply any changes requested to the staging branch. Once we're ready to merge this update I will cut a zstd release at the commit we merge, so we have a known zstd release in the kernel.
The implementation of the kernel API is contained in zstd_compress_module.c and zstd_decompress_module.c.
[0] https://github.com/facebook/zstd/commit/f45fa724fef1e70449fa3806dc2360dda467561b
[1] https://github.com/terrelln/linux/commit/ac2ee65dcb7318afe426ad08f6a844faf3aebb41
--- include/linux/zstd.h | 9 +- include/linux/zstd_errors.h | 76 + include/linux/zstd_lib.h | 3021 ++++++++---- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 150 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 258 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 124 + lib/zstd/common/zstd_internal.h | 438 ++ lib/zstd/compress.c | 3518 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4157 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 465 ++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2571 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 15 +- lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 103 + lib/zstd/zstd_decompress_module.c | 96 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 67 files changed, 24317 insertions(+), 12964 deletions(-) create mode 100644 include/linux/zstd_errors.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/compress.c create mode 100644 lib/zstd/compress/fse_compress.c create mode 100644 lib/zstd/compress/hist.c create mode 100644 lib/zstd/compress/hist.h create mode 100644 lib/zstd/compress/huf_compress.c create mode 100644 lib/zstd/compress/zstd_compress.c create mode 100644 lib/zstd/compress/zstd_compress_internal.h create mode 100644 lib/zstd/compress/zstd_compress_literals.c create mode 100644 lib/zstd/compress/zstd_compress_literals.h create mode 100644 lib/zstd/compress/zstd_compress_sequences.c create mode 100644 lib/zstd/compress/zstd_compress_sequences.h create mode 100644 lib/zstd/compress/zstd_compress_superblock.c create mode 100644 lib/zstd/compress/zstd_compress_superblock.h create mode 100644 lib/zstd/compress/zstd_cwksp.h create mode 100644 lib/zstd/compress/zstd_double_fast.c create mode 100644 lib/zstd/compress/zstd_double_fast.h create mode 100644 lib/zstd/compress/zstd_fast.c create mode 100644 lib/zstd/compress/zstd_fast.h create mode 100644 lib/zstd/compress/zstd_lazy.c create mode 100644 lib/zstd/compress/zstd_lazy.h create mode 100644 lib/zstd/compress/zstd_ldm.c create mode 100644 lib/zstd/compress/zstd_ldm.h create mode 100644 lib/zstd/compress/zstd_opt.c create mode 100644 lib/zstd/compress/zstd_opt.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_compress.c delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_compress.c delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_compress_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h
include/linux/zstd.h +7 -2
@@ -27,6 +27,7 @@/* ====== Dependency ====== */ #include <linux/types.h> +#include <linux/zstd_errors.h> #include <linux/zstd_lib.h> /* ====== Helper Functions ====== */
@@ -408,12 +409,16 @@ size_t zstd_find_frame_compressed_size(const void *src, size_t src_size);/** * struct zstd_frame_params - zstd frame parameters stored in the frame header -* @frameContentSize: The frame content size, or 0 if not present. +* @frameContentSize: The frame content size, or ZSTD_CONTENTSIZE_UNKNOWN if not +* present. * @windowSize: The window size, or 0 if the frame is a skippable frame. +* @blockSizeMax: The maximum block size. +* @frameType: The frame type (zstd or skippable) +* @headerSize: The size of the frame header. * @dictID: The dictionary id, or 0 if not present. * @checksumFlag: Whether a checksum was used. */ -typedef ZSTD_frameParams zstd_frame_header ; +typedef ZSTD_frameHeader zstd_frame_header ;/** * zstd_get_frame_header() - extracts parameters from a zstd or skippable frame
include/linux/zstd_errors.h new file +76 -0
76 lines not on this page · raw message
include/linux/zstd_lib.h +1984 -1037
2,899 lines not on this page · raw message
lib/zstd/bitstream.h deleted +0 -379
379 lines not on this page · raw message
lib/zstd/common/bitstream.h new file +437 -0
437 lines not on this page · raw message
lib/zstd/common/compiler.h new file +150 -0
150 lines not on this page · raw message
lib/zstd/common/cpu.h new file +194 -0
194 lines not on this page · raw message
lib/zstd/common/debug.h new file +101 -0
101 lines not on this page · raw message
lib/zstd/common/entropy_common.c new file +355 -0
355 lines not on this page · raw message
lib/zstd/common/error_private.c new file +55 -0
55 lines not on this page · raw message
lib/zstd/common/error_private.h new file +66 -0
66 lines not on this page · raw message
lib/zstd/common/fse.h new file +709 -0
709 lines not on this page · raw message
lib/zstd/common/fse_decompress.c new file +380 -0
380 lines not on this page · raw message
lib/zstd/common/huf.h new file +352 -0
352 lines not on this page · raw message
lib/zstd/common/mem.h new file +258 -0
258 lines not on this page · raw message
lib/zstd/common/zstd_common.c new file +83 -0
83 lines not on this page · raw message
lib/zstd/common/zstd_deps.h new file +124 -0
124 lines not on this page · raw message
lib/zstd/common/zstd_internal.h new file +438 -0
438 lines not on this page · raw message
lib/zstd/compress.c deleted +0 -3518
3,518 lines not on this page · raw message
lib/zstd/compress/fse_compress.c new file +625 -0
625 lines not on this page · raw message
lib/zstd/compress/hist.c new file +165 -0
165 lines not on this page · raw message
lib/zstd/compress/hist.h new file +75 -0
75 lines not on this page · raw message
lib/zstd/compress/huf_compress.c new file +764 -0
764 lines not on this page · raw message
lib/zstd/compress/zstd_compress.c new file +4157 -0
4,157 lines not on this page · raw message
lib/zstd/compress/zstd_compress_internal.h new file +1103 -0
1,103 lines not on this page · raw message
lib/zstd/compress/zstd_compress_literals.c new file +158 -0
158 lines not on this page · raw message
lib/zstd/compress/zstd_compress_literals.h new file +29 -0
Lines of this file are not shown. View the raw message.
lib/zstd/compress/zstd_compress_sequences.c new file +433 -0
433 lines not on this page · raw message
lib/zstd/compress/zstd_compress_sequences.h new file +54 -0
54 lines not on this page · raw message
lib/zstd/compress/zstd_compress_superblock.c new file +849 -0
849 lines not on this page · raw message
lib/zstd/compress/zstd_compress_superblock.h new file +32 -0
Lines of this file are not shown. View the raw message.
lib/zstd/compress/zstd_cwksp.h new file +465 -0
465 lines not on this page · raw message
lib/zstd/compress/zstd_double_fast.c new file +521 -0
521 lines not on this page · raw message
lib/zstd/compress/zstd_double_fast.h new file +32 -0
Lines of this file are not shown. View the raw message.
lib/zstd/compress/zstd_fast.c new file +496 -0
496 lines not on this page · raw message
lib/zstd/compress/zstd_fast.h new file +31 -0
Lines of this file are not shown. View the raw message.
lib/zstd/compress/zstd_lazy.c new file +1138 -0
1,138 lines not on this page · raw message
lib/zstd/compress/zstd_lazy.h new file +61 -0
61 lines not on this page · raw message
lib/zstd/compress/zstd_ldm.c new file +619 -0
619 lines not on this page · raw message
lib/zstd/compress/zstd_ldm.h new file +104 -0
104 lines not on this page · raw message
lib/zstd/compress/zstd_opt.c new file +1200 -0
1,200 lines not on this page · raw message
lib/zstd/compress/zstd_opt.h new file +50 -0
50 lines not on this page · raw message
lib/zstd/decompress.c deleted +0 -2571
2,571 lines not on this page · raw message
lib/zstd/decompress/huf_decompress.c new file +1205 -0
1,205 lines not on this page · raw message
lib/zstd/decompress/zstd_ddict.c new file +241 -0
241 lines not on this page · raw message
lib/zstd/decompress/zstd_ddict.h new file +44 -0
44 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress.c new file +1836 -0
1,836 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_block.c new file +1540 -0
1,540 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_block.h new file +62 -0
62 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_internal.h new file +195 -0
195 lines not on this page · raw message
lib/zstd/entropy_common.c deleted +0 -243
243 lines not on this page · raw message
lib/zstd/error_private.h deleted +0 -53
53 lines not on this page · raw message
lib/zstd/fse.h deleted +0 -575
575 lines not on this page · raw message
lib/zstd/fse_compress.c deleted +0 -795
795 lines not on this page · raw message
lib/zstd/fse_decompress.c deleted +0 -325
325 lines not on this page · raw message
lib/zstd/huf.h deleted +0 -212
212 lines not on this page · raw message
lib/zstd/huf_compress.c deleted +0 -772
772 lines not on this page · raw message
lib/zstd/huf_decompress.c deleted +0 -960
960 lines not on this page · raw message
lib/zstd/mem.h deleted +0 -151
151 lines not on this page · raw message
lib/zstd/zstd_common.c deleted +0 -75
75 lines not on this page · raw message
lib/zstd/zstd_compress_module.c new file +103 -0
103 lines not on this page · raw message
lib/zstd/zstd_decompress_module.c new file +96 -0
96 lines not on this page · raw message
lib/zstd/zstd_internal.h deleted +0 -273
273 lines not on this page · raw message
lib/zstd/zstd_opt.h deleted +0 -1014
1,014 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 8 lines
2.29.2 _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Part of the diff is not shown here. View the raw message.
Nick Terrell
maintainerQuoted 11 lines · Nick Terrell
From: Nick Terrell <[email protected]>
Please pull from
[email protected]:terrelln/linux.git tags/v7-zstd-1.4.6
to get these changes. Alternatively the patchset is included.
Is it possible to get this patchset merged in the 5.11 merge window? It applies cleanly to the latest master. Please let me know if there is anything that I can do to drive this patchset towards merge.
Thanks,
Nick
Quoted 262 lines
This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upstream zstd-1.3.1. At the time it was integrated, zstd wasn't ready to be used in the kernel as-is. But, it is now possible to use upstream zstd directly in the kernel.
I have not yet released zstd-1.4.6 upstream. I want the zstd version in the kernel to match up with a known upstream release, so we know exactly what code is running. Whenever this patchset is ready for merge, I will cut a release at the upstream commit that gets merged. This should not be necessary for future releases.
The kernel zstd library is automatically generated from upstream zstd. A script makes the necessary changes and imports it into the kernel. The changes are:
1. Replace all libc dependencies with kernel replacements and rewrite includes.
2. Remove unncessary portability macros like: #ifdefined(_MSC_VER).
3. Use the kernel xxhash instead of bundling it.This automation gets tested every commit by upstream's continuous integration. When we cut a new zstd release, we will submit a patch to the kernel to update the zstd version in the kernel.
I've updated zstd to upstream with one big patch because every commit must build, so that precludes partial updates. Since the commit is 100% generated, I hope the review burden is lightened. I considered replaying upstream commits, but that is not possible because there have been ~3500 upstream commits since the last zstd import, and the commits don't all build individually. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug. After this big switch in how the kernel consumes zstd, future patches will be smaller, because they will only have one upstream release worth of changes each.
This patchset adds a new kernel-style wrapper around zstd. This wrapper API is functionally equivalent to the subset of the current zstd API that is currently used. The wrapper API changes to be kernel style so that the symbols don't collide with zstd's symbols. The update to zstd-1.4.6 maintains the same API and preserves the semantics, so that none of the callers need to be updated.
This patchset comes in 2 parts:
1. The first 2 patches prepare for the zstd upgrade. The first patch adds the new kernel style API so zstd can be upgraded without modifying any callers. The second patch adds an indirection for thelib/decompress_unzstd.cincluding of all decompression source files.
2. Import zstd-1.4.6. This patch is completely generated from upstream using automated tooling.I tested every caller of zstd on
x86_64. I tested both after the 1.4.6 upgrade using the compatibility wrapper, and after the final patch in this series.I tested kernel and initramfs decompression in i386 and arm.
I ran benchmarks to compare the current zstd in the kernel with zstd-1.4.6. I benchmarked on
x86_64using QEMU with KVM enabled on an Intel i9-9900k. I found:
* BtrFS zstd compression at levels 1 and 3 is 5% faster
* BtrFS zstd decompression+read is 15% faster
* SquashFS zstd decompression+read is 15% faster
* F2FS zstd compression+write at level 3 is 8% faster
* F2FS zstd decompression+read is 20% faster
* ZRAM decompression+read is 30% faster
* Kernel zstd decompression is 35% faster
* Initramfs zstd decompression+build is 5% fasterThe latest zstd also offers bug fixes and a 1 KB reduction in stack uage during compression. For example the recent problem with large kernel decompression has been fixed upstream for over 2 years https://lkml.org/lkml/2020/9/29/27.
Please let me know if there is anything that I can do to ease the way for these patches. I think it is important because it gets large performance improvements, contains bug fixes, and is switching to a more maintainable model of consuming upstream zstd directly, making it easy to keep up to date.
Best,
Nick Terrellv1 -> v2:
* Successfully tested F2FS with help from Chao Yu to fix my test.
* (1/9) FixZSTD_initCStream()wrapper to handle pledged_src_size=0 means unknown. This fixes F2FS with the zstd-1.4.6 compatibility wrapper, exposed by the test.v2 -> v3:
* (3/9) Silence warnings by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2324
Stack size warnings remain, but these aren't new, and the functions it warns on are either unused or not in the maximum stack path. This patchset reduces zstd compression stack usage by 1 KB overall. I've gotten the low hanging fruit, and more stack reduction would require significant changes that have the potential to introduce new bugs. However, I do hope to continue to reduce zstd stack usage in future versions.v3 -> v4:
* (3/9) Fix errors and warnings reported by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2326
- Replacemem.hwith a custom kernel implementation that matches the currentlib/zstd/mem.hin the kernel. This avoids calls to __builtin_bswap*() which don't work on certain architectures, as exposed by the Kernel Test Robot.
- Remove ASAN/MSAN (un)poisoning code which doesn't work in the kernel, as exposed by the Kernel Test Robot.
- I've fixed all of the valid cppcheck warnings reported, but there were many false positives, where cppcheck was incorrectly analyzing the situation, which I did not fix. I don't believe it is reasonable to expect that upstream zstd silences all the static analyzer false positives. Upstream zstd uses clang scan-build for its static analysis. We find that supporting multiple static analysis tools multiplies the burden of silencing false positives, without providing enough marginal value over running a single static analysis tool.v4 -> v5:
* Rebase onto v5.10-rc2
* (6/9) Merge with other F2FS changes (no functional change in patch).v5 -> v6:
* Rebase onto v5.10-rc6.
* Switch to using a kernel style wrapper API as suggested by Cristoph.v6 -> v7:
* Expose the upstream library header asinclude/linux/zstd_lib.h. Instead of creating new structs mirroring the upstream zstd structs use upstream's structs directly with a typedef to get a kernel style name. This removes the memcpy cruft.
* (1/3) UndoZSTD_WINDOWLOG_MAXandhandle_zstd_errorchanges.
* (3/3) Exposezstd_errors.hasinclude/linux/zstd_errors.hbecause it is needed by the kernel wrapper API.Nick Terrell (3):
lib: zstd: Add kernel-specific API
lib: zstd: Adddecompress_sources.hfordecompress_unzstd
lib: zstd: Upgrade to latest upstream zstd version 1.4.6crypto/zstd.c | 28 +- fs/btrfs/zstd.c | 68 +- fs/f2fs/compress.c | 56 +- fs/pstore/platform.c | 2 +- fs/squashfs/zstd_wrapper.c | 16 +- include/linux/zstd.h | 1225 +---- include/linux/zstd_errors.h | 76 + include/linux/zstd_lib.h | 2104 +++++++++ lib/decompress_unzstd.c | 48 +- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 150 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 258 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 124 + lib/zstd/common/zstd_internal.h | 438 ++ lib/zstd/compress.c | 3485 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4157 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 465 ++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2531 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 19 + lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 103 + lib/zstd/zstd_decompress_module.c | 96 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 73 files changed, 24800 insertions(+), 12929 deletions(-)create mode 100644
include/linux/zstd_errors.h
create mode 100644include/linux/zstd_lib.h
delete mode 100644lib/zstd/bitstream.h
create mode 100644lib/zstd/common/bitstream.h
create mode 100644lib/zstd/common/compiler.h
create mode 100644lib/zstd/common/cpu.h
create mode 100644lib/zstd/common/debug.c
create mode 100644lib/zstd/common/debug.h
create mode 100644lib/zstd/common/entropy_common.ccreate mode 100644lib/zstd/common/error_private.ccreate mode 100644lib/zstd/common/error_private.hcreate mode 100644lib/zstd/common/fse.h
create mode 100644lib/zstd/common/fse_decompress.ccreate mode 100644lib/zstd/common/huf.h
create mode 100644lib/zstd/common/mem.h
create mode 100644lib/zstd/common/zstd_common.c
create mode 100644lib/zstd/common/zstd_deps.h
create mode 100644lib/zstd/common/zstd_internal.hdelete mode 100644lib/zstd/compress.c
create mode 100644lib/zstd/compress/fse_compress.ccreate mode 100644lib/zstd/compress/hist.c
create mode 100644lib/zstd/compress/hist.h
create mode 100644lib/zstd/compress/huf_compress.ccreate mode 100644lib/zstd/compress/zstd_compress.ccreate mode 100644lib/zstd/compress/zstd_compress_internal.hcreate mode 100644lib/zstd/compress/zstd_compress_literals.ccreate mode 100644lib/zstd/compress/zstd_compress_literals.hcreate mode 100644lib/zstd/compress/zstd_compress_sequences.ccreate mode 100644lib/zstd/compress/zstd_compress_sequences.hcreate mode 100644lib/zstd/compress/zstd_compress_superblock.ccreate mode 100644lib/zstd/compress/zstd_compress_superblock.hcreate mode 100644lib/zstd/compress/zstd_cwksp.hcreate mode 100644lib/zstd/compress/zstd_double_fast.ccreate mode 100644lib/zstd/compress/zstd_double_fast.hcreate mode 100644lib/zstd/compress/zstd_fast.c
create mode 100644lib/zstd/compress/zstd_fast.h
create mode 100644lib/zstd/compress/zstd_lazy.c
create mode 100644lib/zstd/compress/zstd_lazy.h
create mode 100644lib/zstd/compress/zstd_ldm.c
create mode 100644lib/zstd/compress/zstd_ldm.h
create mode 100644lib/zstd/compress/zstd_opt.c
create mode 100644lib/zstd/compress/zstd_opt.h
delete mode 100644lib/zstd/decompress.c
create mode 100644lib/zstd/decompress/huf_decompress.ccreate mode 100644lib/zstd/decompress/zstd_ddict.ccreate mode 100644lib/zstd/decompress/zstd_ddict.hcreate mode 100644lib/zstd/decompress/zstd_decompress.ccreate mode 100644lib/zstd/decompress/zstd_decompress_block.ccreate mode 100644lib/zstd/decompress/zstd_decompress_block.hcreate mode 100644lib/zstd/decompress/zstd_decompress_internal.hcreate mode 100644lib/zstd/decompress_sources.h
delete mode 100644lib/zstd/entropy_common.c
delete mode 100644lib/zstd/error_private.h
delete mode 100644lib/zstd/fse.h
delete mode 100644lib/zstd/fse_compress.c
delete mode 100644lib/zstd/fse_decompress.c
delete mode 100644lib/zstd/huf.h
Quoted 12 lines
delete mode 100644
lib/zstd/huf_compress.cdelete mode 100644lib/zstd/huf_decompress.cdelete mode 100644lib/zstd/mem.h
delete mode 100644lib/zstd/zstd_common.ccreate mode 100644lib/zstd/zstd_compress_module.ccreate mode 100644lib/zstd/zstd_decompress_module.cdelete mode 100644lib/zstd/zstd_internal.hdelete mode 100644lib/zstd/zstd_opt.hSignature · 1 line
2.29.2
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Quoted 10 lines · On Tue, Dec 15, 2020 at 08:58:52PM +0000, Nick Terrell via Linux-f2fs-devel…
Quoted 10 lines · Nick Terrell
From: Nick Terrell <[email protected]>
Please pull from
[email protected]:terrelln/linux.git tags/v7-zstd-1.4.6
Quoted 8 lines
Quoted 1 line
to get these changes. Alternatively the patchset is included.
Is it possible to get this patchset merged in the 5.11 merge window? It applies cleanly to the latest master. Please let me know if there is anything that I can do to drive this patchset towards merge.
Thanks,
Nick
Well, it's too late for 5.11 for patches that weren't already in linux-next, so you'll have to aim for 5.12.
It looks like you're asking Herbert to pull this into the crypto tree? If he's interested in doing that, that could work. However lib/zstd/ isn't that strongly "crypto-related", and it doesn't actually have a maintainer listed in MAINTAINERS. Perhaps another path forwards is for you to volunteer to maintain lib/zstd/ and send pull requests for it directly to Linus?
- Eric
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Nick Terrell
maintainerQuoted 24 lines · Eric Biggers
Quoted 10 lines · On Tue, Dec 15, 2020 at 08:58:52PM +0000, Nick Terrell via Linux-f2fs-devel…
Quoted 10 lines · Nick Terrell
From: Nick Terrell <[email protected]>
Please pull from
[email protected]:terrelln/linux.git tags/v7-zstd-1.4.6
Quoted 8 lines
Quoted 1 line
to get these changes. Alternatively the patchset is included.
Is it possible to get this patchset merged in the 5.11 merge window? It applies cleanly to the latest master. Please let me know if there is anything that I can do to drive this patchset towards merge.
Thanks,
Nick
Quoted 8 lines
Well, it's too late for 5.11 for patches that weren't already in linux-next, so you'll have to aim for 5.12.
It looks like you're asking Herbert to pull this into the crypto tree? If he's interested in doing that, that could work. However
lib/zstd/isn't that strongly "crypto-related", and it doesn't actually have a maintainer listed inMAINTAINERS. Perhaps another path forwards is for you to volunteer to maintainlib/zstd/and send pull requests for it directly to Linus?
Thanks for the advice! The first zstd patches went through Herbert’s tree, which is why I’ve sent them this way.
I’d be happy to maintain zstd (and lz4, and xxhash), though I don’t know exactly what that entails.
Best,
Nick
Quoted 1 line
- Eric
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Herbert Xu
maintainerQuoted 3 lines · Nick Terrell
Thanks for the advice! The first zstd patches went through Herbert’s tree, which is why I’ve sent them this way.
Sorry, but I'm not touch these patches as Christoph's objections don't seem to have been addressed.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Nick Terrell
maintainerQuoted 11 lines · Herbert Xu
Quoted 3 lines · Nick Terrell
Thanks for the advice! The first zstd patches went through Herbert’s tree, which is why I’ve sent them this way.
Sorry, but I'm not touch these patches as Christoph's objections don't seem to have been addressed.
I believe I’ve addressed Christoph's objections. He suggested creating a wrapper API to avoid changing callers upon the zstd update. I’ve done that, the only difference between the current API, and the changes I’ve proposed patch 1, is that I’ve changed the prefix from ZSTD_ to zstd_ to avoid conflicts & confusion with the upstream zstd API.
Cristoph, if you get a chance to take a look at these patches, please let me know what you think about the current iteration of patches, and if I’ve addressed all of your concerns.
Best,
Nick
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Quoted 7 lines · Herbert Xu
Quoted 3 lines · Nick Terrell
Thanks for the advice! The first zstd patches went through Herbert’s tree, which is why I’ve sent them this way.
Sorry, but I'm not touch these patches as Christoph's objections don't seem to have been addressed.
I have objections to the current patchset as well, the build bot has found that some of the function frames are overly large (up to 3800 bytes) [1], besides the original complaint that the patch 3/3 is 1.5MiB.
[1] https://lore.kernel.org/lkml/[email protected]/
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Nick Terrell
maintainerQuoted 10 lines · David Sterba
Quoted 5 lines · Herbert Xu
Quoted 3 lines · Nick Terrell
Thanks for the advice! The first zstd patches went through Herbert’s tree, which is why I’ve sent them this way.
Quoted 6 lines
Quoted 2 lines
Sorry, but I'm not touch these patches as Christoph's objections don't seem to have been addressed.
I have objections to the current patchset as well, the build bot has found that some of the function frames are overly large (up to 3800 bytes) [1],
Sorry I missed your reply David, it didn’t make it to my inbox.
Compiled with x86-64, arm, and aarch64 that function does not trigger any -Wframe-larger-than= warnings during the kernel build. It seems like the compiler backend for the parisc architecture (the architecture that the build bot used) is doing a particularly bad job at optimizing this function, because there is nothing in there that should be using that much stack space.
I have a test in upstream zstd that measures the stack high water mark for all usage of zstd compression currently in-use the kernel. It says that zstd uses 2KB of stack space in total on x86-64. I used this test to remove 1KB of stack usage from upstream zstd. But, this is still 400 bytes more than the current version of zstd in the kernel. I will look into squeezing out those last 400 bytes of stack usage.
Quoted 3 lines
besides the original complaint that the patch 3/3 is 1.5MiB.
It is very large. If it helps, in the commit message I’ve provided this link [0], which provides the diff between upstream zstd as-is and the imported zstd, which has been modified by the automated tooling to work in the kernel.
[0] https://github.com/terrelln/linux/commit/ac2ee65dcb7318afe426ad08f6a844faf3aebb41
Best,
Nick
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
On Wed, Dec 16, 2020 at 10:07:38PM +0000, Nick Terrell wrote:
[...]
Quoted 4 lines
It is very large. If it helps, in the commit message I’ve provided this link [0], which provides the diff between upstream zstd as-is and the imported zstd, which has been modified by the automated tooling to work in the kernel.
[0] https://github.com/terrelln/linux/commit/ac2ee65dcb7318afe426ad08f6a844faf3aebb41
I looks like you could remove a bit more dead code by noting __GNUC__ >= 4 (gcc-4.9 is currently the oldest supported [1]).
[1] Documentation/process/changes.rst
Best Regards
Michał Mirosław
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Nick Terrell
maintainerQuoted 13 lines · Michał Mirosław
On Wed, Dec 16, 2020 at 10:07:38PM +0000, Nick Terrell wrote:
[...]Quoted 4 lines
It is very large. If it helps, in the commit message I’ve provided this link [0], which provides the diff between upstream zstd as-is and the imported zstd, which has been modified by the automated tooling to work in the kernel.
[0] https://github.com/terrelln/linux/commit/ac2ee65dcb7318afe426ad08f6a844faf3aebb41I looks like you could remove a bit more dead code by noting
__GNUC__>= 4 (gcc-4.9 is currently the oldest supported [1]).
Yeah, that would certainly be possible. My goal was to remove the most egregiously irrelevant code from the kernel, in addition to unused functions which would generate -Wframe-larger-than compiler warnings. My tooling doesn’t have the logic to reason about >= relationships yet. If it isn’t too hard to add, I may go ahead and do that, otherwise I will leave it for future work. I view that as a “nice to have” instead of a hard requirement, though let me know if you disagree.
Best,
Nick
Quoted 4 lines
[1]
Documentation/process/changes.rstBest Regards
Michał Mirosław
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
Hi.
Quoted 270 lines · Nick Terrell
From: Nick Terrell <[email protected]>
Please pull from
[email protected]:terrelln/linux.git tags/v7-zstd-1.4.6
to get these changes. Alternatively the patchset is included.
This patchset upgrades the zstd library to the latest upstream release. The current zstd version in the kernel is a modified version of upstream zstd-1.3.1. At the time it was integrated, zstd wasn't ready to be used in the kernel as-is. But, it is now possible to use upstream zstd directly in the kernel.
I have not yet released zstd-1.4.6 upstream. I want the zstd version in the kernel to match up with a known upstream release, so we know exactly what code is running. Whenever this patchset is ready for merge, I will cut a release at the upstream commit that gets merged. This should not be necessary for future releases.
The kernel zstd library is automatically generated from upstream zstd. A script makes the necessary changes and imports it into the kernel. The changes are:
1. Replace all libc dependencies with kernel replacements and rewrite includes.
2. Remove unncessary portability macros like: #ifdefined(_MSC_VER).
3. Use the kernel xxhash instead of bundling it.This automation gets tested every commit by upstream's continuous integration. When we cut a new zstd release, we will submit a patch to the kernel to update the zstd version in the kernel.
I've updated zstd to upstream with one big patch because every commit must build, so that precludes partial updates. Since the commit is 100% generated, I hope the review burden is lightened. I considered replaying upstream commits, but that is not possible because there have been ~3500 upstream commits since the last zstd import, and the commits don't all build individually. The bulk update preserves bisectablity because bugs can be bisected to the zstd version update. At that point the update can be reverted, and we can work with upstream to find and fix the bug. After this big switch in how the kernel consumes zstd, future patches will be smaller, because they will only have one upstream release worth of changes each.
This patchset adds a new kernel-style wrapper around zstd. This wrapper API is functionally equivalent to the subset of the current zstd API that is currently used. The wrapper API changes to be kernel style so that the symbols don't collide with zstd's symbols. The update to zstd-1.4.6 maintains the same API and preserves the semantics, so that none of the callers need to be updated.
This patchset comes in 2 parts:
1. The first 2 patches prepare for the zstd upgrade. The first patch adds the new kernel style API so zstd can be upgraded without modifying any callers. The second patch adds an indirection for thelib/decompress_unzstd.cincluding of all decompression source files.
2. Import zstd-1.4.6. This patch is completely generated from upstream using automated tooling.I tested every caller of zstd on
x86_64. I tested both after the 1.4.6 upgrade using the compatibility wrapper, and after the final patch in this series.I tested kernel and initramfs decompression in i386 and arm.
I ran benchmarks to compare the current zstd in the kernel with zstd-1.4.6. I benchmarked on
x86_64using QEMU with KVM enabled on an Intel i9-9900k. I found:
* BtrFS zstd compression at levels 1 and 3 is 5% faster
* BtrFS zstd decompression+read is 15% faster
* SquashFS zstd decompression+read is 15% faster
* F2FS zstd compression+write at level 3 is 8% faster
* F2FS zstd decompression+read is 20% faster
* ZRAM decompression+read is 30% faster
* Kernel zstd decompression is 35% faster
* Initramfs zstd decompression+build is 5% fasterThe latest zstd also offers bug fixes and a 1 KB reduction in stack uage during compression. For example the recent problem with large kernel decompression has been fixed upstream for over 2 years https://lkml.org/lkml/2020/9/29/27.
Please let me know if there is anything that I can do to ease the way for these patches. I think it is important because it gets large performance improvements, contains bug fixes, and is switching to a more maintainable model of consuming upstream zstd directly, making it easy to keep up to date.
Best,
Nick Terrellv1 -> v2:
* Successfully tested F2FS with help from Chao Yu to fix my test.
* (1/9) FixZSTD_initCStream()wrapper to handle pledged_src_size=0 means unknown. This fixes F2FS with the zstd-1.4.6 compatibility wrapper, exposed by the test.v2 -> v3:
* (3/9) Silence warnings by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2324
Stack size warnings remain, but these aren't new, and the functions it warns on are either unused or not in the maximum stack path. This patchset reduces zstd compression stack usage by 1 KB overall. I've gotten the low hanging fruit, and more stack reduction would require significant changes that have the potential to introduce new bugs. However, I do hope to continue to reduce zstd stack usage in future versions.v3 -> v4:
* (3/9) Fix errors and warnings reported by Kernel Test Robot:
https://github.com/facebook/zstd/pull/2326
- Replacemem.hwith a custom kernel implementation that matches the currentlib/zstd/mem.hin the kernel. This avoids calls to __builtin_bswap*() which don't work on certain architectures, as exposed by the Kernel Test Robot.
- Remove ASAN/MSAN (un)poisoning code which doesn't work in the kernel, as exposed by the Kernel Test Robot.
- I've fixed all of the valid cppcheck warnings reported, but there were many false positives, where cppcheck was incorrectly analyzing the situation, which I did not fix. I don't believe it is reasonable to expect that upstream zstd silences all the static analyzer false positives. Upstream zstd uses clang scan-build for its static analysis. We find that supporting multiple static analysis tools multiplies the burden of silencing false positives, without providing enough marginal value over running a single static analysis tool.v4 -> v5:
* Rebase onto v5.10-rc2
* (6/9) Merge with other F2FS changes (no functional change in patch).v5 -> v6:
* Rebase onto v5.10-rc6.
* Switch to using a kernel style wrapper API as suggested by Cristoph.v6 -> v7:
* Expose the upstream library header asinclude/linux/zstd_lib.h. Instead of creating new structs mirroring the upstream zstd structs use upstream's structs directly with a typedef to get a kernel style name. This removes the memcpy cruft.
* (1/3) UndoZSTD_WINDOWLOG_MAXandhandle_zstd_errorchanges.
* (3/3) Exposezstd_errors.hasinclude/linux/zstd_errors.hbecause it is needed by the kernel wrapper API.Nick Terrell (3):
lib: zstd: Add kernel-specific API
lib: zstd: Adddecompress_sources.hfordecompress_unzstd
lib: zstd: Upgrade to latest upstream zstd version 1.4.6crypto/zstd.c | 28 +- fs/btrfs/zstd.c | 68 +- fs/f2fs/compress.c | 56 +- fs/pstore/platform.c | 2 +- fs/squashfs/zstd_wrapper.c | 16 +- include/linux/zstd.h | 1225 +---- include/linux/zstd_errors.h | 76 + include/linux/zstd_lib.h | 2104 +++++++++ lib/decompress_unzstd.c | 48 +- lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 379 -- lib/zstd/common/bitstream.h | 437 ++ lib/zstd/common/compiler.h | 150 + lib/zstd/common/cpu.h | 194 + lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 101 + lib/zstd/common/entropy_common.c | 355 ++ lib/zstd/common/error_private.c | 55 + lib/zstd/common/error_private.h | 66 + lib/zstd/common/fse.h | 709 +++ lib/zstd/common/fse_decompress.c | 380 ++ lib/zstd/common/huf.h | 352 ++ lib/zstd/common/mem.h | 258 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 124 + lib/zstd/common/zstd_internal.h | 438 ++ lib/zstd/compress.c | 3485 -------------- lib/zstd/compress/fse_compress.c | 625 +++ lib/zstd/compress/hist.c | 165 + lib/zstd/compress/hist.h | 75 + lib/zstd/compress/huf_compress.c | 764 +++ lib/zstd/compress/zstd_compress.c | 4157 +++++++++++++++++ lib/zstd/compress/zstd_compress_internal.h | 1103 +++++ lib/zstd/compress/zstd_compress_literals.c | 158 + lib/zstd/compress/zstd_compress_literals.h | 29 + lib/zstd/compress/zstd_compress_sequences.c | 433 ++ lib/zstd/compress/zstd_compress_sequences.h | 54 + lib/zstd/compress/zstd_compress_superblock.c | 849 ++++ lib/zstd/compress/zstd_compress_superblock.h | 32 + lib/zstd/compress/zstd_cwksp.h | 465 ++ lib/zstd/compress/zstd_double_fast.c | 521 +++ lib/zstd/compress/zstd_double_fast.h | 32 + lib/zstd/compress/zstd_fast.c | 496 ++ lib/zstd/compress/zstd_fast.h | 31 + lib/zstd/compress/zstd_lazy.c | 1138 +++++ lib/zstd/compress/zstd_lazy.h | 61 + lib/zstd/compress/zstd_ldm.c | 619 +++ lib/zstd/compress/zstd_ldm.h | 104 + lib/zstd/compress/zstd_opt.c | 1200 +++++ lib/zstd/compress/zstd_opt.h | 50 + lib/zstd/decompress.c | 2531 ---------- lib/zstd/decompress/huf_decompress.c | 1205 +++++ lib/zstd/decompress/zstd_ddict.c | 241 + lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 1836 ++++++++ lib/zstd/decompress/zstd_decompress_block.c | 1540 ++++++ lib/zstd/decompress/zstd_decompress_block.h | 62 + .../decompress/zstd_decompress_internal.h | 195 + lib/zstd/decompress_sources.h | 19 + lib/zstd/entropy_common.c | 243 - lib/zstd/error_private.h | 53 - lib/zstd/fse.h | 575 --- lib/zstd/fse_compress.c | 795 ---- lib/zstd/fse_decompress.c | 325 -- lib/zstd/huf.h | 212 - lib/zstd/huf_compress.c | 772 --- lib/zstd/huf_decompress.c | 960 ---- lib/zstd/mem.h | 151 - lib/zstd/zstd_common.c | 75 - lib/zstd/zstd_compress_module.c | 103 + lib/zstd/zstd_decompress_module.c | 96 + lib/zstd/zstd_internal.h | 273 -- lib/zstd/zstd_opt.h | 1014 ---- 73 files changed, 24800 insertions(+), 12929 deletions(-) create mode 100644 include/linux/zstd_errors.h create mode 100644 include/linux/zstd_lib.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/compress.c create mode 100644 lib/zstd/compress/fse_compress.c create mode 100644 lib/zstd/compress/hist.c create mode 100644 lib/zstd/compress/hist.h create mode 100644 lib/zstd/compress/huf_compress.c create mode 100644 lib/zstd/compress/zstd_compress.c create mode 100644 lib/zstd/compress/zstd_compress_internal.h create mode 100644 lib/zstd/compress/zstd_compress_literals.c create mode 100644 lib/zstd/compress/zstd_compress_literals.h create mode 100644 lib/zstd/compress/zstd_compress_sequences.c create mode 100644 lib/zstd/compress/zstd_compress_sequences.h create mode 100644 lib/zstd/compress/zstd_compress_superblock.c create mode 100644 lib/zstd/compress/zstd_compress_superblock.h create mode 100644 lib/zstd/compress/zstd_cwksp.h create mode 100644 lib/zstd/compress/zstd_double_fast.c create mode 100644 lib/zstd/compress/zstd_double_fast.h create mode 100644 lib/zstd/compress/zstd_fast.c create mode 100644 lib/zstd/compress/zstd_fast.h create mode 100644 lib/zstd/compress/zstd_lazy.c create mode 100644 lib/zstd/compress/zstd_lazy.h create mode 100644 lib/zstd/compress/zstd_ldm.c create mode 100644 lib/zstd/compress/zstd_ldm.h create mode 100644 lib/zstd/compress/zstd_opt.c create mode 100644 lib/zstd/compress/zstd_opt.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_compress.c delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h
Quoted 12 lines
delete mode 100644 lib/zstd/huf_compress.c delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_compress_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.hSignature · 1 line
2.29.2
So, what's the fate of this submission please?
Thanks.
Signature · 7 lines
Oleksandr Natalenko (post-factum) _______________________________________________ Linux-f2fs-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel