lib: zstd: update to latest Linux zstd 1.5.2
Subsystems (3)
PATCH v2 1/2 lib: zstd: update to latest Linux zstd 1.5.2
Update the zstd implementation to match Linux zstd 1.5.2 from commit 2aa14b1ab2.
This was motivated by running into decompression corruption issues when trying to uncompress files compressed with newer versions of zstd. zstd users also claim significantly improved decompression times with newer zstd versions which is a side benefit.
Original zstd code was copied from Linux commit 2aa14b1ab2 which is a custom-built implementation based on zstd 1.3.1. Linux switched to an implementation that is a copy of the upstream zstd code in Linux commit e0c1b49f5b, this results in a large code diff. However this should make future updates easier along with other benefits[1].
This commit is a straight mirror of the Linux zstd code, except to:
- update a few #include that do not translate cleanly
- linux/swab.h -> asm/byteorder.h
- linux/limits.h -> linux/kernel.h
- linux/module.h -> linux/compat.h
- remove assert() from debug.h so it doesn't conflict with u-boot's assert()
- strip out the compressor code as was done in the previous u-boot zstd
- update existing zstd users to the new Linux zstd API
- add a new KConfig option that sets zstd build options to minify code based on zstd's ZSTD_LIB_MINIFY[2].
These changes were tested by booting a zstd 1.5.2 compressed kernel inside a FIT. And the squashfs changes by loading a file from zstd compressed squashfs with sqfsload. buildman was used to compile test other boards and check for binary bloat, as follows:
Quoted 8 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of https://source.denx.de/u-boot/custodians/u-boot-tpm arm: w+ m53menlo dh_imx6 02: lib: zstd: update to latest Linux zstd 1.5.2 aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0 arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6 sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0
[1] https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8
[2] https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31
---
v2:
- zstd_decompress(): Handle junk at the end of the decompression buffer by checking actual size with zstd_find_frame_compressed_size().
--- fs/squashfs/sqfs_decompressor.c | 15 +- include/linux/zstd.h | 1244 ++------ include/linux/zstd_errors.h | 77 + include/linux/zstd_lib.h | 2551 +++++++++++++++++ lib/Kconfig | 15 + lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 344 --- lib/zstd/common/bitstream.h | 446 +++ lib/zstd/common/compiler.h | 184 ++ lib/zstd/common/cpu.h | 194 ++ lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 92 + lib/zstd/common/entropy_common.c | 357 +++ lib/zstd/common/error_private.c | 56 + lib/zstd/common/error_private.h | 145 + lib/zstd/common/fse.h | 711 +++++ lib/zstd/common/fse_decompress.c | 390 +++ lib/zstd/common/huf.h | 358 +++ lib/zstd/common/mem.h | 261 ++ lib/zstd/common/portability_macros.h | 93 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 125 + lib/zstd/common/zstd_internal.h | 443 +++ lib/zstd/decompress.c | 2516 ---------------- lib/zstd/decompress/huf_decompress.c | 1740 +++++++++++ lib/zstd/decompress/zstd_ddict.c | 241 ++ lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 2131 ++++++++++++++ lib/zstd/decompress/zstd_decompress_block.c | 2072 +++++++++++++ lib/zstd/decompress/zstd_decompress_block.h | 68 + .../decompress/zstd_decompress_internal.h | 228 ++ lib/zstd/decompress_sources.h | 34 + lib/zstd/entropy_common.c | 213 -- lib/zstd/error_private.h | 43 - lib/zstd/fse.h | 545 ---- lib/zstd/fse_decompress.c | 302 -- lib/zstd/huf.h | 182 -- lib/zstd/huf_decompress.c | 930 ------ lib/zstd/mem.h | 142 - lib/zstd/zstd.c | 52 +- lib/zstd/zstd_common.c | 66 - lib/zstd/zstd_common_module.c | 32 + lib/zstd/zstd_decompress_module.c | 105 + lib/zstd/zstd_internal.h | 253 -- lib/zstd/zstd_opt.h | 1004 ------- 45 files changed, 13638 insertions(+), 7548 deletions(-) create mode 100644 include/linux/zstd_errors.h create mode 100644 include/linux/zstd_lib.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/portability_macros.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_common_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h
fs/squashfs/sqfs_decompressor.c +9 -6
@@ -40,7 +40,7 @@ int sqfs_decompressor_init(struct squashfs_ctxt *ctxt)#endif #if IS_ENABLED (CONFIG_ZSTD )case SQFS_COMP_ZSTD : - ctxt->zstd_workspace =malloc (ZSTD_DCtxWorkspaceBound ()); + ctxt->zstd_workspace =malloc (zstd_dctx_workspace_bound ());if (!ctxt->zstd_workspace)return -ENOMEM ;break ;
@@ -99,11 +99,14 @@ static int sqfs_zstd_decompress(struct squashfs_ctxt *ctxt, void *dest,size_t wsize;int ret; - wsize =ZSTD_DCtxWorkspaceBound (); - ctx =ZSTD_initDCtx (ctxt->zstd_workspace, wsize); - ret =ZSTD_decompressDCtx (ctx, dest, dest_len, source, src_len); + wsize =zstd_dctx_workspace_bound (); - return ZSTD_isError(ret); + ctx =zstd_init_dctx (ctxt->zstd_workspace, wsize); +if (!ctx) +return -EINVAL ; + ret =zstd_decompress_dctx (ctx, dest, dest_len, source, src_len); + + return zstd_is_error(ret); }#endif /* CONFIG_ZSTD */
@@ -140,7 +143,7 @@ int sqfs_decompress(struct squashfs_ctxt *ctxt, void *dest,case SQFS_COMP_ZSTD : ret =sqfs_zstd_decompress (ctxt, dest, *dest_len, source, src_len);if (ret) { -printf ("ZSTD Error code: %d\n" ,ZSTD_getErrorCode (ret)); +printf ("ZSTD Error: %d\n" ,zstd_get_error_code (ret));return -EINVAL ; }
include/linux/zstd.h +272 -972
1,422 lines not on this page · raw message
include/linux/zstd_errors.h new file +77 -0
77 lines not on this page · raw message
include/linux/zstd_lib.h new file +2551 -0
1,467 lines not on this page · raw message
lib/zstd/bitstream.h deleted +0 -344
344 lines not on this page · raw message
lib/zstd/common/bitstream.h new file +446 -0
446 lines not on this page · raw message
lib/zstd/common/compiler.h new file +184 -0
184 lines not on this page · raw message
lib/zstd/common/cpu.h new file +194 -0
194 lines not on this page · raw message
lib/zstd/common/debug.h new file +92 -0
92 lines not on this page · raw message
lib/zstd/common/entropy_common.c new file +357 -0
357 lines not on this page · raw message
lib/zstd/common/error_private.c new file +56 -0
56 lines not on this page · raw message
lib/zstd/common/error_private.h new file +145 -0
145 lines not on this page · raw message
lib/zstd/common/fse.h new file +711 -0
711 lines not on this page · raw message
lib/zstd/common/fse_decompress.c new file +390 -0
390 lines not on this page · raw message
lib/zstd/common/huf.h new file +358 -0
358 lines not on this page · raw message
lib/zstd/common/mem.h new file +261 -0
261 lines not on this page · raw message
lib/zstd/common/portability_macros.h new file +93 -0
93 lines not on this page · raw message
lib/zstd/common/zstd_common.c new file +83 -0
83 lines not on this page · raw message
lib/zstd/common/zstd_deps.h new file +125 -0
125 lines not on this page · raw message
lib/zstd/common/zstd_internal.h new file +443 -0
443 lines not on this page · raw message
lib/zstd/decompress.c deleted +0 -2516
2,516 lines not on this page · raw message
lib/zstd/decompress/huf_decompress.c new file +1740 -0
1,740 lines not on this page · raw message
lib/zstd/decompress/zstd_ddict.c new file +241 -0
241 lines not on this page · raw message
lib/zstd/decompress/zstd_ddict.h new file +44 -0
44 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress.c new file +2131 -0
2,131 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_block.c new file +2072 -0
2,072 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_block.h new file +68 -0
68 lines not on this page · raw message
lib/zstd/decompress/zstd_decompress_internal.h new file +228 -0
228 lines not on this page · raw message
lib/zstd/decompress_sources.h new file +34 -0
Lines of this file are not shown. View the raw message.
lib/zstd/entropy_common.c deleted +0 -213
213 lines not on this page · raw message
lib/zstd/error_private.h deleted +0 -43
43 lines not on this page · raw message
lib/zstd/fse.h deleted +0 -545
545 lines not on this page · raw message
lib/zstd/fse_decompress.c deleted +0 -302
302 lines not on this page · raw message
lib/zstd/huf.h deleted +0 -182
182 lines not on this page · raw message
lib/zstd/huf_decompress.c deleted +0 -930
930 lines not on this page · raw message
lib/zstd/mem.h deleted +0 -142
142 lines not on this page · raw message
lib/zstd/zstd.c +24 -28
52 lines not on this page · raw message
lib/zstd/zstd_common.c deleted +0 -66
66 lines not on this page · raw message
lib/zstd/zstd_common_module.c new file +32 -0
Lines of this file are not shown. View the raw message.
lib/zstd/zstd_decompress_module.c new file +105 -0
105 lines not on this page · raw message
lib/zstd/zstd_internal.h deleted +0 -253
253 lines not on this page · raw message
lib/zstd/zstd_opt.h deleted +0 -1004
1,004 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.39.0
Part of the diff is not shown here. View the raw message.
PATCH v2 2/2 test: compression: add zstd uncompression test
--- test/Kconfig | 2 +- test/compression.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 1 deletion(-)
test/Kconfig +1 -1
@@ -64,7 +64,7 @@ endif config UT_COMPRESSION bool "Unit test for compression" depends on UNIT_TEST - depends on CMDLINE && GZIP_COMPRESSED && BZIP2 && LZMA && LZO && LZ4 + depends on CMDLINE && GZIP_COMPRESSED && BZIP2 && LZMA && LZO && LZ4 && ZSTD default y help Enables tests for compression and decompression routines for simple
test/compression.c +71 -0
107 lines not on this page · raw message
Signature · 1 line
2.39.0
Re: [PATCH v2 2/2] test: compression: add zstd uncompression test
Quoted 1 line · Brandon Maier
+static const unsigned long
zstd_compressed_size= 197;
You can use ARRAY_SIZE (or sizeof) for this.
--Sean
Re: [PATCH v2 2/2] test: compression: add zstd uncompression test
Hi Sean,
Quoted 5 lines
From: Sean Anderson <[email protected]>
Quoted 1 line · Brandon Maier
+static const unsigned long
zstd_compressed_size= 197;You can use
ARRAY_SIZE(or sizeof) for this.
sizeof() - 1 as it's a string, but I had put the size here manually as that's what all the other tests do. I can switch them all to sizeof() if that is preferred.
Quoted 3 lines
--Sean
Thanks,
Brandon
Re: [PATCH v2 2/2] test: compression: add zstd uncompression test
Hi Brandon,
Quoted 10 lines · Brandon L Collins
Hi Sean,
Quoted 5 lines
From: Sean Anderson <[email protected]>
Quoted 1 line · Brandon Maier
+static const unsigned long
zstd_compressed_size= 197;You can use
ARRAY_SIZE(or sizeof) for this.
sizeof()- 1 as it's a string, but I had put the size here manually as that's what all the other tests do. I can switch them all tosizeof()if that is preferred.
Yes I think it is better too, but please change the others in a separate patch.
Regards,
Simon
Hi Brandon,
Quoted 27 lines · Brandon Maier
Update the zstd implementation to match Linux zstd 1.5.2 from commit
2aa14b1ab2.This was motivated by running into decompression corruption issues when trying to uncompress files compressed with newer versions of zstd. zstd users also claim significantly improved decompression times with newer zstd versions which is a side benefit.
Original zstd code was copied from Linux commit
2aa14b1ab2which is a custom-built implementation based on zstd 1.3.1. Linux switched to an implementation that is a copy of the upstream zstd code in Linux commite0c1b49f5b, this results in a large code diff. However this should make future updates easier along with other benefits[1].This commit is a straight mirror of the Linux zstd code, except to:
- update a few #include that do not translate cleanly
-linux/swab.h->asm/byteorder.h
-linux/limits.h->linux/kernel.h
-linux/module.h->linux/compat.h
- removeassert()fromdebug.hso it doesn't conflict with u-boot'sassert()
- strip out the compressor code as was done in the previous u-boot zstd
- update existing zstd users to the new Linux zstd API
- add a new KConfig option that sets zstd build options to minify code based on zstd's ZSTD_LIB_MINIFY[2].
Quoted 12 lines
These changes were tested by booting a zstd 1.5.2 compressed kernel inside a FIT. And the squashfs changes by loading a file from zstd compressed squashfs with sqfsload. buildman was used to compile test other boards and check for binary bloat, as follows:
Quoted 7 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of https://source.denx.de/u-boot/custodians/u-boot-tpm arm: w+ m53menlo dh_imx6 02: lib: zstd: update to latest Linux zstd 1.5.2 aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0 arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?
Quoted 98 lines
Quoted 1 line
sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0
[1] https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8
[2] https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31Signed-off-by Brandon Maier <[email protected]>---
v2:
-zstd_decompress(): Handle junk at the end of the decompression buffer by checking actual size withzstd_find_frame_compressed_size().--- fs/squashfs/sqfs_decompressor.c | 15 +- include/linux/zstd.h | 1244 ++------ include/linux/zstd_errors.h | 77 + include/linux/zstd_lib.h | 2551 +++++++++++++++++ lib/Kconfig | 15 + lib/zstd/Makefile | 35 +- lib/zstd/bitstream.h | 344 --- lib/zstd/common/bitstream.h | 446 +++ lib/zstd/common/compiler.h | 184 ++ lib/zstd/common/cpu.h | 194 ++ lib/zstd/common/debug.c | 24 + lib/zstd/common/debug.h | 92 + lib/zstd/common/entropy_common.c | 357 +++ lib/zstd/common/error_private.c | 56 + lib/zstd/common/error_private.h | 145 + lib/zstd/common/fse.h | 711 +++++ lib/zstd/common/fse_decompress.c | 390 +++ lib/zstd/common/huf.h | 358 +++ lib/zstd/common/mem.h | 261 ++ lib/zstd/common/portability_macros.h | 93 + lib/zstd/common/zstd_common.c | 83 + lib/zstd/common/zstd_deps.h | 125 + lib/zstd/common/zstd_internal.h | 443 +++ lib/zstd/decompress.c | 2516 ---------------- lib/zstd/decompress/huf_decompress.c | 1740 +++++++++++ lib/zstd/decompress/zstd_ddict.c | 241 ++ lib/zstd/decompress/zstd_ddict.h | 44 + lib/zstd/decompress/zstd_decompress.c | 2131 ++++++++++++++ lib/zstd/decompress/zstd_decompress_block.c | 2072 +++++++++++++ lib/zstd/decompress/zstd_decompress_block.h | 68 + .../decompress/zstd_decompress_internal.h | 228 ++ lib/zstd/decompress_sources.h | 34 + lib/zstd/entropy_common.c | 213 -- lib/zstd/error_private.h | 43 - lib/zstd/fse.h | 545 ---- lib/zstd/fse_decompress.c | 302 -- lib/zstd/huf.h | 182 -- lib/zstd/huf_decompress.c | 930 ------ lib/zstd/mem.h | 142 - lib/zstd/zstd.c | 52 +- lib/zstd/zstd_common.c | 66 - lib/zstd/zstd_common_module.c | 32 + lib/zstd/zstd_decompress_module.c | 105 + lib/zstd/zstd_internal.h | 253 -- lib/zstd/zstd_opt.h | 1004 ------- 45 files changed, 13638 insertions(+), 7548 deletions(-) create mode 100644 include/linux/zstd_errors.h create mode 100644 include/linux/zstd_lib.h delete mode 100644 lib/zstd/bitstream.h create mode 100644 lib/zstd/common/bitstream.h create mode 100644 lib/zstd/common/compiler.h create mode 100644 lib/zstd/common/cpu.h create mode 100644 lib/zstd/common/debug.c create mode 100644 lib/zstd/common/debug.h create mode 100644 lib/zstd/common/entropy_common.c create mode 100644 lib/zstd/common/error_private.c create mode 100644 lib/zstd/common/error_private.h create mode 100644 lib/zstd/common/fse.h create mode 100644 lib/zstd/common/fse_decompress.c create mode 100644 lib/zstd/common/huf.h create mode 100644 lib/zstd/common/mem.h create mode 100644 lib/zstd/common/portability_macros.h create mode 100644 lib/zstd/common/zstd_common.c create mode 100644 lib/zstd/common/zstd_deps.h create mode 100644 lib/zstd/common/zstd_internal.h delete mode 100644 lib/zstd/decompress.c create mode 100644 lib/zstd/decompress/huf_decompress.c create mode 100644 lib/zstd/decompress/zstd_ddict.c create mode 100644 lib/zstd/decompress/zstd_ddict.h create mode 100644 lib/zstd/decompress/zstd_decompress.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.c create mode 100644 lib/zstd/decompress/zstd_decompress_block.h create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h create mode 100644 lib/zstd/decompress_sources.h delete mode 100644 lib/zstd/entropy_common.c delete mode 100644 lib/zstd/error_private.h delete mode 100644 lib/zstd/fse.h delete mode 100644 lib/zstd/fse_decompress.c delete mode 100644 lib/zstd/huf.h delete mode 100644 lib/zstd/huf_decompress.c delete mode 100644 lib/zstd/mem.h delete mode 100644 lib/zstd/zstd_common.c create mode 100644 lib/zstd/zstd_common_module.c create mode 100644 lib/zstd/zstd_decompress_module.c delete mode 100644 lib/zstd/zstd_internal.h delete mode 100644 lib/zstd/zstd_opt.h
Hello Simon,
Quoted 11 lines
-----Original Message-----
From: Simon Glass <[email protected]>Quoted 2 lines
Quoted 2 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basi c,turris_mox,turris_omnia -sS
Quoted 2 lines
Quoted 2 lines
Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of
https://urldefense.com/v3/__https://source.denx.de/u-boot/custodians/u-
boot-tpm__;!!MvWE!SiCDjQxi4IOlpR2LYEviKe-
1_MQirGfhHKYya94BvpeZgxBZ1YBd84kMf-FR0-VX0pY$
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
arm: w+ m53menlo
dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?
I understand Linux replaced their custom zstd code with the Facebook zstd code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:
aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0
arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6
sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text +58188.0
I haven't had any luck finding another way to decrease build size.
[1] https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0
Thanks,
Brandon
Quoted 26 lines · Brandon L Collins
Hello Simon,
Quoted 11 lines
-----Original Message-----
From: Simon Glass <[email protected]>Quoted 2 lines
Quoted 2 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basi c,turris_mox,turris_omnia -sS
Quoted 2 lines
Quoted 2 lines
Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of
https://urldefense.com/v3/__https://source.denx.de/u-boot/custodians/u-
boot-tpm__;!!MvWE!SiCDjQxi4IOlpR2LYEviKe-
1_MQirGfhHKYya94BvpeZgxBZ1YBd84kMf-FR0-VX0pY$Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
arm: w+ m53menlo
dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?
I understand Linux replaced their custom zstd code with the Facebook zstd code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:
Quoted 7 lines
aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0
arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text +58188.0
I haven't had any luck finding another way to decrease build size.
[1] https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build
After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty? Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from. Finally, what is the whole list of platforms that grow? It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.
Signature · 1 line
Tom
Quoted 14 lines
From: Tom Rini <[email protected]>
On Mon, Jan 02, 2023 at 04:02:06PM +0000, Maier, Brandon LQuoted 2 lines
I understand Linux replaced their custom zstd code with the Facebook zstd
code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:
Quoted 4 lines
aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0
arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text
+58188.0
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
I haven't had any luck finding another way to decrease build size.
[1]
https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build
After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty?
That is one of the settings I am already enabling. To clarify from that README, the settings I am enabling in U-Boot are HUF_FORCE_DECOMPRESS_X1, ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT, ZSTD_NO_INLINE, and ZSTD_STRIP_ERROR_STRINGS. Which is what upstream zstd uses for their own ZSTD_LIB_MINIFY option in lib/libzstd.mk.
If I drop the ZSTD_NO_INLINE, the build sizes increase as follows:
aarch64: (for 2/2 boards) all +6212.0 rodata +8.0 text +6204.0
arm: (for 5/5 boards) all +7408.0 text +7408.0
sandbox: (for 2/2 boards) all +21928.0 bss -16.0 data +16.0 rodata +224.0 text +21704.0
Quoted 2 lines
Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from.
I looked over each platform, but the size deltas are consistent within each architecture.
What stands out though is that the old ZSTD ARM code is about 4.6kb smaller (25%) than the old ZSTD AARCH64 code. The newer ZSTD doesn't do that, both ARM and AARCH64 are about the same in size. That is why there is a 4k jump in size for just the arm boards. Comparing the functions list between the two, both archs appear similar except ZSTD_decompressBlock_internal() grows significantly on aarch64 for some reason. I haven't seen any obvious root cause though.
Here is the full breakdown of platform size deltas
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0
turris_mox : all +572 rodata +968 text -396
mvebu_espressobin-88f3720: all +499 rodata +963 text -464
arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
turris_omnia : all +4608 rodata +940 text +3668
m53menlo : all +4460 rodata +940 text +3520
dh_imx6 : all +4460 rodata +940 text +3520
stm32mp15_dhcom_basic: all +4460 rodata +940 text +3520
stm32mp15_dhcor_basic: all +4460 rodata +940 text +3520
sandbox: (for 2/2 boards) all +8.0 rodata +784.0 text -776.0
sandbox : all +528 bss -32 rodata +768 text -208
sandbox64 : all -512 bss +32 rodata +800 text -1344Quoted 1 line
Finally, what is the whole list of platforms that grow?
All of them except sandbox64 grow by some amount.
Quoted 3 lines
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.
I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.
Quoted 3 lines
--
Tom
Quoted 22 lines · Brandon L Collins
Quoted 14 lines
From: Tom Rini <[email protected]>
On Mon, Jan 02, 2023 at 04:02:06PM +0000, Maier, Brandon LQuoted 2 lines
I understand Linux replaced their custom zstd code with the Facebook zstd
code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:
Quoted 4 lines
aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0
arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text
+58188.0
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
I haven't had any luck finding another way to decrease build size.
[1]
https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build
After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty?
Quoted 7 lines
That is one of the settings I am already enabling. To clarify from that README, the settings I am enabling in U-Boot are
HUF_FORCE_DECOMPRESS_X1,ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT,ZSTD_NO_INLINE, andZSTD_STRIP_ERROR_STRINGS. Which is what upstream zstd uses for their ownZSTD_LIB_MINIFYoption inlib/libzstd.mk.If I drop the
ZSTD_NO_INLINE, the build sizes increase as follows:aarch64: (for 2/2 boards) all +6212.0 rodata +8.0 text +6204.0
arm: (for 5/5 boards) all +7408.0 text +7408.0sandbox: (for 2/2 boards) all +21928.0 bss -16.0 data +16.0 rodata +224.0 text +21704.0
On top of the previous I assume, OK, darn.
Quoted 31 lines
Quoted 2 lines
Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from.
I looked over each platform, but the size deltas are consistent within each architecture.
What stands out though is that the old ZSTD ARM code is about 4.6kb smaller (25%) than the old ZSTD AARCH64 code. The newer ZSTD doesn't do that, both ARM and AARCH64 are about the same in size. That is why there is a 4k jump in size for just the arm boards. Comparing the functions list between the two, both archs appear similar except
ZSTD_decompressBlock_internal()grows significantly on aarch64 for some reason. I haven't seen any obvious root cause though.Here is the full breakdown of platform size deltas
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0 turris_mox : all +572 rodata +968 text -396 mvebu_espressobin-88f3720: all +499 rodata +963 text -464 arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6 turris_omnia : all +4608 rodata +940 text +3668 m53menlo : all +4460 rodata +940 text +3520 dh_imx6 : all +4460 rodata +940 text +3520 stm32mp15_dhcom_basic: all +4460 rodata +940 text +3520 stm32mp15_dhcor_basic: all +4460 rodata +940 text +3520 sandbox: (for 2/2 boards) all +8.0 rodata +784.0 text -776.0 sandbox : all +528 bss -32 rodata +768 text -208 sandbox64 : all -512 bss +32 rodata +800 text -1344Quoted 1 line
Finally, what is the whole list of platforms that grow?
All of them except sandbox64 grow by some amount.
Quoted 3 lines
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.
Quoted 7 lines
I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.
Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!
Signature · 1 line
Tom
Hi Tom
Quoted 14 lines
On Fri, Jan 06, 2023 at 11:26:36PM +0000, Maier, Brandon L:
Quoted 5 lines
Quoted 4 lines
From: Tom Rini <[email protected]>
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.Quoted 7 lines
I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.
Quoted 3 lines
Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!
I did some deeper investigation and discovered a bunch of functions were being inlined that caused about 3k of growth. As well as a BMI2 feature that can be disabled to save about 800B on x86 platforms. With those it brings the code size change to about -3k for arm64, and +1k for arm. I'll send a v4 patch series shortly.
Quoted 3 lines
--
Tom
Quoted 16 lines · Brandon L Collins
Hi Tom
Quoted 14 lines
On Fri, Jan 06, 2023 at 11:26:36PM +0000, Maier, Brandon L:
Quoted 12 lines
Quoted 4 lines
From: Tom Rini <[email protected]>
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.
Quoted 5 lines
Quoted 3 lines
Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!
I did some deeper investigation and discovered a bunch of functions were being inlined that caused about 3k of growth. As well as a BMI2 feature that can be disabled to save about 800B on x86 platforms. With those it brings the code size change to about -3k for arm64, and +1k for arm. I'll send a v4 patch series shortly.
Oh nice, thanks!
Signature · 1 line
Tom