Skip to contentSkip to conversation

lib: zstd: update to latest Linux zstd 1.5.2

PATCH Superseded Brandon Maier 4 participants 12 messages Dec 27, 2022 – Jan 12, 2023

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 lib: zstd: update to latest Linux zstd 1.5.2

Update the zstd implementation to match Linux zstd 1.5.2 from commit 2aa14b1ab2.

This was motivated by running into decompression corruption issues when trying to uncompress files compressed with newer versions of zstd. zstd users also claim significantly improved decompression times with newer zstd versions which is a side benefit.

Original zstd code was copied from Linux commit 2aa14b1ab2 which is a custom-built implementation based on zstd 1.3.1. Linux switched to an implementation that is a copy of the upstream zstd code in Linux commit e0c1b49f5b, this results in a large code diff. However this should make future updates easier along with other benefits[1].

This commit is a straight mirror of the Linux zstd code, except to:
- update a few #include that do not translate cleanly
- linux/swab.h -> asm/byteorder.h
- linux/limits.h -> linux/kernel.h
- linux/module.h -> linux/compat.h
- remove assert() from debug.h so it doesn't conflict with u-boot's assert()
- strip out the compressor code as was done in the previous u-boot zstd
- update existing zstd users to the new Linux zstd API
- add a new KConfig option that sets zstd build options to minify code based on zstd's ZSTD_LIB_MINIFY[2].

These changes were tested by booting a zstd 1.5.2 compressed kernel inside a FIT. And the squashfs changes by loading a file from zstd compressed squashfs with sqfsload. buildman was used to compile test other boards and check for binary bloat, as follows:

Quoted 8 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS
Summary of 3 commits for 9 boards (8 threads, 1 job per thread)
01: Merge tag 'tpm-23122022' of https://source.denx.de/u-boot/custodians/u-boot-tpm
       arm:  w+   m53menlo dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
   aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0
       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
   sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0

[1] https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8
[2] https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31

---
v2:
- zstd_decompress(): Handle junk at the end of the decompression buffer by checking actual size with zstd_find_frame_compressed_size().

---
 fs/squashfs/sqfs_decompressor.c               |   15 +-
 include/linux/zstd.h                          | 1244 ++------
 include/linux/zstd_errors.h                   |   77 +
 include/linux/zstd_lib.h                      | 2551 +++++++++++++++++
 lib/Kconfig                                   |   15 +
 lib/zstd/Makefile                             |   35 +-
 lib/zstd/bitstream.h                          |  344 ---
 lib/zstd/common/bitstream.h                   |  446 +++
 lib/zstd/common/compiler.h                    |  184 ++
 lib/zstd/common/cpu.h                         |  194 ++
 lib/zstd/common/debug.c                       |   24 +
 lib/zstd/common/debug.h                       |   92 +
 lib/zstd/common/entropy_common.c              |  357 +++
 lib/zstd/common/error_private.c               |   56 +
 lib/zstd/common/error_private.h               |  145 +
 lib/zstd/common/fse.h                         |  711 +++++
 lib/zstd/common/fse_decompress.c              |  390 +++
 lib/zstd/common/huf.h                         |  358 +++
 lib/zstd/common/mem.h                         |  261 ++
 lib/zstd/common/portability_macros.h          |   93 +
 lib/zstd/common/zstd_common.c                 |   83 +
 lib/zstd/common/zstd_deps.h                   |  125 +
 lib/zstd/common/zstd_internal.h               |  443 +++
 lib/zstd/decompress.c                         | 2516 ----------------
 lib/zstd/decompress/huf_decompress.c          | 1740 +++++++++++
 lib/zstd/decompress/zstd_ddict.c              |  241 ++
 lib/zstd/decompress/zstd_ddict.h              |   44 +
 lib/zstd/decompress/zstd_decompress.c         | 2131 ++++++++++++++
 lib/zstd/decompress/zstd_decompress_block.c   | 2072 +++++++++++++
 lib/zstd/decompress/zstd_decompress_block.h   |   68 +
 .../decompress/zstd_decompress_internal.h     |  228 ++
 lib/zstd/decompress_sources.h                 |   34 +
 lib/zstd/entropy_common.c                     |  213 --
 lib/zstd/error_private.h                      |   43 -
 lib/zstd/fse.h                                |  545 ----
 lib/zstd/fse_decompress.c                     |  302 --
 lib/zstd/huf.h                                |  182 --
 lib/zstd/huf_decompress.c                     |  930 ------
 lib/zstd/mem.h                                |  142 -
 lib/zstd/zstd.c                               |   52 +-
 lib/zstd/zstd_common.c                        |   66 -
 lib/zstd/zstd_common_module.c                 |   32 +
 lib/zstd/zstd_decompress_module.c             |  105 +
 lib/zstd/zstd_internal.h                      |  253 --
 lib/zstd/zstd_opt.h                           | 1004 -------
 45 files changed, 13638 insertions(+), 7548 deletions(-)
 create mode 100644 include/linux/zstd_errors.h
 create mode 100644 include/linux/zstd_lib.h
 delete mode 100644 lib/zstd/bitstream.h
 create mode 100644 lib/zstd/common/bitstream.h
 create mode 100644 lib/zstd/common/compiler.h
 create mode 100644 lib/zstd/common/cpu.h
 create mode 100644 lib/zstd/common/debug.c
 create mode 100644 lib/zstd/common/debug.h
 create mode 100644 lib/zstd/common/entropy_common.c
 create mode 100644 lib/zstd/common/error_private.c
 create mode 100644 lib/zstd/common/error_private.h
 create mode 100644 lib/zstd/common/fse.h
 create mode 100644 lib/zstd/common/fse_decompress.c
 create mode 100644 lib/zstd/common/huf.h
 create mode 100644 lib/zstd/common/mem.h
 create mode 100644 lib/zstd/common/portability_macros.h
 create mode 100644 lib/zstd/common/zstd_common.c
 create mode 100644 lib/zstd/common/zstd_deps.h
 create mode 100644 lib/zstd/common/zstd_internal.h
 delete mode 100644 lib/zstd/decompress.c
 create mode 100644 lib/zstd/decompress/huf_decompress.c
 create mode 100644 lib/zstd/decompress/zstd_ddict.c
 create mode 100644 lib/zstd/decompress/zstd_ddict.h
 create mode 100644 lib/zstd/decompress/zstd_decompress.c
 create mode 100644 lib/zstd/decompress/zstd_decompress_block.c
 create mode 100644 lib/zstd/decompress/zstd_decompress_block.h
 create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h
 create mode 100644 lib/zstd/decompress_sources.h
 delete mode 100644 lib/zstd/entropy_common.c
 delete mode 100644 lib/zstd/error_private.h
 delete mode 100644 lib/zstd/fse.h
 delete mode 100644 lib/zstd/fse_decompress.c
 delete mode 100644 lib/zstd/huf.h
 delete mode 100644 lib/zstd/huf_decompress.c
 delete mode 100644 lib/zstd/mem.h
 delete mode 100644 lib/zstd/zstd_common.c
 create mode 100644 lib/zstd/zstd_common_module.c
 create mode 100644 lib/zstd/zstd_decompress_module.c
 delete mode 100644 lib/zstd/zstd_internal.h
 delete mode 100644 lib/zstd/zstd_opt.h
fs/squashfs/sqfs_decompressor.c +9 -6
index d69ddb24a41..fbb1266eea1 100644
@@ -40,7 +40,7 @@ int sqfs_decompressor_init(struct squashfs_ctxt *ctxt)
 #endif
 #if IS_ENABLED(CONFIG_ZSTD)
 	case SQFS_COMP_ZSTD:
-		ctxt->zstd_workspace = malloc(ZSTD_DCtxWorkspaceBound());
+		ctxt->zstd_workspace = malloc(zstd_dctx_workspace_bound());
 		if (!ctxt->zstd_workspace)
 			return -ENOMEM;
 		break;
@@ -99,11 +99,14 @@ static int sqfs_zstd_decompress(struct squashfs_ctxt *ctxt, void *dest,
 	size_t wsize;
 	int ret;
 
-	wsize = ZSTD_DCtxWorkspaceBound();
-	ctx = ZSTD_initDCtx(ctxt->zstd_workspace, wsize);
-	ret = ZSTD_decompressDCtx(ctx, dest, dest_len, source, src_len);
+	wsize = zstd_dctx_workspace_bound();
 
-	return ZSTD_isError(ret);
+	ctx = zstd_init_dctx(ctxt->zstd_workspace, wsize);
+	if (!ctx)
+		return -EINVAL;
+	ret = zstd_decompress_dctx(ctx, dest, dest_len, source, src_len);
+
+	return zstd_is_error(ret);
 }
 #endif /* CONFIG_ZSTD */
 
@@ -140,7 +143,7 @@ int sqfs_decompress(struct squashfs_ctxt *ctxt, void *dest,
 	case SQFS_COMP_ZSTD:
 		ret = sqfs_zstd_decompress(ctxt, dest, *dest_len, source, src_len);
 		if (ret) {
-			printf("ZSTD Error code: %d\n", ZSTD_getErrorCode(ret));
+			printf("ZSTD Error: %d\n", zstd_get_error_code(ret));
 			return -EINVAL;
 		}
 
include/linux/zstd.h +272 -972

1,422 lines not on this page · raw message

include/linux/zstd_errors.h new file +77 -0

77 lines not on this page · raw message

include/linux/zstd_lib.h new file +2551 -0

1,467 lines not on this page · raw message

lib/Kconfig +15 -0
index 3c5a4ab3861..c0a8740f83d 100644

Lines of this file are not shown. View the raw message.

lib/zstd/Makefile +33 -2
index 12170892923..ac198700f50 100644

Lines of this file are not shown. View the raw message.

lib/zstd/bitstream.h deleted +0 -344

344 lines not on this page · raw message

lib/zstd/common/bitstream.h new file +446 -0

446 lines not on this page · raw message

lib/zstd/common/compiler.h new file +184 -0

184 lines not on this page · raw message

lib/zstd/common/cpu.h new file +194 -0

194 lines not on this page · raw message

lib/zstd/common/debug.c new file +24 -0
new file mode 100644
index 00000000000..bb863c9ea61

Lines of this file are not shown. View the raw message.

lib/zstd/common/debug.h new file +92 -0

92 lines not on this page · raw message

lib/zstd/common/entropy_common.c new file +357 -0

357 lines not on this page · raw message

lib/zstd/common/error_private.c new file +56 -0

56 lines not on this page · raw message

lib/zstd/common/error_private.h new file +145 -0

145 lines not on this page · raw message

lib/zstd/common/fse.h new file +711 -0

711 lines not on this page · raw message

lib/zstd/common/fse_decompress.c new file +390 -0

390 lines not on this page · raw message

lib/zstd/common/huf.h new file +358 -0

358 lines not on this page · raw message

lib/zstd/common/mem.h new file +261 -0

261 lines not on this page · raw message

lib/zstd/common/portability_macros.h new file +93 -0

93 lines not on this page · raw message

lib/zstd/common/zstd_common.c new file +83 -0

83 lines not on this page · raw message

lib/zstd/common/zstd_deps.h new file +125 -0

125 lines not on this page · raw message

lib/zstd/common/zstd_internal.h new file +443 -0

443 lines not on this page · raw message

lib/zstd/decompress.c deleted +0 -2516

2,516 lines not on this page · raw message

lib/zstd/decompress/huf_decompress.c new file +1740 -0

1,740 lines not on this page · raw message

lib/zstd/decompress/zstd_ddict.c new file +241 -0

241 lines not on this page · raw message

lib/zstd/decompress/zstd_ddict.h new file +44 -0

44 lines not on this page · raw message

lib/zstd/decompress/zstd_decompress.c new file +2131 -0

2,131 lines not on this page · raw message

lib/zstd/decompress/zstd_decompress_block.c new file +2072 -0

2,072 lines not on this page · raw message

lib/zstd/decompress/zstd_decompress_block.h new file +68 -0

68 lines not on this page · raw message

lib/zstd/decompress/zstd_decompress_internal.h new file +228 -0

228 lines not on this page · raw message

lib/zstd/decompress_sources.h new file +34 -0
new file mode 100644
index 00000000000..a06ca187aab

Lines of this file are not shown. View the raw message.

lib/zstd/entropy_common.c deleted +0 -213

213 lines not on this page · raw message

lib/zstd/error_private.h deleted +0 -43

43 lines not on this page · raw message

lib/zstd/fse.h deleted +0 -545

545 lines not on this page · raw message

lib/zstd/fse_decompress.c deleted +0 -302

302 lines not on this page · raw message

lib/zstd/huf.h deleted +0 -182

182 lines not on this page · raw message

lib/zstd/huf_decompress.c deleted +0 -930

930 lines not on this page · raw message

lib/zstd/mem.h deleted +0 -142

142 lines not on this page · raw message

lib/zstd/zstd.c +24 -28

52 lines not on this page · raw message

lib/zstd/zstd_common.c deleted +0 -66

66 lines not on this page · raw message

lib/zstd/zstd_common_module.c new file +32 -0
new file mode 100644
index 00000000000..73b97c2778c

Lines of this file are not shown. View the raw message.

lib/zstd/zstd_decompress_module.c new file +105 -0

105 lines not on this page · raw message

lib/zstd/zstd_internal.h deleted +0 -253

253 lines not on this page · raw message

lib/zstd/zstd_opt.h deleted +0 -1004

1,004 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.39.0

Part of the diff is not shown here. View the raw message.

#1 lore raw

PATCH v2 2/2 test: compression: add zstd uncompression test

---
 test/Kconfig       |  2 +-
 test/compression.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 72 insertions(+), 1 deletion(-)
test/Kconfig +1 -1
index a6b463e4d06..dd2f082ce63 100644
@@ -64,7 +64,7 @@ endif
 config UT_COMPRESSION
 	bool "Unit test for compression"
 	depends on UNIT_TEST
-	depends on CMDLINE && GZIP_COMPRESSED && BZIP2 && LZMA && LZO && LZ4
+	depends on CMDLINE && GZIP_COMPRESSED && BZIP2 && LZMA && LZO && LZ4 && ZSTD
 	default y
 	help
 	  Enables tests for compression and decompression routines for simple
test/compression.c +71 -0

107 lines not on this page · raw message

Signature · 1 line
2.39.0
#2 lore raw

Re: [PATCH v2 2/2] test: compression: add zstd uncompression test

Quoted 1 line · Brandon Maier

+static const unsigned long zstd_compressed_size = 197;

You can use ARRAY_SIZE (or sizeof) for this.

--Sean

#3 lore raw

Re: [PATCH v2 2/2] test: compression: add zstd uncompression test

Hi Sean,

Quoted 5 lines

From: Sean Anderson <[email protected]>

Quoted 1 line · Brandon Maier

+static const unsigned long zstd_compressed_size = 197;

You can use ARRAY_SIZE (or sizeof) for this.

sizeof() - 1 as it's a string, but I had put the size here manually as that's what all the other tests do. I can switch them all to sizeof() if that is preferred.

Quoted 3 lines

--Sean

Thanks,
Brandon

#4 lore raw

Re: [PATCH v2 2/2] test: compression: add zstd uncompression test

Hi Brandon,

Quoted 10 lines · Brandon L Collins

Hi Sean,

Quoted 5 lines

From: Sean Anderson <[email protected]>

Quoted 1 line · Brandon Maier

+static const unsigned long zstd_compressed_size = 197;

You can use ARRAY_SIZE (or sizeof) for this.

sizeof() - 1 as it's a string, but I had put the size here manually as that's what all the other tests do. I can switch them all to sizeof() if that is preferred.

Yes I think it is better too, but please change the others in a separate patch.

Regards,
Simon

#5 lore raw
In reply to Brandon Maier

Hi Brandon,

Quoted 27 lines · Brandon Maier

Update the zstd implementation to match Linux zstd 1.5.2 from commit 2aa14b1ab2.

This was motivated by running into decompression corruption issues when trying to uncompress files compressed with newer versions of zstd. zstd users also claim significantly improved decompression times with newer zstd versions which is a side benefit.

Original zstd code was copied from Linux commit 2aa14b1ab2 which is a custom-built implementation based on zstd 1.3.1. Linux switched to an implementation that is a copy of the upstream zstd code in Linux commit e0c1b49f5b, this results in a large code diff. However this should make future updates easier along with other benefits[1].

This commit is a straight mirror of the Linux zstd code, except to:
- update a few #include that do not translate cleanly
- linux/swab.h -> asm/byteorder.h
- linux/limits.h -> linux/kernel.h
- linux/module.h -> linux/compat.h
- remove assert() from debug.h so it doesn't conflict with u-boot's assert()
- strip out the compressor code as was done in the previous u-boot zstd
- update existing zstd users to the new Linux zstd API
- add a new KConfig option that sets zstd build options to minify code based on zstd's ZSTD_LIB_MINIFY[2].

Quoted 12 lines

These changes were tested by booting a zstd 1.5.2 compressed kernel inside a FIT. And the squashfs changes by loading a file from zstd compressed squashfs with sqfsload. buildman was used to compile test other boards and check for binary bloat, as follows:

Quoted 7 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS
Summary of 3 commits for 9 boards (8 threads, 1 job per thread)
01: Merge tag 'tpm-23122022' of https://source.denx.de/u-boot/custodians/u-boot-tpm
       arm:  w+   m53menlo dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
   aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0
       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6

That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?

Quoted 98 lines
Quoted 1 line

sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0

[1] https://github.com/torvalds/linux/commit/e0c1b49f5b674cca7b10549c53b3791d0bbc90a8
[2] https://github.com/facebook/zstd/blob/f302ad8811643c428c4e3498e28f53a0578020d3/lib/libzstd.mk#L31

---
v2:
- zstd_decompress(): Handle junk at the end of the decompression buffer by checking actual size with zstd_find_frame_compressed_size().

---
 fs/squashfs/sqfs_decompressor.c               |   15 +-
 include/linux/zstd.h                          | 1244 ++------
 include/linux/zstd_errors.h                   |   77 +
 include/linux/zstd_lib.h                      | 2551 +++++++++++++++++
 lib/Kconfig                                   |   15 +
 lib/zstd/Makefile                             |   35 +-
 lib/zstd/bitstream.h                          |  344 ---
 lib/zstd/common/bitstream.h                   |  446 +++
 lib/zstd/common/compiler.h                    |  184 ++
 lib/zstd/common/cpu.h                         |  194 ++
 lib/zstd/common/debug.c                       |   24 +
 lib/zstd/common/debug.h                       |   92 +
 lib/zstd/common/entropy_common.c              |  357 +++
 lib/zstd/common/error_private.c               |   56 +
 lib/zstd/common/error_private.h               |  145 +
 lib/zstd/common/fse.h                         |  711 +++++
 lib/zstd/common/fse_decompress.c              |  390 +++
 lib/zstd/common/huf.h                         |  358 +++
 lib/zstd/common/mem.h                         |  261 ++
 lib/zstd/common/portability_macros.h          |   93 +
 lib/zstd/common/zstd_common.c                 |   83 +
 lib/zstd/common/zstd_deps.h                   |  125 +
 lib/zstd/common/zstd_internal.h               |  443 +++
 lib/zstd/decompress.c                         | 2516 ----------------
 lib/zstd/decompress/huf_decompress.c          | 1740 +++++++++++
 lib/zstd/decompress/zstd_ddict.c              |  241 ++
 lib/zstd/decompress/zstd_ddict.h              |   44 +
 lib/zstd/decompress/zstd_decompress.c         | 2131 ++++++++++++++
 lib/zstd/decompress/zstd_decompress_block.c   | 2072 +++++++++++++
 lib/zstd/decompress/zstd_decompress_block.h   |   68 +
 .../decompress/zstd_decompress_internal.h     |  228 ++
 lib/zstd/decompress_sources.h                 |   34 +
 lib/zstd/entropy_common.c                     |  213 --
 lib/zstd/error_private.h                      |   43 -
 lib/zstd/fse.h                                |  545 ----
 lib/zstd/fse_decompress.c                     |  302 --
 lib/zstd/huf.h                                |  182 --
 lib/zstd/huf_decompress.c                     |  930 ------
 lib/zstd/mem.h                                |  142 -
 lib/zstd/zstd.c                               |   52 +-
 lib/zstd/zstd_common.c                        |   66 -
 lib/zstd/zstd_common_module.c                 |   32 +
 lib/zstd/zstd_decompress_module.c             |  105 +
 lib/zstd/zstd_internal.h                      |  253 --
 lib/zstd/zstd_opt.h                           | 1004 -------
 45 files changed, 13638 insertions(+), 7548 deletions(-)
 create mode 100644 include/linux/zstd_errors.h
 create mode 100644 include/linux/zstd_lib.h
 delete mode 100644 lib/zstd/bitstream.h
 create mode 100644 lib/zstd/common/bitstream.h
 create mode 100644 lib/zstd/common/compiler.h
 create mode 100644 lib/zstd/common/cpu.h
 create mode 100644 lib/zstd/common/debug.c
 create mode 100644 lib/zstd/common/debug.h
 create mode 100644 lib/zstd/common/entropy_common.c
 create mode 100644 lib/zstd/common/error_private.c
 create mode 100644 lib/zstd/common/error_private.h
 create mode 100644 lib/zstd/common/fse.h
 create mode 100644 lib/zstd/common/fse_decompress.c
 create mode 100644 lib/zstd/common/huf.h
 create mode 100644 lib/zstd/common/mem.h
 create mode 100644 lib/zstd/common/portability_macros.h
 create mode 100644 lib/zstd/common/zstd_common.c
 create mode 100644 lib/zstd/common/zstd_deps.h
 create mode 100644 lib/zstd/common/zstd_internal.h
 delete mode 100644 lib/zstd/decompress.c
 create mode 100644 lib/zstd/decompress/huf_decompress.c
 create mode 100644 lib/zstd/decompress/zstd_ddict.c
 create mode 100644 lib/zstd/decompress/zstd_ddict.h
 create mode 100644 lib/zstd/decompress/zstd_decompress.c
 create mode 100644 lib/zstd/decompress/zstd_decompress_block.c
 create mode 100644 lib/zstd/decompress/zstd_decompress_block.h
 create mode 100644 lib/zstd/decompress/zstd_decompress_internal.h
 create mode 100644 lib/zstd/decompress_sources.h
 delete mode 100644 lib/zstd/entropy_common.c
 delete mode 100644 lib/zstd/error_private.h
 delete mode 100644 lib/zstd/fse.h
 delete mode 100644 lib/zstd/fse_decompress.c
 delete mode 100644 lib/zstd/huf.h
 delete mode 100644 lib/zstd/huf_decompress.c
 delete mode 100644 lib/zstd/mem.h
 delete mode 100644 lib/zstd/zstd_common.c
 create mode 100644 lib/zstd/zstd_common_module.c
 create mode 100644 lib/zstd/zstd_decompress_module.c
 delete mode 100644 lib/zstd/zstd_internal.h
 delete mode 100644 lib/zstd/zstd_opt.h
#6 lore raw

Hello Simon,

Quoted 11 lines

-----Original Message-----
From: Simon Glass <[email protected]>

Quoted 2 lines
Quoted 2 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-

88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basi c,turris_mox,turris_omnia -sS

Quoted 2 lines
Quoted 2 lines

Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of

https://urldefense.com/v3/__https://source.denx.de/u-boot/custodians/u-
boot-tpm__;!!MvWE!SiCDjQxi4IOlpR2LYEviKe-
1_MQirGfhHKYya94BvpeZgxBZ1YBd84kMf-FR0-VX0pY$

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

arm: w+ m53menlo dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0

       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6

That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?

I understand Linux replaced their custom zstd code with the Facebook zstd code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:

aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0

       arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6

sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text +58188.0

I haven't had any luck finding another way to decrease build size.

[1] https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build

Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

sandbox: (for 2/2 boards) all +12.0 rodata +768.0 text -756.0

Thanks,
Brandon

#7 lore raw
Quoted 26 lines · Brandon L Collins

Hello Simon,

Quoted 11 lines

-----Original Message-----
From: Simon Glass <[email protected]>

Quoted 2 lines
Quoted 2 lines
$ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-

88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basi c,turris_mox,turris_omnia -sS

Quoted 2 lines
Quoted 2 lines

Summary of 3 commits for 9 boards (8 threads, 1 job per thread) 01: Merge tag 'tpm-23122022' of

https://urldefense.com/v3/__https://source.denx.de/u-boot/custodians/u-
boot-tpm__;!!MvWE!SiCDjQxi4IOlpR2LYEviKe-
1_MQirGfhHKYya94BvpeZgxBZ1YBd84kMf-FR0-VX0pY$

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines

arm: w+ m53menlo dh_imx6
02: lib: zstd: update to latest Linux zstd 1.5.2
aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0

       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6

That's a pretty stiff penalty. Do you know what has changed? Does it support a lot more optional features? If so, could we make it optional (even if default y)?

I understand Linux replaced their custom zstd code with the Facebook zstd code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:

Quoted 7 lines

aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0

       arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6

sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text +58188.0

I haven't had any luck finding another way to decrease build size.

[1] https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build

After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty? Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from. Finally, what is the whole list of platforms that grow? It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.

Signature · 1 line
Tom
#8 lore raw
Quoted 14 lines

From: Tom Rini <[email protected]>
On Mon, Jan 02, 2023 at 04:02:06PM +0000, Maier, Brandon L

Quoted 2 lines

I understand Linux replaced their custom zstd code with the Facebook zstd

code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:

Quoted 4 lines

aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0

       arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6

sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text

+58188.0

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

I haven't had any luck finding another way to decrease build size.

[1]

https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build

After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty?

That is one of the settings I am already enabling. To clarify from that README, the settings I am enabling in U-Boot are HUF_FORCE_DECOMPRESS_X1, ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT, ZSTD_NO_INLINE, and ZSTD_STRIP_ERROR_STRINGS. Which is what upstream zstd uses for their own ZSTD_LIB_MINIFY option in lib/libzstd.mk.

If I drop the ZSTD_NO_INLINE, the build sizes increase as follows:

aarch64: (for 2/2 boards) all +6212.0 rodata +8.0 text +6204.0

       arm: (for 5/5 boards) all +7408.0 text +7408.0

sandbox: (for 2/2 boards) all +21928.0 bss -16.0 data +16.0 rodata +224.0 text +21704.0

Quoted 2 lines

Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from.

I looked over each platform, but the size deltas are consistent within each architecture.

What stands out though is that the old ZSTD ARM code is about 4.6kb smaller (25%) than the old ZSTD AARCH64 code. The newer ZSTD doesn't do that, both ARM and AARCH64 are about the same in size. That is why there is a 4k jump in size for just the arm boards. Comparing the functions list between the two, both archs appear similar except ZSTD_decompressBlock_internal() grows significantly on aarch64 for some reason. I haven't seen any obvious root cause though.

Here is the full breakdown of platform size deltas

aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0

            turris_mox     : all +572 rodata +968 text -396
            mvebu_espressobin-88f3720: all +499 rodata +963 text -464
       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
            turris_omnia   : all +4608 rodata +940 text +3668
            m53menlo       : all +4460 rodata +940 text +3520
            dh_imx6        : all +4460 rodata +940 text +3520
            stm32mp15_dhcom_basic: all +4460 rodata +940 text +3520
            stm32mp15_dhcor_basic: all +4460 rodata +940 text +3520
   sandbox: (for 2/2 boards) all +8.0 rodata +784.0 text -776.0
            sandbox        : all +528 bss -32 rodata +768 text -208
            sandbox64      : all -512 bss +32 rodata +800 text -1344
Quoted 1 line

Finally, what is the whole list of platforms that grow?

All of them except sandbox64 grow by some amount.

Quoted 3 lines

It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.

I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.

Quoted 3 lines

--
Tom

#9 lore raw
Quoted 22 lines · Brandon L Collins
Quoted 14 lines

From: Tom Rini <[email protected]>
On Mon, Jan 02, 2023 at 04:02:06PM +0000, Maier, Brandon L

Quoted 2 lines

I understand Linux replaced their custom zstd code with the Facebook zstd

code, so the changes are widespread. The new zstd does support many optional performance features as documented in zstd's lib/README[1], which I have already added a KConfig to disable all of them in this patch. For reference, if I enable the performance options the build size increases as so:

Quoted 4 lines

aarch64: (for 2/2 boards) all +27432.0 rodata +320.0 text +27112.0

       arm: (for 5/5 boards) all +21801.6 rodata +256.0 text +21545.6

sandbox: (for 2/2 boards) all +59004.0 bss +16.0 rodata +800.0 text

+58188.0

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

I haven't had any luck finding another way to decrease build size.

[1]

https://github.com/facebook/zstd/blob/dev/lib/README.md#modular-build

After skimming over that, did you see if not disabling inlining, and letting the compiler figure it out is a big size penalty?

Quoted 7 lines

That is one of the settings I am already enabling. To clarify from that README, the settings I am enabling in U-Boot are HUF_FORCE_DECOMPRESS_X1, ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT, ZSTD_NO_INLINE, and ZSTD_STRIP_ERROR_STRINGS. Which is what upstream zstd uses for their own ZSTD_LIB_MINIFY option in lib/libzstd.mk.

If I drop the ZSTD_NO_INLINE, the build sizes increase as follows:

aarch64: (for 2/2 boards) all +6212.0 rodata +8.0 text +6204.0

       arm: (for 5/5 boards) all +7408.0 text +7408.0

sandbox: (for 2/2 boards) all +21928.0 bss -16.0 data +16.0 rodata +224.0 text +21704.0

On top of the previous I assume, OK, darn.

Quoted 31 lines
Quoted 2 lines

Also, while the arch-level details are good, a specific platform set of numbers instead might give some ideas on where maybe reductions could come from.

I looked over each platform, but the size deltas are consistent within each architecture.

What stands out though is that the old ZSTD ARM code is about 4.6kb smaller (25%) than the old ZSTD AARCH64 code. The newer ZSTD doesn't do that, both ARM and AARCH64 are about the same in size. That is why there is a 4k jump in size for just the arm boards. Comparing the functions list between the two, both archs appear similar except ZSTD_decompressBlock_internal() grows significantly on aarch64 for some reason. I haven't seen any obvious root cause though.

Here is the full breakdown of platform size deltas

   aarch64: (for 2/2 boards) all +535.5 rodata +965.5 text -430.0
            turris_mox     : all +572 rodata +968 text -396
            mvebu_espressobin-88f3720: all +499 rodata +963 text -464
       arm: (for 5/5 boards) all +4489.6 rodata +940.0 text +3549.6
            turris_omnia   : all +4608 rodata +940 text +3668
            m53menlo       : all +4460 rodata +940 text +3520
            dh_imx6        : all +4460 rodata +940 text +3520
            stm32mp15_dhcom_basic: all +4460 rodata +940 text +3520
            stm32mp15_dhcor_basic: all +4460 rodata +940 text +3520
   sandbox: (for 2/2 boards) all +8.0 rodata +784.0 text -776.0
            sandbox        : all +528 bss -32 rodata +768 text -208
            sandbox64      : all -512 bss +32 rodata +800 text -1344
Quoted 1 line

Finally, what is the whole list of platforms that grow?

All of them except sandbox64 grow by some amount.

Quoted 3 lines

It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.

Quoted 7 lines

I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.

Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!

Signature · 1 line
Tom
#10 lore raw

Hi Tom

Quoted 14 lines

On Fri, Jan 06, 2023 at 11:26:36PM +0000, Maier, Brandon L:

Quoted 5 lines
Quoted 4 lines

From: Tom Rini <[email protected]>
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.

Quoted 7 lines

I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.

Quoted 3 lines

Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!

I did some deeper investigation and discovered a bunch of functions were being inlined that caused about 3k of growth. As well as a BMI2 feature that can be disabled to save about 800B on x86 platforms. With those it brings the code size change to about -3k for arm64, and +1k for arm. I'll send a v4 patch series shortly.

Quoted 3 lines

--
Tom

#11 lore raw
Quoted 16 lines · Brandon L Collins

Hi Tom

Quoted 14 lines

On Fri, Jan 06, 2023 at 11:26:36PM +0000, Maier, Brandon L:

Quoted 12 lines
Quoted 4 lines

From: Tom Rini <[email protected]>
It's all not ideal, yes, but it looks like BTRFS is the main user, right now, which isn't widely enabled. So maybe we can look towards improving upstream a bit here, if motivated.

I can't speak for the BTRFS code as I don't have a suitable platform to test it on. But the zstd code does appear to be broken on master. Running the zstd compression test in sandbox or sandbox64 both segfault. And I have similar problems trying to decompress FIT images on our AARCH64 platform. The primary motivation to this patch is that the new version does work. And this seems like the appropriate fix given Linux has abandoned the original version of this code.

Quoted 5 lines
Quoted 3 lines

Well, I guess in the end, non-functional smaller code is factually worse than larger functional code, so, I'll pick this up post v2023.01, thanks!

I did some deeper investigation and discovered a bunch of functions were being inlined that caused about 3k of growth. As well as a BMI2 feature that can be disabled to save about 800B on x86 platforms. With those it brings the code size change to about -3k for arm64, and +1k for arm. I'll send a v4 patch series shortly.

Oh nice, thanks!

Signature · 1 line
Tom