fs/squashfs: Add new decompression algorithms
cover fs/squashfs: Add new decompression algorithms
Hello,
Following the SquashFS support, this series adds support for LZO and ZSTD algorithms. The only compression type enabled by default is ZLIB, so LZO and ZSTD need to be manually selected.
Joao Marcos Costa (2):
fs/squashfs: add support for LZO decompression
fs/squashfs: add support for ZSTD decompression
fs/squashfs/sqfs_decompressor.c | 60 +++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-)
Signature · 1 line
2.17.1
PATCH 1/2 fs/squashfs: add support for LZO decompression
Add call to lzo's lzo1x_decompress_safe() and rename source length parameter from 'lenp' to 'src_len'.
--- fs/squashfs/sqfs_decompressor.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)
fs/squashfs/sqfs_decompressor.c +19 -2
@@ -9,6 +9,11 @@#include <stdint.h> #include <stdio.h> #include <stdlib.h> + +#if IS_ENABLED (CONFIG_LZO ) +#include <linux/lzo.h> +#endif +#if IS_ENABLED (CONFIG_ZLIB )#include <u-boot/zlib.h> #endif
@@ -35,20 +40,32 @@ static void zlib_decompression_status(int ret)#endif int sqfs_decompress (u16 comp_type,void *dest,unsigned long *dest_len, -void *source,u32 lenp) +void *source,u32 src_len) {int ret =0 ;switch (comp_type) {#if IS_ENABLED (CONFIG_ZLIB )case SQFS_COMP_ZLIB : - ret =uncompress (dest, dest_len, source, lenp); + ret =uncompress (dest, dest_len, source, src_len);if (ret) {zlib_decompression_status (ret);return -EINVAL ; }break ; +#endif +#if IS_ENABLED (CONFIG_LZO ) +case SQFS_COMP_LZO : { +size_t lzo_dest_len = *dest_len; + ret =lzo1x_decompress_safe (source, src_len, dest, &lzo_dest_len); +if (ret) { +printf ("LZO decompression failed. Error code: %d\n" , ret); +return -EINVAL ; + } + +break ; + }#endif default :printf ("Error: unknown compression type.\n" );
Signature · 1 line
2.17.1
PATCH 2/2 fs/squashfs: add support for ZSTD decompression
In reply to Joao Marcos CostaAdd call to ZSTD's ZSTD_decompressDCtx(). In this use case, the caller can upper bound the decompressed size, which will be the SquashFS data block (or metadata block) size, so there is no need to use streaming API.
--- fs/squashfs/sqfs_decompressor.c | 39 +++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+)
fs/squashfs/sqfs_decompressor.c +39 -0
@@ -18,6 +18,10 @@#include <u-boot/zlib.h> #endif +#if IS_ENABLED (CONFIG_ZSTD ) +#include <linux/zstd.h> +#endif +#include "sqfs_decompressor.h" #include "sqfs_filesystem.h" #include "sqfs_utils.h"
@@ -39,6 +43,31 @@ static void zlib_decompression_status(int ret) }#endif +#if IS_ENABLED (CONFIG_ZSTD ) +static int sqfs_zstd_decompress (void *dest,unsigned long dest_len, +void *source,u32 src_len) +{ +void *workspace; +ZSTD_DCtx *ctx; +size_t wsize; +int ret; + + wsize =ZSTD_DCtxWorkspaceBound (); + + workspace =malloc (wsize); +if (!workspace) +return -ENOMEM ; + + ctx =ZSTD_initDCtx (workspace, wsize); + + ret =ZSTD_decompressDCtx (ctx, dest, dest_len, source, src_len); + +free (workspace); + +return ZSTD_isError (ret); +} +#endif /* CONFIG_ZSTD */ +int sqfs_decompress (u16 comp_type,void *dest,unsigned long *dest_len,void *source,u32 src_len) {
@@ -67,6 +96,16 @@ int sqfs_decompress(u16 comp_type, void *dest, unsigned long *dest_len, break; }#endif +#if IS_ENABLED (CONFIG_ZSTD ) +case SQFS_COMP_ZSTD : + ret =sqfs_zstd_decompress (dest, *dest_len, source, src_len); +if (ret) { +printf ("ZSTD Error code: %d\n" ,ZSTD_getErrorCode (ret)); +return -EINVAL ; + } + +break ; +#endif /* CONFIG_ZSTD */ default :printf ("Error: unknown compression type.\n" );return -EINVAL ;
Signature · 1 line
2.17.1
PATCH 2/2 fs/squashfs: add support for ZSTD decompression
fs/squashfs/sqfs_decompressor.c static void zlib_decompression_status(int ret) · line 55
Quoted 10 lines · On Tue, 11 Aug 2020 15:17:55 +0200 Joao Marcos Costa…
fs/squashfs/sqfs_decompressor.c +10
-0+#if IS_ENABLED (CONFIG_ZSTD ) +static int sqfs_zstd_decompress (void *dest,unsigned long dest_len, +void *source,u32 src_len) +{ +void *workspace; +ZSTD_DCtx *ctx; +size_t wsize; +int ret; + + wsize =ZSTD_DCtxWorkspaceBound ();
So apparently this "workspace" has a constant size, which does not depend on the size of the input buffer. Correct ?
If that's the case, can we instead allocate it once for all in sqfs_probe() if the filesystem is zstd-compressed ? I.e perhaps sqfs_probe() could call a sqfs_decompressor_init() function, implemented in sqfs_decompressor.c, which will do this sort of initialization. And of course this memory area would be freed up in sqfs_close(), calling a sqfs_decompressor_{deinit,cleanup,close} function.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/2 fs/squashfs: add support for ZSTD decompression
Quoted 16 lines · On Tue, 11 Aug 2020 15:29:51 +0200 Thomas Petazzoni…
fs/squashfs/sqfs_decompressor.c static void zlib_decompression_status(int ret) · line 55
Quoted 10 lines · On Tue, 11 Aug 2020 15:17:55 +0200 Joao Marcos Costa…
fs/squashfs/sqfs_decompressor.c +10
-0+#if IS_ENABLED (CONFIG_ZSTD ) +static int sqfs_zstd_decompress (void *dest,unsigned long dest_len, +void *source,u32 src_len) +{ +void *workspace; +ZSTD_DCtx *ctx; +size_t wsize; +int ret; + + wsize =ZSTD_DCtxWorkspaceBound ();So apparently this "workspace" has a constant size, which does not depend on the size of the input buffer. Correct ?
Yes, correct.
Quoted 7 lines
If that's the case, can we instead allocate it once for all in
sqfs_probe()if the filesystem is zstd-compressed ? I.e perhapssqfs_probe()could call asqfs_decompressor_init()function, implemented insqfs_decompressor.c, which will do this sort of initialization. And of course this memory area would be freed up insqfs_close(), calling a sqfs_decompressor_{deinit,cleanup,close} function.
Sure, it can be done. I will prepare a v2 addressing your suggestion. Thanks!
Quoted 3 lines
Best regards,
Thomas
Best regards,
Joao
cover fs/squashfs: Add new decompression algorithms
In reply to Joao Marcos CostaQuoted 6 lines · Joao Marcos Costa
Hello,
Following the SquashFS support, this series adds support for LZO and ZSTD algorithms. The only compression type enabled by default is ZLIB, so LZO and ZSTD need to be manually selected.
Quoted 6 lines
Joao Marcos Costa (2):
fs/squashfs: add support for LZO decompression
fs/squashfs: add support for ZSTD decompressionfs/squashfs/sqfs_decompressor.c | 60 +++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-)
Can we add tests for this to sandbox? Thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200811/2d9c1636/attachment.sig>
cover fs/squashfs: Add new decompression algorithms
Quoted 12 lines · On Tue, 11 Aug 2020 11:13:41 -0400 Tom Rini <[email protected]> wrote:
Quoted 10 lines · Joao Marcos Costa
Hello,
Following the SquashFS support, this series adds support for LZO and ZSTD algorithms. The only compression type enabled by default is ZLIB, so LZO and ZSTD need to be manually selected.
Joao Marcos Costa (2):
fs/squashfs: add support for LZO decompression
fs/squashfs: add support for ZSTD decompression
Quoted 5 lines
Quoted 3 lines
fs/squashfs/sqfs_decompressor.c | 60+++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-)
Can we add tests for this to sandbox? Thanks!
Sure, I will enable the lzo and zstd configs for, and I will make some actual changes to the test scripts I added, because right now they are not really "scalable" (thinking of new compression algorithms).
Best regards,
Joao
cover fs/squashfs: Add new decompression algorithms
In reply to Tom RiniQuoted 12 lines · On Tue, 11 Aug 2020 11:13:41 -0400 Tom Rini <[email protected]> wrote:
Quoted 10 lines · Joao Marcos Costa
Hello,
Following the SquashFS support, this series adds support for LZO and ZSTD algorithms. The only compression type enabled by default is ZLIB, so LZO and ZSTD need to be manually selected.
Joao Marcos Costa (2):
fs/squashfs: add support for LZO decompression
fs/squashfs: add support for ZSTD decompression
Quoted 6 lines
Quoted 3 lines
fs/squashfs/sqfs_decompressor.c | 60+++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-)
Can we add tests for this to sandbox? Thanks!
Hello,
Is there any way to check the package version with the python API? I already use 'requiredtool' marker, but apparently it is not enough. My host machine (Ubuntu 18.04) has mksquashfs at its version 4.3, without zstd support. However, v4.4 has such support, and I had to download it and build it myself to use zstd compression. The tests will fail if the user has an older version of mksquashfs installed. Would you have any suggestion on how to proceed?
Best regards,
Joao
cover fs/squashfs: Add new decompression algorithms
Quoted 24 lines · Joao Marcos Costa
Quoted 12 lines · On Tue, 11 Aug 2020 11:13:41 -0400 Tom Rini <[email protected]> wrote:
Quoted 10 lines · Joao Marcos Costa
Hello,
Following the SquashFS support, this series adds support for LZO and ZSTD algorithms. The only compression type enabled by default is ZLIB, so LZO and ZSTD need to be manually selected.
Joao Marcos Costa (2):
fs/squashfs: add support for LZO decompression
fs/squashfs: add support for ZSTD decompressionQuoted 6 lines
Quoted 3 lines
fs/squashfs/sqfs_decompressor.c | 60+++++++++++++++++++++++++++++++-- 1 file changed, 58 insertions(+), 2 deletions(-)
Can we add tests for this to sandbox? Thanks!
Hello,
Quoted 7 lines
Is there any way to check the package version with the python API? I already use 'requiredtool' marker, but apparently it is not enough. My host machine (Ubuntu 18.04) has mksquashfs at its version 4.3, without zstd support. However, v4.4 has such support, and I had to download it and build it myself to use zstd compression. The tests will fail if the user has an older version of mksquashfs installed. Would you have any suggestion on how to proceed?
This sounds like the cases where we have to build the tools we need. There's an example in .travis.yml for QEMU already, and we use
https://gitlab.denx.de/u-boot/gitlab-ci-runner/ for Azure/GitLab. Testing those with your own container does require setting up an Azure account or using your own GitLab + runner.
That said, if you unit test things and are confident it won't blow up when I test it, that can be good enough. Thanks!
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: not available URL: <https://lists.denx.de/pipermail/u-boot/attachments/20200812/957e2d3f/attachment.sig>