Skip to contentSkip to conversation

btrfs: Add zstd support to grub btrfs

PATCH Superseded Nick Terrell 2 participants 7 messages Oct 31, 2018 – Nov 7, 2018

Subsystems (3)
Kbuild Btrfs file system Zstd

v4 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Nick Terrell

author maintainer
#0 lore raw

v4 cover btrfs: Add zstd support to grub btrfs

Hi all,

This patch set imports the upstream zstd library, adds zstd support to the btrfs module, and adds a test case. I've also tested the patch set by storing my boot partition in btrfs with and without zstd compression and rebooting.

The fist patch imports the files needed to support zstd decompression from zstd-1.3.6 as-is. It is a very large patch. In case it doesn't make it, I've included the commit hash and the script I used to download the files.

Upstream zstd commit hash: 4fa456d7f12f8b27bd3b2f5dfd4f46898cb31c24 Upstream zstd commit name: Merge pull request #1354 from facebook/dev

```
#!/bin/sh -e

curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz
curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz.sha256
sha256sum --check zstd-1.3.6.tar.gz.sha256
tar xzf zstd-1.3.6.tar.gz

SRC_LIB="zstd-1.3.6/lib"
DST_LIB="grub-core/lib/zstd"
rm -rf $DST_LIB
mkdir -p $DST_LIB
cp $SRC_LIB/zstd.h $DST_LIB/
cp $SRC_LIB/common/*.[hc] $DST_LIB/
cp $SRC_LIB/decompress/*.[hc] $DST_LIB/
rm $DST_LIB/{pool.[hc],threading.[hc]}
rm -rf zstd-1.3.6*
echo SUCCESS!
```

Best,
Nick Terrell

Changelog:

v1 -> v2:
- Switch to upstream zstd-1.3.6 and drop all the local patches.
- Fix comments from Daniel Kiper.

v2 -> v3:
- Remove an extra file accidentally included in the first patch.
- Use grub_error() to set grub_errno in grub_btrfs_zstd_decompress().
- Fix style and formatting comments.

v3 -> v4:
- Put zstd in its own module.
- Update commit messages.
- Use attribute unused.
- Rebase on top of RAID patchset.

Nick Terrell (2):
Import upstream zstd-1.3.6
btrfs: Add zstd support to grub btrfs

 Makefile.util.def                    |   10 +-
 grub-core/Makefile.core.def          |   17 +-
 grub-core/fs/btrfs.c                 |  108 +-
 grub-core/lib/zstd/bitstream.h       |  458 ++++
 grub-core/lib/zstd/compiler.h        |  133 ++
 grub-core/lib/zstd/cpu.h             |  215 ++
 grub-core/lib/zstd/debug.c           |   44 +
 grub-core/lib/zstd/debug.h           |  123 +
 grub-core/lib/zstd/entropy_common.c  |  236 ++
 grub-core/lib/zstd/error_private.c   |   48 +
 grub-core/lib/zstd/error_private.h   |   76 +
 grub-core/lib/zstd/fse.h             |  708 ++++++
 grub-core/lib/zstd/fse_decompress.c  |  309 +++
 grub-core/lib/zstd/huf.h             |  334 +++
 grub-core/lib/zstd/huf_decompress.c  | 1096 +++++++++
 grub-core/lib/zstd/mem.h             |  374 ++++
 grub-core/lib/zstd/module.c          |    3 +
 grub-core/lib/zstd/xxhash.c          |  876 ++++++++
 grub-core/lib/zstd/xxhash.h          |  305 +++
 grub-core/lib/zstd/zstd.h            | 1516 +++++++++++++
 grub-core/lib/zstd/zstd_common.c     |   81 +
 grub-core/lib/zstd/zstd_decompress.c | 3108 ++++++++++++++++++++++++++
 grub-core/lib/zstd/zstd_errors.h     |   92 +
 grub-core/lib/zstd/zstd_internal.h   |  257 +++
 tests/btrfs_test.in                  |    1 +
 tests/util/grub-fs-tester.in         |    2 +-
 26 files changed, 10526 insertions(+), 4 deletions(-)
 create mode 100644 grub-core/lib/zstd/bitstream.h
 create mode 100644 grub-core/lib/zstd/compiler.h
 create mode 100644 grub-core/lib/zstd/cpu.h
 create mode 100644 grub-core/lib/zstd/debug.c
 create mode 100644 grub-core/lib/zstd/debug.h
 create mode 100644 grub-core/lib/zstd/entropy_common.c
 create mode 100644 grub-core/lib/zstd/error_private.c
 create mode 100644 grub-core/lib/zstd/error_private.h
 create mode 100644 grub-core/lib/zstd/fse.h
 create mode 100644 grub-core/lib/zstd/fse_decompress.c
 create mode 100644 grub-core/lib/zstd/huf.h
 create mode 100644 grub-core/lib/zstd/huf_decompress.c
 create mode 100644 grub-core/lib/zstd/mem.h
 create mode 100644 grub-core/lib/zstd/module.c
 create mode 100644 grub-core/lib/zstd/xxhash.c
 create mode 100644 grub-core/lib/zstd/xxhash.h
 create mode 100644 grub-core/lib/zstd/zstd.h
 create mode 100644 grub-core/lib/zstd/zstd_common.c
 create mode 100644 grub-core/lib/zstd/zstd_decompress.c
 create mode 100644 grub-core/lib/zstd/zstd_errors.h
 create mode 100644 grub-core/lib/zstd/zstd_internal.h

--
2.17.1

author maintainer
#1 lore raw

PATCH v4 1/2 Import upstream zstd-1.3.6

Import zstd-1.3.6 from upstream [1]. Only the files need for decompression are imported. Additionally makes zstd a module by adding module.c which contains the license, and updates Makefile.core.def.

I used the latest zstd release, which includes patches [2] to build cleanly in GRUB.

Upstream zstd commit hash: 4fa456d7f12f8b27bd3b2f5dfd4f46898cb31c24 Upstream zstd commit name: Merge pull request #1354 from facebook/dev

I've included the script used to import zstd-1.3.6 below.

[1] https://github.com/facebook/zstd/releases/tag/v1.3.6
[2] https://github.com/facebook/zstd/pull/1344

```
#!/bin/sh -e

curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz
curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz.sha256
sha256sum --check zstd-1.3.6.tar.gz.sha256
tar xzf zstd-1.3.6.tar.gz

SRC_LIB="zstd-1.3.6/lib"
DST_LIB="grub-core/lib/zstd"
rm -rf $DST_LIB
mkdir -p $DST_LIB
cp $SRC_LIB/zstd.h $DST_LIB/
cp $SRC_LIB/common/*.[hc] $DST_LIB/
cp $SRC_LIB/decompress/*.[hc] $DST_LIB/
rm $DST_LIB/{pool.[hc],threading.[hc]}
rm -rf zstd-1.3.6*
echo SUCCESS!
```
Signed-off-by Nick Terrell <[email protected]>

---
v1 -> v2:
- Switch to upstream zstd-1.3.6 rather than using the kernel zstd.

v2 -> v3:
- Remove the symlink "up" accidently incldued in the patch.

v3 -> v4:
- Put zstd in its own module.
- Update commit messages.

 grub-core/Makefile.core.def          |   15 +
 grub-core/lib/zstd/bitstream.h       |  458 ++++
 grub-core/lib/zstd/compiler.h        |  133 ++
 grub-core/lib/zstd/cpu.h             |  215 ++
 grub-core/lib/zstd/debug.c           |   44 +
 grub-core/lib/zstd/debug.h           |  123 +
 grub-core/lib/zstd/entropy_common.c  |  236 ++
 grub-core/lib/zstd/error_private.c   |   48 +
 grub-core/lib/zstd/error_private.h   |   76 +
 grub-core/lib/zstd/fse.h             |  708 ++++++
 grub-core/lib/zstd/fse_decompress.c  |  309 +++
 grub-core/lib/zstd/huf.h             |  334 +++
 grub-core/lib/zstd/huf_decompress.c  | 1096 +++++++++
 grub-core/lib/zstd/mem.h             |  374 ++++
 grub-core/lib/zstd/module.c          |    3 +
 grub-core/lib/zstd/xxhash.c          |  876 ++++++++
 grub-core/lib/zstd/xxhash.h          |  305 +++
 grub-core/lib/zstd/zstd.h            | 1516 +++++++++++++
 grub-core/lib/zstd/zstd_common.c     |   81 +
 grub-core/lib/zstd/zstd_decompress.c | 3108 ++++++++++++++++++++++++++
 grub-core/lib/zstd/zstd_errors.h     |   92 +
 grub-core/lib/zstd/zstd_internal.h   |  257 +++
 22 files changed, 10407 insertions(+)
 create mode 100644 grub-core/lib/zstd/bitstream.h
 create mode 100644 grub-core/lib/zstd/compiler.h
 create mode 100644 grub-core/lib/zstd/cpu.h
 create mode 100644 grub-core/lib/zstd/debug.c
 create mode 100644 grub-core/lib/zstd/debug.h
 create mode 100644 grub-core/lib/zstd/entropy_common.c
 create mode 100644 grub-core/lib/zstd/error_private.c
 create mode 100644 grub-core/lib/zstd/error_private.h
 create mode 100644 grub-core/lib/zstd/fse.h
 create mode 100644 grub-core/lib/zstd/fse_decompress.c
 create mode 100644 grub-core/lib/zstd/huf.h
 create mode 100644 grub-core/lib/zstd/huf_decompress.c
 create mode 100644 grub-core/lib/zstd/mem.h
 create mode 100644 grub-core/lib/zstd/module.c
 create mode 100644 grub-core/lib/zstd/xxhash.c
 create mode 100644 grub-core/lib/zstd/xxhash.h
 create mode 100644 grub-core/lib/zstd/zstd.h
 create mode 100644 grub-core/lib/zstd/zstd_common.c
 create mode 100644 grub-core/lib/zstd/zstd_decompress.c
 create mode 100644 grub-core/lib/zstd/zstd_errors.h
 create mode 100644 grub-core/lib/zstd/zstd_internal.h
grub-core/Makefile.core.def +15 -0
index 9590e87d9..540a31f78 100644
@@ -1260,6 +1260,21 @@ module = {
   common = fs/bfs.c;
 };
 
+module = {
+  name = zstd;
+  common = lib/zstd/debug.c;
+  common = lib/zstd/entropy_common.c;
+  common = lib/zstd/error_private.c;
+  common = lib/zstd/fse_decompress.c;
+  common = lib/zstd/huf_decompress.c;
+  common = lib/zstd/module.c;
+  common = lib/zstd/xxhash.c;
+  common = lib/zstd/zstd_common.c;
+  common = lib/zstd/zstd_decompress.c;
+  cflags = '$(CFLAGS_POSIX) -Wno-undef';
+  cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/zstd';
+};
+
 module = {
   name = btrfs;
   common = fs/btrfs.c;
grub-core/lib/zstd/bitstream.h new file +458 -0

458 lines not on this page · raw message

grub-core/lib/zstd/compiler.h new file +133 -0

133 lines not on this page · raw message

grub-core/lib/zstd/cpu.h new file +215 -0

215 lines not on this page · raw message

grub-core/lib/zstd/debug.c new file +44 -0

44 lines not on this page · raw message

grub-core/lib/zstd/debug.h new file +123 -0

123 lines not on this page · raw message

grub-core/lib/zstd/entropy_common.c new file +236 -0

236 lines not on this page · raw message

grub-core/lib/zstd/error_private.c new file +48 -0

48 lines not on this page · raw message

grub-core/lib/zstd/error_private.h new file +76 -0

76 lines not on this page · raw message

grub-core/lib/zstd/fse.h new file +708 -0

708 lines not on this page · raw message

grub-core/lib/zstd/fse_decompress.c new file +309 -0

309 lines not on this page · raw message

grub-core/lib/zstd/huf.h new file +334 -0

334 lines not on this page · raw message

grub-core/lib/zstd/huf_decompress.c new file +1096 -0

295 lines not on this page · raw message

grub-core/lib/zstd/mem.h new file +374 -0

374 lines not on this page · raw message

grub-core/lib/zstd/module.c new file +3 -0
new file mode 100644
index 000000000..e4d0cace8

Lines of this file are not shown. View the raw message.

grub-core/lib/zstd/xxhash.c new file +876 -0

876 lines not on this page · raw message

grub-core/lib/zstd/xxhash.h new file +305 -0

305 lines not on this page · raw message

grub-core/lib/zstd/zstd.h new file +1516 -0

1,516 lines not on this page · raw message

grub-core/lib/zstd/zstd_common.c new file +81 -0

81 lines not on this page · raw message

grub-core/lib/zstd/zstd_decompress.c new file +3108 -0

3,108 lines not on this page · raw message

grub-core/lib/zstd/zstd_errors.h new file +92 -0

92 lines not on this page · raw message

grub-core/lib/zstd/zstd_internal.h new file +257 -0

257 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

--
2.17.1

Part of the diff is not shown here. View the raw message.

#2 lore raw

Re: [PATCH v4 1/2] Import upstream zstd-1.3.6

Quoted 35 lines · Nick Terrell

Import zstd-1.3.6 from upstream [1]. Only the files need for decompression are imported. Additionally makes zstd a module by adding module.c which contains the license, and updates Makefile.core.def.

I used the latest zstd release, which includes patches [2] to build cleanly in GRUB.

Upstream zstd commit hash: 4fa456d7f12f8b27bd3b2f5dfd4f46898cb31c24 Upstream zstd commit name: Merge pull request #1354 from facebook/dev

I've included the script used to import zstd-1.3.6 below.

[1] https://github.com/facebook/zstd/releases/tag/v1.3.6
[2] https://github.com/facebook/zstd/pull/1344

```
#!/bin/sh -e

curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz
curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz.sha256
sha256sum --check zstd-1.3.6.tar.gz.sha256
tar xzf zstd-1.3.6.tar.gz

SRC_LIB="zstd-1.3.6/lib"
DST_LIB="grub-core/lib/zstd"
rm -rf $DST_LIB
mkdir -p $DST_LIB
cp $SRC_LIB/zstd.h $DST_LIB/
cp $SRC_LIB/common/*.[hc] $DST_LIB/
cp $SRC_LIB/decompress/*.[hc] $DST_LIB/
rm $DST_LIB/{pool.[hc],threading.[hc]}
rm -rf zstd-1.3.6*
echo SUCCESS!
```
Quoted 1 line
Signed-off-by Nick Terrell <[email protected]>

I have a few concerns. First of all I can see that you include a lot of standard headers, e.g. stdlib.h, string.h, stddef.h, etc. Where do they come from? OS? If yes that is wrong. Additionally, I think that you should not use standard types, e.g. size_t, because this may not work if you cross compile. You have to use, e.g. grub_size_t. IMO the simplest solution would be definition and usage of relevant zstd_* types, e.g. zstd_size_t. Then e.g. zstd_size_t should be defined as grub_size_t.

[...]

grub-core/lib/zstd/module.c

Quoted 6 lines
grub-core/lib/zstd/module.c new file +0 -0
new file mode 100644
index 000000000..e4d0cace8
@@ -0,0 +1,3 @@

Could you copy the header e.g. from grub-core/loader/multiboot_mbi2.c? Of course you have to update the dates accordingly.

grub-core/lib/zstd/module.c line 3

Quoted 3 lines
grub-core/lib/zstd/module.c new file +3 -0
+#include <grub/dl.h>
+
+GRUB_MOD_LICENSE ("GPLv3");

Daniel

author maintainer
#3 lore raw

Re: [PATCH v4 1/2] Import upstream zstd-1.3.6

Quoted 42 lines · Daniel Kiper
Quoted 35 lines · Nick Terrell

Import zstd-1.3.6 from upstream [1]. Only the files need for decompression are imported. Additionally makes zstd a module by adding module.c which contains the license, and updates Makefile.core.def.

I used the latest zstd release, which includes patches [2] to build cleanly in GRUB.

Upstream zstd commit hash: 4fa456d7f12f8b27bd3b2f5dfd4f46898cb31c24 Upstream zstd commit name: Merge pull request #1354 from facebook/dev

I've included the script used to import zstd-1.3.6 below.

[1] https://github.com/facebook/zstd/releases/tag/v1.3.6
[2] https://github.com/facebook/zstd/pull/1344

```
#!/bin/sh -e

curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz
curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz.sha256
sha256sum --check zstd-1.3.6.tar.gz.sha256
tar xzf zstd-1.3.6.tar.gz

SRC_LIB="zstd-1.3.6/lib"
DST_LIB="grub-core/lib/zstd"
rm -rf $DST_LIB
mkdir -p $DST_LIB
cp $SRC_LIB/zstd.h $DST_LIB/
cp $SRC_LIB/common/*.[hc] $DST_LIB/
cp $SRC_LIB/decompress/*.[hc] $DST_LIB/
rm $DST_LIB/{pool.[hc],threading.[hc]}
rm -rf zstd-1.3.6*
echo SUCCESS!
```
Quoted 1 line
Signed-off-by Nick Terrell <[email protected]>
Quoted 7 lines

I have a few concerns. First of all I can see that you include a lot of standard headers, e.g. stdlib.h, string.h, stddef.h, etc. Where do they come from? OS? If yes that is wrong. Additionally, I think that you should not use standard types, e.g. size_t, because this may not work if you cross compile. You have to use, e.g. grub_size_t. IMO the simplest solution would be definition and usage of relevant zstd_* types, e.g. zstd_size_t. Then e.g. zstd_size_t should be defined as grub_size_t.

I'm getting the standard headers from grub-core/lib/posix_wrap. I need to add stddef.h. I'd like to keep using standard types, which are typedefed to the grub_ equivalent in posix_wrap/sys/types.h.

I agree it would be nice if zstd put all of its dependencies into a configuration file that we could replace. I'll look into adding this in a future version. However, since we are using upstream zstd as-is right now, and posix_wrapper is already present, I think that the benefit of using upstream as-is outweighs the cost of using the posix_wrapper. Does that sound reasonable to you?

To test that I'm using the right headers, I looked that the .Po files generated, and saw that stddef.h was using the system, but the rest were using the posix_wrapper. Is that the right approach? Is there something else I should be testing?

Quoted 1 line

[...]

Nick

#4 lore raw

Re: [PATCH v4 1/2] Import upstream zstd-1.3.6

Quoted 57 lines · Nick Terrell
Quoted 39 lines · Daniel Kiper
Quoted 35 lines · Nick Terrell

Import zstd-1.3.6 from upstream [1]. Only the files need for decompression are imported. Additionally makes zstd a module by adding module.c which contains the license, and updates Makefile.core.def.

I used the latest zstd release, which includes patches [2] to build cleanly in GRUB.

Upstream zstd commit hash: 4fa456d7f12f8b27bd3b2f5dfd4f46898cb31c24 Upstream zstd commit name: Merge pull request #1354 from facebook/dev

I've included the script used to import zstd-1.3.6 below.

[1] https://github.com/facebook/zstd/releases/tag/v1.3.6
[2] https://github.com/facebook/zstd/pull/1344

```
#!/bin/sh -e

curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz
curl -L -O https://github.com/facebook/zstd/releases/download/v1.3.6/zstd-1.3.6.tar.gz.sha256
sha256sum --check zstd-1.3.6.tar.gz.sha256
tar xzf zstd-1.3.6.tar.gz

SRC_LIB="zstd-1.3.6/lib"
DST_LIB="grub-core/lib/zstd"
rm -rf $DST_LIB
mkdir -p $DST_LIB
cp $SRC_LIB/zstd.h $DST_LIB/
cp $SRC_LIB/common/*.[hc] $DST_LIB/
cp $SRC_LIB/decompress/*.[hc] $DST_LIB/
rm $DST_LIB/{pool.[hc],threading.[hc]}
rm -rf zstd-1.3.6*
echo SUCCESS!
```
Quoted 1 line
Signed-off-by Nick Terrell <[email protected]>
Quoted 7 lines

I have a few concerns. First of all I can see that you include a lot of standard headers, e.g. stdlib.h, string.h, stddef.h, etc. Where do they come from? OS? If yes that is wrong. Additionally, I think that you should not use standard types, e.g. size_t, because this may not work if you cross compile. You have to use, e.g. grub_size_t. IMO the simplest solution would be definition and usage of relevant zstd_* types, e.g. zstd_size_t. Then e.g. zstd_size_t should be defined as grub_size_t.

I'm getting the standard headers from grub-core/lib/posix_wrap. I need to add stddef.h. I'd like to keep using standard types, which are typedefed to the grub_ equivalent in posix_wrap/sys/types.h.

I agree it would be nice if zstd put all of its dependencies into a configuration file that we could replace. I'll look into adding this in a future version. However, since we are using upstream zstd as-is right now, and posix_wrapper is already present, I think that the benefit of using upstream as-is outweighs the cost of using the posix_wrapper. Does that sound reasonable to you?

Quoted 4 lines

To test that I'm using the right headers, I looked that the .Po files generated, and saw that stddef.h was using the system, but the rest were using the posix_wrapper. Is that the right approach? Is there something else I should be testing?

If you are able to assure me that you are not pulling OS stuff into GRUB2 binary then I am OK with that. However, I would like to see a few words about the issue in the commit message.

Daniel

author maintainer
#5 lore raw

PATCH v4 2/2 btrfs: Add zstd support to grub btrfs

In reply to Nick Terrell

Adds zstd support to the btrfs module.

Tested on Ubuntu-18.04 with a btrfs /boot partition with and without zstd compression. A test case was also added to the test suite that fails before the patch, and passes after.

Signed-off-by Nick Terrell <[email protected]>

---
v1 -> v2:
- Fix comments from Daniel Kiper.

v2 -> v3:
- Use grub_error() to set grub_errno in grub_btrfs_zstd_decompress().
- Fix style and formatting comments.

v3 -> v4:
- Use attribute unused.

 Makefile.util.def            |  10 +++-
 grub-core/Makefile.core.def  |   2 +-
 grub-core/fs/btrfs.c         | 108 ++++++++++++++++++++++++++++++++++-
 tests/btrfs_test.in          |   1 +
 tests/util/grub-fs-tester.in |   2 +-
 5 files changed, 119 insertions(+), 4 deletions(-)
Makefile.util.def +9 -1
index 9ae45f351..3ce9388ae 100644
@@ -54,7 +54,7 @@ library = {
 library = {
   name = libgrubmods.a;
   cflags = '-fno-builtin -Wno-undef';
-  cppflags = '-I$(top_srcdir)/grub-core/lib/minilzo -I$(srcdir)/grub-core/lib/xzembed -DMINILZO_HAVE_CONFIG_H';
+  cppflags = '-I$(srcdir)/grub-core/lib/minilzo -I$(srcdir)/grub-core/lib/xzembed -I$(srcdir)/grub-core/lib/zstd -DMINILZO_HAVE_CONFIG_H';
 
   common_nodist = grub_script.tab.c;
   common_nodist = grub_script.yy.c;

@@ -165,6 +165,14 @@ library = {
   common = grub-core/lib/xzembed/xz_dec_bcj.c;
   common = grub-core/lib/xzembed/xz_dec_lzma2.c;
   common = grub-core/lib/xzembed/xz_dec_stream.c;
+  common = grub-core/lib/zstd/debug.c;
+  common = grub-core/lib/zstd/entropy_common.c;
+  common = grub-core/lib/zstd/error_private.c;
+  common = grub-core/lib/zstd/fse_decompress.c;
+  common = grub-core/lib/zstd/huf_decompress.c;
+  common = grub-core/lib/zstd/xxhash.c;
+  common = grub-core/lib/zstd/zstd_common.c;
+  common = grub-core/lib/zstd/zstd_decompress.c;
 };
 
 program = {
grub-core/Makefile.core.def +1 -1
index 540a31f78..b09319328 100644
@@ -1280,7 +1280,7 @@ module = {
   common = fs/btrfs.c;
   common = lib/crc.c;
   cflags = '$(CFLAGS_POSIX) -Wno-undef';
-  cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/minilzo -DMINILZO_HAVE_CONFIG_H';
+  cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/minilzo -I$(srcdir)/lib/zstd -DMINILZO_HAVE_CONFIG_H';
 };
 
 module = {
grub-core/fs/btrfs.c +107 -1

156 lines not on this page · raw message

tests/btrfs_test.in +1 -0
index 2b37ddd33..0c9bf3a68 100644
@@ -18,6 +18,7 @@ fi
 "@builddir@/grub-fs-tester" btrfs
 "@builddir@/grub-fs-tester" btrfs_zlib
 "@builddir@/grub-fs-tester" btrfs_lzo
+"@builddir@/grub-fs-tester" btrfs_zstd
 "@builddir@/grub-fs-tester" btrfs_raid0
 "@builddir@/grub-fs-tester" btrfs_raid1
 "@builddir@/grub-fs-tester" btrfs_single
tests/util/grub-fs-tester.in +1 -1
index ef65fbc93..bc14a05ca 100644
@@ -629,7 +629,7 @@ for LOGSECSIZE in $(range "$MINLOGSECSIZE" "$MAXLOGSECSIZE" 1); do
 		    ;;
 		x"btrfs")
 		    "mkfs.btrfs" -s $SECSIZE -L "$FSLABEL" "${MOUNTDEVICE}" ;;
-		x"btrfs_zlib" | x"btrfs_lzo")
+		x"btrfs_zlib" | x"btrfs_lzo" | x"btrfs_zstd")
 		    "mkfs.btrfs" -s $SECSIZE -L "$FSLABEL" "${MOUNTDEVICE}"
 		    MOUNTOPTS="compress=${fs/btrfs_/},"
 		    MOUNTFS="btrfs"

--
2.17.1

#6 lore raw
Quoted 30 lines · Nick Terrell

Adds zstd support to the btrfs module.

Tested on Ubuntu-18.04 with a btrfs /boot partition with and without zstd compression. A test case was also added to the test suite that fails before the patch, and passes after.

Signed-off-by Nick Terrell <[email protected]>

---
v1 -> v2:
- Fix comments from Daniel Kiper.

v2 -> v3:
- Use grub_error() to set grub_errno in grub_btrfs_zstd_decompress().
- Fix style and formatting comments.

v3 -> v4:
- Use attribute unused.

 Makefile.util.def            |  10 +++-
 grub-core/Makefile.core.def  |   2 +-
 grub-core/fs/btrfs.c         | 108 ++++++++++++++++++++++++++++++++++-
 tests/btrfs_test.in          |   1 +
 tests/util/grub-fs-tester.in |   2 +-
 5 files changed, 119 insertions(+), 4 deletions(-)
Makefile.util.def +0 -0
index 9ae45f351..3ce9388ae 100644

Makefile.util.def library = { · line 57

Quoted 5 lines
Makefile.util.def +1 -1
@@ -54,7 +54,7 @@ library = {
 library = {
   name = libgrubmods.a;
   cflags = '-fno-builtin -Wno-undef';
-  cppflags = '-I$(top_srcdir)/grub-core/lib/minilzo -I$(srcdir)/grub-core/lib/xzembed -DMINILZO_HAVE_CONFIG_H';
+  cppflags = '-I$(srcdir)/grub-core/lib/minilzo -I$(srcdir)/grub-core/lib/xzembed -I$(srcdir)/grub-core/lib/zstd -DMINILZO_HAVE_CONFIG_H';

I am not strongly against s/top_srcdir/srcdir/ here. However, if you do such things I will ask you to add a blurb why to the commit message.

[...]

Quoted 9 lines
Makefile.util.def +9 -0
+static grub_ssize_t
+grub_btrfs_zstd_decompress (char *ibuf, grub_size_t isize, grub_off_t off,
+			    char *obuf, grub_size_t osize)
+{
+	void *allocated = NULL;
+	char *otmpbuf = obuf;
+	grub_size_t otmpsize = osize;
+	ZSTD_DCtx *dctx = NULL;
+	grub_size_t zstd_ret;

Makefile.util.def library = {

Quoted 10 lines
Makefile.util.def +10 -0
+	grub_ssize_t ret = -1;
+
+	/*
+	 * Zstd will fail if it can't fit the entire output in the destination
+	 * buffer, so if osize isn't large enough, allocate a temporary buffer.
+	 */
+	if (otmpsize < ZSTD_BTRFS_MAX_INPUT) {
+		allocated = grub_malloc (ZSTD_BTRFS_MAX_INPUT);
+		if (!allocated) {
+			goto err;

Hmmm... You fail silently. Should not you say something here?

Makefile.util.def

Quoted 1 line
Makefile.util.def +1 -0
+		}

If above is OK then drop this curly brackets.

Makefile.util.def

Quoted 8 lines
Makefile.util.def +8 -0
+		otmpbuf = (char*)allocated;
+		otmpsize = ZSTD_BTRFS_MAX_INPUT;
+	}
+
+	/* Create the ZSTD_DCtx. */
+	dctx = ZSTD_createDCtx_advanced (grub_zstd_allocator ());
+	if (!dctx) {
+		grub_error (GRUB_ERR_OUT_OF_MEMORY, "zstd out of memory");

...and here you complain. OK, but why GRUB_ERR_OUT_OF_MEMORY? This probably applies to grub_zstd_allocator() but maybe not to ZSTD_createDCtx_advanced(). AIUI both functions may return different errors.

Daniel