Skip to contentSkip to conversation

i386-pc: build btrfs zstd support into separate module

PATCH In review Michael Chang 2 participants 9 messages Aug 31, 2021 – Oct 27, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 i386-pc: build btrfs zstd support into separate module

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

Therefore if the system has enough space of embedding area for grub then zstd support for btrfs will be enabled automatically in the process of running grub-install through inserting btrfs_zstd module to the on-disk image, otherwise a warning will be logged on screen to indicate user that zstd support for btrfs is disabled due to the size limit.

---
 Makefile.util.def           |   1 +
 grub-core/Makefile.core.def |  11 ++++
 grub-core/fs/btrfs.c        | 114 +++++---------------------------
 grub-core/fs/btrfs_zstd.c   |  36 +++++++++++
 grub-core/lib/zstd.c        | 126 ++++++++++++++++++++++++++++++++++++
 include/grub/btrfs.h        |   6 ++
 include/grub/lib/zstd.h     |  26 ++++++++
 include/grub/util/install.h |   2 +
 util/grub-install.c         |  20 ++++++
 util/setup.c                |  43 ++++++++++++
 10 files changed, 288 insertions(+), 97 deletions(-)
 create mode 100644 grub-core/fs/btrfs_zstd.c
 create mode 100644 grub-core/lib/zstd.c
 create mode 100644 include/grub/lib/zstd.h
Makefile.util.def +1 -0
index 2f2881cb7..ac2b6aab1 100644
@@ -176,6 +176,7 @@ library = {
   common = grub-core/lib/zstd/xxhash.c;
   common = grub-core/lib/zstd/zstd_common.c;
   common = grub-core/lib/zstd/zstd_decompress.c;
+  common = grub-core/lib/zstd.c;
 };
 
 program = {
grub-core/Makefile.core.def +11 -0
index dd1777bd3..b5328d7a0 100644
@@ -1370,10 +1370,21 @@ module = {
   name = btrfs;
   common = fs/btrfs.c;
   common = lib/crc.c;
+  nopc = lib/zstd.c;
   cflags = '$(CFLAGS_POSIX) -Wno-undef';
+  i386_pc_cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/minilzo -DMINILZO_HAVE_CONFIG_H';
   cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/minilzo -I$(srcdir)/lib/zstd -DMINILZO_HAVE_CONFIG_H';
 };
 
+module = {
+  name = btrfs_zstd;
+  common = fs/btrfs_zstd.c;
+  common = lib/zstd.c;
+  cflags = '$(CFLAGS_POSIX) -Wno-undef';
+  cppflags = '-I$(srcdir)/lib/posix_wrap -I$(srcdir)/lib/zstd';
+  enable = i386_pc;
+};
+
 module = {
   name = archelp;
   common = fs/archelp.c;
grub-core/fs/btrfs.c +17 -97

144 lines not on this page · raw message

grub-core/fs/btrfs_zstd.c new file +36 -0
new file mode 100644
index 000000000..d5d1e013c
@@ -0,0 +1,36 @@
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2008  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <grub/types.h>
+#include <grub/dl.h>
+/* For NULL.  */
+#include <grub/mm.h>
+#include <grub/btrfs.h>
+#include <grub/lib/zstd.h>
+
+GRUB_MOD_LICENSE ("GPLv3+");
+
+GRUB_MOD_INIT (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = grub_zstd_decompress;
+}
+
+GRUB_MOD_FINI (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = NULL;
+}
grub-core/lib/zstd.c new file +126 -0

126 lines not on this page · raw message

include/grub/btrfs.h +6 -0
index 234ad9767..f8e551982 100644
@@ -69,4 +69,10 @@ struct grub_btrfs_inode_ref
   char name[0];
 };
 
+#ifdef GRUB_MACHINE_PCBIOS
+extern grub_ssize_t (*EXPORT_VAR (grub_btrfs_zstd_decompress_func)) (char *ibuf,
+		grub_size_t isize, grub_off_t off,
+		char *obuf, grub_size_t osize);
+#endif
+
 #endif
include/grub/lib/zstd.h new file +26 -0
new file mode 100644
index 000000000..0867b0c34
@@ -0,0 +1,26 @@
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2008  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef GRUB_ZSTD_HEADER
+#define GRUB_ZSTD_HEADER	1
+
+grub_ssize_t
+grub_zstd_decompress (char *ibuf, grub_size_t isize, grub_off_t off,
+		 char *obuf, grub_size_t osize);
+
+#endif /* ! GRUB_ZSTD_HEADER */
include/grub/util/install.h +2 -0
index faf9ff1ab..9e83e1339 100644
@@ -208,6 +208,8 @@ grub_util_sparc_setup (const char *dir,
 		       const char *dest, int force,
 		       int fs_probe, int allow_floppy,
 		       int add_rs_codes, int warn_short_mbr_gap);
+int
+grub_util_try_partmap_embed (const char *dest, unsigned int *nsec);
 
 char *
 grub_install_get_image_targets_string (void);
util/grub-install.c +20 -0
index 891e3ced8..a2286b3dd 100644
@@ -1482,6 +1482,26 @@ main (int argc, char *argv[])
 	}
     }
 
+  if (install_drive
+      && platform == GRUB_INSTALL_PLATFORM_I386_PC
+      && grub_strcmp (grub_fs->name, "btrfs") == 0)
+    {
+      char *mod;
+      mod = grub_util_path_concat (2, grub_install_source_directory, "btrfs_zstd.mod");
+      if (grub_util_is_regular (mod))
+	{
+          unsigned int nsec = GRUB_MIN_RECOMMENDED_MBR_GAP;
+	  int ret = grub_util_try_partmap_embed (install_drive, &nsec);
+	  if (ret == 0)
+	    grub_install_push_module ("btrfs_zstd");
+	  else if (ret == 1)
+	    grub_util_warn ("%s", _("btrfs zstd compression is disabled, please change install device to disk"));
+	  else
+	    grub_util_warn ("%s", _("btrfs zstd compression is disabled due to not enough space to embed image"));
+	}
+      grub_free (mod);
+    }
+
   if (!have_abstractions)
     {
       if ((disk_module && grub_strcmp (disk_module, "biosdisk") != 0)
util/setup.c +43 -0

49 lines not on this page · raw message

Signature · 1 line
2.31.1
#1 lore raw
Quoted 9 lines · Michael Chang via Grub-devel

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

I am OK with extracting zstd code from btrfs code. However, I want that be done for all architectures and platforms. No exceptions.

[...]

Quoted 26 lines
grub-core/fs/btrfs_zstd.c new file +20 -0
new file mode 100644
index 000000000..d5d1e013c
@@ -0,0 +1,36 @@
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2008  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <grub/types.h>
+#include <grub/dl.h>

grub-core/fs/btrfs_zstd.c line 30

Quoted 10 lines
grub-core/fs/btrfs_zstd.c new file +10 -0
+/* For NULL.  */
+#include <grub/mm.h>
+#include <grub/btrfs.h>
+#include <grub/lib/zstd.h>
+
+GRUB_MOD_LICENSE ("GPLv3+");
+
+GRUB_MOD_INIT (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = grub_zstd_decompress;

Please create a function in btrfs module which will do this for you.

grub-core/fs/btrfs_zstd.c line 35

Quoted 5 lines
grub-core/fs/btrfs_zstd.c new file +5 -0
+}
+
+GRUB_MOD_FINI (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = NULL;

Ditto.

Last but not least, please CC original author of zstd support next time.

Daniel

#2 lore raw
Quoted 5 lines · Daniel Kiper
Quoted 4 lines · Michael Chang via Grub-devel

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

Quoted 8 lines
Quoted 5 lines

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

I am OK with extracting zstd code from btrfs code. However, I want that be done for all architectures and platforms. No exceptions.

May I ask for more background about this decision ?

Given that btrfs compression is per file property and can be recompressed on the fly, there is no way to detect it beforehand thus the safest bet is to assume that it is always needed. In other words if the platform has no known limitation on the size of image, the zstd code should be indivisible to btrfs.mo.

Quoted 33 lines

[...]

Quoted 30 lines
grub-core/fs/btrfs_zstd.c new file +24 -0
new file mode 100644
index 000000000..d5d1e013c
@@ -0,0 +1,36 @@
+/*
+ *  GRUB  --  GRand Unified Bootloader
+ *  Copyright (C) 2008  Free Software Foundation, Inc.
+ *
+ *  GRUB is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  GRUB is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <grub/types.h>
+#include <grub/dl.h>
+/* For NULL.  */
+#include <grub/mm.h>
+#include <grub/btrfs.h>
+#include <grub/lib/zstd.h>
Quoted 8 lines

grub-core/fs/btrfs_zstd.c line 30

Quoted 6 lines
grub-core/fs/btrfs_zstd.c new file +6 -0
+
+GRUB_MOD_LICENSE ("GPLv3+");
+
+GRUB_MOD_INIT (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = grub_zstd_decompress;

Please create a function in btrfs module which will do this for you.

Sure. np.

Quoted 8 lines

grub-core/fs/btrfs_zstd.c line 35

Quoted 6 lines
grub-core/fs/btrfs_zstd.c new file +5 -0
+}
+
+GRUB_MOD_FINI (btrfs_zstd)
+{
+  grub_btrfs_zstd_decompress_func = NULL;

Ditto.

OK.

Quoted 2 lines

Last but not least, please CC original author of zstd support next time.

OK. I will CC Nick Terrell next time.

Thank you very much for the review.

Regards,
Michael

Quoted 7 lines

Daniel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#3 lore raw
Quoted 12 lines · Michael Chang via Grub-devel
Quoted 11 lines · Daniel Kiper
Quoted 9 lines · Michael Chang via Grub-devel

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

Quoted 4 lines
Quoted 2 lines

I am OK with extracting zstd code from btrfs code. However, I want that be done for all architectures and platforms. No exceptions.

May I ask for more background about this decision ?

Yes, I want the same code and behavior for all architectures and platforms if possible.

Quoted 5 lines

Given that btrfs compression is per file property and can be recompressed on the fly, there is no way to detect it beforehand thus the safest bet is to assume that it is always needed. In other words if the platform has no known limitation on the size of image, the zstd code should be indivisible to btrfs.mo.

Wait, you said this in the commit message too:

Therefore if the system has enough space of embedding area for grub then zstd support for btrfs will be enabled automatically in the process of running grub-install through inserting btrfs_zstd module to the on-disk image, otherwise a warning will be logged on screen to indicate user that zstd support for btrfs is disabled due to the size limit.

So, I thought this may work in the same way in all cases. If this is not true you have to fix this. Otherwise I cannot take this patch.

TBH, I should reject this patch immediately because this is "small MBR gap stuff". And as I said a few months ago, I want to stop to pay attention to "small MBR gap stuff" in upstream. Sorry for being blunt but it is full can of worms, i.e., each patch like that one adds more cruft which we have to take care because a few folks in the wild could not spend a few hours to repartition their systems. I think distros should start putting more effort in educating their users WRT to small MBR gap and problems related to it instead of pushing again and again upstream patches which make the GRUB code more complicated.

Daniel

#4 lore raw
Quoted 29 lines · Daniel Kiper
Quoted 12 lines · Michael Chang via Grub-devel
Quoted 11 lines · Daniel Kiper
Quoted 9 lines · Michael Chang via Grub-devel

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

Quoted 4 lines
Quoted 2 lines

I am OK with extracting zstd code from btrfs code. However, I want that be done for all architectures and platforms. No exceptions.

May I ask for more background about this decision ?

Yes, I want the same code and behavior for all architectures and platforms if possible.

Quoted 5 lines

Given that btrfs compression is per file property and can be recompressed on the fly, there is no way to detect it beforehand thus the safest bet is to assume that it is always needed. In other words if the platform has no known limitation on the size of image, the zstd code should be indivisible to btrfs.mo.

Wait, you said this in the commit message too:

Quoted 8 lines

Therefore if the system has enough space of embedding area for grub then zstd support for btrfs will be enabled automatically in the process of running grub-install through inserting btrfs_zstd module to the on-disk image, otherwise a warning will be logged on screen to indicate user that zstd support for btrfs is disabled due to the size limit.

So, I thought this may work in the same way in all cases. If this is not true you have to fix this. Otherwise I cannot take this patch.

I literally don't like to propagate this (somewhat awkward) fix to other architectures that do not have sensible requirement to it, though it looks possible to do so.

If this doesn't meet the expection then I'm sorry.

Quoted 10 lines

TBH, I should reject this patch immediately because this is "small MBR gap stuff". And as I said a few months ago, I want to stop to pay attention to "small MBR gap stuff" in upstream. Sorry for being blunt but it is full can of worms, i.e., each patch like that one adds more cruft which we have to take care because a few folks in the wild could not spend a few hours to repartition their systems. I think distros should start putting more effort in educating their users WRT to small MBR gap and problems related to it instead of pushing again and again upstream patches which make the GRUB code more complicated.

Well, I think the short mbr gap issue is all well received been discussed several times. The reason I'm poking this beehive again is just that I can't resist to fix problem from our users who opted to use "btrfs partition" which differs to "short mbr gap" with a lot more user base and sensible usecases.

FWIW we want to address this problem, because mbr gap is adjustable via re-partitioning but btrfs bootloader area is not.

Thanks,
Michael

Quoted 7 lines

Daniel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#5 lore raw
Quoted 55 lines · Michael Chang via Grub-devel
Quoted 29 lines · Daniel Kiper
Quoted 12 lines · Michael Chang via Grub-devel
Quoted 11 lines · Daniel Kiper
Quoted 9 lines · Michael Chang via Grub-devel

The zstd support in btrfs brings significant size increment to the on-disk image that it can no longer fit into btrfs bootloader area and short mbr gap.

In order to support grub update on outstanding i386-pc setup with these size constraints remain in place, here we build the zstd suppprt of btrfs into a separate module, named btrfs_zstd, to alleviate the size change. Please note this only makes it's way to i386-pc, other architecture is not affected.

Quoted 4 lines
Quoted 2 lines

I am OK with extracting zstd code from btrfs code. However, I want that be done for all architectures and platforms. No exceptions.

May I ask for more background about this decision ?

Yes, I want the same code and behavior for all architectures and platforms if possible.

Quoted 5 lines

Given that btrfs compression is per file property and can be recompressed on the fly, there is no way to detect it beforehand thus the safest bet is to assume that it is always needed. In other words if the platform has no known limitation on the size of image, the zstd code should be indivisible to btrfs.mo.

Wait, you said this in the commit message too:

Quoted 8 lines

Therefore if the system has enough space of embedding area for grub then zstd support for btrfs will be enabled automatically in the process of running grub-install through inserting btrfs_zstd module to the on-disk image, otherwise a warning will be logged on screen to indicate user that zstd support for btrfs is disabled due to the size limit.

So, I thought this may work in the same way in all cases. If this is not true you have to fix this. Otherwise I cannot take this patch.

I literally don't like to propagate this (somewhat awkward) fix to other architectures that do not have sensible requirement to it, though it looks possible to do so.

If this doesn't meet the expection then I'm sorry.

Quoted 9 lines

TBH, I should reject this patch immediately because this is "small MBR gap stuff". And as I said a few months ago, I want to stop to pay attention to "small MBR gap stuff" in upstream. Sorry for being blunt but it is full can of worms, i.e., each patch like that one adds more cruft which we have to take care because a few folks in the wild could not spend a few hours to repartition their systems. I think distros should start putting more effort in educating their users WRT to small MBR gap and problems related to it instead of pushing again and again upstream patches which make the GRUB code more complicated.

Quoted 2 lines

Well, I think the short mbr gap issue is all well received been discussed several times. The reason I'm poking this beehive again is

Cool! Thanks!

Quoted 6 lines

just that I can't resist to fix problem from our users who opted to use "btrfs partition" which differs to "short mbr gap" with a lot more user base and sensible usecases.

FWIW we want to address this problem, because mbr gap is adjustable via re-partitioning but btrfs bootloader area is not.

Huh! The "btrfs partition" sounds like not resizable MBR gap! I am not happy with it just at the beginning. Anyway, could you explain your use case in more details including example commands and why core.img seems landing in the "btrfs partition". I am especially curious about the latter because I think the "btrfs partition" and things like that was designed for something else, e.g., storing grubenv data. And why this solution is i386 specific?

Daniel

#6 lore raw
Quoted 5 lines · Daniel Kiper
Quoted 4 lines · Michael Chang via Grub-devel
Quoted 3 lines · Daniel Kiper
Quoted 2 lines · Michael Chang via Grub-devel
Quoted 1 line · Daniel Kiper

On Tue, Aug 31, 2021 at 03:12:28PM +0800, Michael Chang via Grub-devel wrote:

[snip]

Quoted 7 lines
Quoted 6 lines

just that I can't resist to fix problem from our users who opted to use "btrfs partition" which differs to "short mbr gap" with a lot more user base and sensible usecases.

FWIW we want to address this problem, because mbr gap is adjustable via re-partitioning but btrfs bootloader area is not.

Quoted 7 lines

Huh! The "btrfs partition" sounds like not resizable MBR gap! I am not happy with it just at the beginning. Anyway, could you explain your use case in more details including example commands and why core.img seems landing in the "btrfs partition". I am especially curious about the latter because I think the "btrfs partition" and things like that was designed for something else, e.g., storing grubenv data. And why this solution is i386 specific?

Installing to btrfs partition is not something exotic to grub, it is actually a supported usecase by grub for a long time. Also zfs has similar design, see:

c7ba4f698 Support BtrFS embedding.
ba102053c Support ZFS embedding.

To make it happen, you just have to point the btrfs device to grub-install and it will setup and embed image there. That worked quite nicely until the zstd support came into play.

On current git head, the command fails with core.img is too large.

# ./grub-install -d ./grub-core /dev/vda2
  Installing for i386-pc platform.
  ./grub-install: warning: your core.img is unusually large.  It won't fit in the embedding area.
  ./grub-install: error: filesystem `btrfs' doesn't support blocklists.

With this patch, the embedding still works but warns the user zstd is disabled, also instructing them to use MBR if they need the zstd feature to work.

# ./grub-install -d ./grub-core /dev/vda2
Installing for i386-pc platform.
./grub-install: warning: btrfs zstd compression is disabled, please change install device to disk.
Installation finished. No error reported.

It is very i386-pc centric, as it is often used by legacy mbr boot code with which active partition is chainloaded. Some people still thinks this is the right thing to do and feels more comfortable this way to manage their multiple distrubution booting, compared with os-prober.

Thanks,
Michael

Quoted 7 lines

Daniel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#7 lore raw
Quoted 51 lines · Michael Chang via Grub-devel
Quoted 5 lines · Daniel Kiper
Quoted 4 lines · Michael Chang via Grub-devel
Quoted 3 lines · Daniel Kiper
Quoted 2 lines · Michael Chang via Grub-devel
Quoted 1 line · Daniel Kiper

On Tue, Aug 31, 2021 at 03:12:28PM +0800, Michael Chang via Grub-devel wrote:

[snip]

Quoted 7 lines
Quoted 6 lines

just that I can't resist to fix problem from our users who opted to use "btrfs partition" which differs to "short mbr gap" with a lot more user base and sensible usecases.

FWIW we want to address this problem, because mbr gap is adjustable via re-partitioning but btrfs bootloader area is not.

Quoted 7 lines

Huh! The "btrfs partition" sounds like not resizable MBR gap! I am not happy with it just at the beginning. Anyway, could you explain your use case in more details including example commands and why core.img seems landing in the "btrfs partition". I am especially curious about the latter because I think the "btrfs partition" and things like that was designed for something else, e.g., storing grubenv data. And why this solution is i386 specific?

Installing to btrfs partition is not something exotic to grub, it is actually a supported usecase by grub for a long time. Also zfs has similar design, see:

c7ba4f698 Support BtrFS embedding.
ba102053c Support ZFS embedding.

To make it happen, you just have to point the btrfs device to grub-install and it will setup and embed image there. That worked quite nicely until the zstd support came into play.

On current git head, the command fails with core.img is too large.

# ./grub-install -d ./grub-core /dev/vda2
  Installing for i386-pc platform.
  ./grub-install: warning: your core.img is unusually large.  It won't fit in the embedding area.
  ./grub-install: error: filesystem `btrfs' doesn't support blocklists.

With this patch, the embedding still works but warns the user zstd is disabled, also instructing them to use MBR if they need the zstd feature to work.

# ./grub-install -d ./grub-core /dev/vda2
Installing for i386-pc platform.
./grub-install: warning: btrfs zstd compression is disabled, please change install device to disk.
Installation finished. No error reported.
Quoted 4 lines

It is very i386-pc centric, as it is often used by legacy mbr boot code with which active partition is chainloaded. Some people still thinks this is the right thing to do and feels more comfortable this way to manage their multiple distrubution booting, compared with os-prober.

I do not like this solution because it is similar to all the patches trying to solve small MBR problem. Though, after chatting with Btrfs folks in Oracle, it seems to me there is better solution for this issue. I think we should make the GRUB installer smarter. If it cannot put the core image in first 64 KiB before primary superblock it should install the core image behind the primary superblock, i.e. starting from 64 KiB + 4 KiB but below 1 MiB. Please take a look at [1] which nicely explains the Btrfs bootloader support. Does it make sense for you?

Daniel

[1] https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)#BOOTLOADER_SUPPORT

#8 lore raw
Quoted 57 lines · Daniel Kiper
Quoted 51 lines · Michael Chang via Grub-devel
Quoted 5 lines · Daniel Kiper
Quoted 4 lines · Michael Chang via Grub-devel
Quoted 3 lines · Daniel Kiper
Quoted 2 lines · Michael Chang via Grub-devel
Quoted 1 line · Daniel Kiper

On Tue, Aug 31, 2021 at 03:12:28PM +0800, Michael Chang via Grub-devel wrote:

[snip]

Quoted 7 lines
Quoted 6 lines

just that I can't resist to fix problem from our users who opted to use "btrfs partition" which differs to "short mbr gap" with a lot more user base and sensible usecases.

FWIW we want to address this problem, because mbr gap is adjustable via re-partitioning but btrfs bootloader area is not.

Quoted 7 lines

Huh! The "btrfs partition" sounds like not resizable MBR gap! I am not happy with it just at the beginning. Anyway, could you explain your use case in more details including example commands and why core.img seems landing in the "btrfs partition". I am especially curious about the latter because I think the "btrfs partition" and things like that was designed for something else, e.g., storing grubenv data. And why this solution is i386 specific?

Installing to btrfs partition is not something exotic to grub, it is actually a supported usecase by grub for a long time. Also zfs has similar design, see:

c7ba4f698 Support BtrFS embedding.
ba102053c Support ZFS embedding.

To make it happen, you just have to point the btrfs device to grub-install and it will setup and embed image there. That worked quite nicely until the zstd support came into play.

On current git head, the command fails with core.img is too large.

# ./grub-install -d ./grub-core /dev/vda2
  Installing for i386-pc platform.
  ./grub-install: warning: your core.img is unusually large.  It won't fit in the embedding area.
  ./grub-install: error: filesystem `btrfs' doesn't support blocklists.

With this patch, the embedding still works but warns the user zstd is disabled, also instructing them to use MBR if they need the zstd feature to work.

# ./grub-install -d ./grub-core /dev/vda2
Installing for i386-pc platform.
./grub-install: warning: btrfs zstd compression is disabled, please change install device to disk.
Installation finished. No error reported.
Quoted 4 lines

It is very i386-pc centric, as it is often used by legacy mbr boot code with which active partition is chainloaded. Some people still thinks this is the right thing to do and feels more comfortable this way to manage their multiple distrubution booting, compared with os-prober.

Quoted 8 lines

I do not like this solution because it is similar to all the patches trying to solve small MBR problem. Though, after chatting with Btrfs folks in Oracle, it seems to me there is better solution for this issue. I think we should make the GRUB installer smarter. If it cannot put the core image in first 64 KiB before primary superblock it should install the core image behind the primary superblock, i.e. starting from 64 KiB + 4 KiB but below 1 MiB. Please take a look at [1] which nicely explains the Btrfs bootloader support. Does it make sense for you?

Yes. The idea is brilliant and would be the sanest solution out there. I'll try to come up with a patch and discuss here.

Quoted 4 lines

Daniel

[1] https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs(5)#BOOTLOADER_SUPPORT

Thanks for pointing this out. I am looking for it for a long time as I used to come across this paragraph but not being able to make it again.

Thanks,
Michael

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel