Skip to contentSkip to conversation

btrfs: disable zstd support for i386-pc

PATCH In review Michael Chang 8 participants 24 messages Nov 5, 2019 – Jun 23, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs: disable zstd support for i386-pc

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +18 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor in a
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to zstd.
  */
 #define ZSTD_STATIC_LINKING_ONLY
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
 				     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64 + 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode), 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)), size_t size)
 {
   return grub_malloc (size);
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
 	{
 	  grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
 		      "compression type 0x%x not supported",
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#ifndef GRUB_MACHINE_PCBIOS
 	  else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
 	    {
 	      if (grub_btrfs_zstd_decompress (data->extent->inl, data->extsize -
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#endif
 	  else
 	    grub_memcpy (buf, data->extent->inl + extoff, csize);
 	  break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
 	      else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
 		ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#endif
 	      else
 		ret = -1;
 
Signature · 1 line
2.16.4
#1 lore raw

Dear Michael,

Thank you for your patch.

Quoted 1 line · Michael Chang

The zstd support in btrfs has dependenciy to zstd module and core.img

dependency

Quoted 8 lines

grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. /usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to

disables (present tense)

Maybe:

So, disable the zstd support of btrfs for the PC BIOS platform to …

Quoted 2 lines

avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix

is *a* simple

Quoted 1 line

landed upstream.

lands

Is there a feature request/bug report for this? Maybe reference it heare?

Quoted 73 lines
---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +10 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor in a
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to zstd.
  */
 #define ZSTD_STATIC_LINKING_ONLY
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
 				     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64 + 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode), 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)), size_t size)
 {
   return grub_malloc (size);
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data *data, · line 1512

Quoted 9 lines
grub-core/fs/btrfs.c +4 -0
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif

Why not just guard the ZSTD line by switching the order of LZO and ZSTD for the closing bracket?

Quoted 21 lines
grub-core/fs/btrfs.c +2 -0
 	{
 	  grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
 		      "compression type 0x%x not supported",
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#ifndef GRUB_MACHINE_PCBIOS
 	  else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
 	    {
 	      if (grub_btrfs_zstd_decompress (data->extent->inl, data->extsize -
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#endif
 	  else
 	    grub_memcpy (buf, data->extent->inl + extoff, csize);
 	  break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data *data, · line 1607

Quoted 10 lines
grub-core/fs/btrfs.c +2 -0
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
 	      else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
 		ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#endif
 	      else
 		ret = -1;

Does some documentation need to be updated?

Kind regards,

Paul

#2 lore raw
Quoted 9 lines · Paul Menzel

Dear Michael,

Thank you for your patch.

Quoted 1 line · Michael Chang

The zstd support in btrfs has dependenciy to zstd module and core.img

dependency

OK. The typo will be corrected next version.

Quoted 11 lines
Quoted 9 lines

grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area. /usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to

disables (present tense)

OK. It will be corrected next version.

Quoted 9 lines

Maybe:

So, disable the zstd support of btrfs for the PC BIOS platform to …

Quoted 2 lines

avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix

is *a* simple

OK. Next version will correct this.

Quoted 4 lines
Quoted 2 lines

landed upstream.

lands

OK.

Quoted 2 lines

Is there a feature request/bug report for this? Maybe reference it heare?

Yes, it is reported on openSUSE bugzilla.

http://bugzilla.opensuse.org/show_bug.cgi?id=1154809

I will add the bz reference next version.

Quoted 75 lines
Quoted 75 lines
---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +10 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor in a
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to zstd.
  */
 #define ZSTD_STATIC_LINKING_ONLY
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
 				     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64 + 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode), 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)), size_t size)
 {
   return grub_malloc (size);
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 
Quoted 11 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data *data, · line 1512

Quoted 8 lines
grub-core/fs/btrfs.c +4 -0
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
 	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+	  && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif

Why not just guard the ZSTD line by switching the order of LZO and ZSTD for the closing bracket?

Yes. Indeed it is more readable and also makes my vim editor's syntax highlighting happy. I will do it in next version.

Quoted 26 lines
Quoted 26 lines
grub-core/fs/btrfs.c +3 -0
 	{
 	  grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
 		      "compression type 0x%x not supported",
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#ifndef GRUB_MACHINE_PCBIOS
 	  else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
 	    {
 	      if (grub_btrfs_zstd_decompress (data->extent->inl, data->extsize -
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		  != (grub_ssize_t) csize)
 		return -1;
 	    }
+#endif
 	  else
 	    grub_memcpy (buf, data->extent->inl + extoff, csize);
 	  break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
 		ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
 	      else if (data->extent->compression == GRUB_BTRFS_COMPRESSION_ZSTD)
Quoted 8 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data *data, · line 1607

Quoted 6 lines
grub-core/fs/btrfs.c +1 -0
 		ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
 				    + grub_le_to_cpu64 (data->extent->offset),
 				    buf, csize);
+#endif
 	      else
 		ret = -1;

Does some documentation need to be updated?

I didn't know which document is appropriate for this, if there's any I'd like to add. Please let me know if you have any idea about this.

Thanks for your review.

Regards,
Michael

Quoted 6 lines

Kind regards,

Paul

#3 lore raw
In reply to Michael Chang
Quoted 10 lines · Michael Chang

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

Quoted 4 lines

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

So combination zstd+btrfs+i386-pc never worked? Removing support for zstd could lead to unbootable system, but if that has never worked before it'd be ok to make the build conditional.

Looking at zstd code, does not seem to be easy to squeeze the asm to something like 64-56=8K.

#4 lore raw
Quoted 19 lines · David Sterba
Quoted 10 lines · Michael Chang

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

Quoted 4 lines

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

Quoted 6 lines

So combination zstd+btrfs+i386-pc never worked? Removing support for zstd could lead to unbootable system, but if that has never worked before it'd be ok to make the build conditional.

Looking at zstd code, does not seem to be easy to squeeze the asm to something like 64-56=8K.

Yeah, upstream zstd won’t get down to 8K. We recently did some work to make the decompressor smaller for mobile reducing inlining and excluding variants of some functions. On x86_64 the upstream library went from 84KB -> 63KB uncompressed, and 31KB -> 25KB compressed. If we update GRUB to the latest zstd and define a few macros we can get that gain. But it won’t be 8KB.

A custom zstd decompress optimized for size could be made much smaller But that would be a large maintenance burden. For example, we have an “educational decoder” that is 18 KB uncompressed and 8 KB compressed, but it can’t be used in production, since it isn’t hardened against invalid inputs. But, it shows the zstd format can be decompressed with a small library. If we (upstream) add a smaller zstd decompressor, we will put a patch up to use it in GRUB.

-Nick

#5 lore raw
In reply to David Sterba
Quoted 11 lines · David Sterba
Quoted 10 lines · Michael Chang

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

Quoted 6 lines
Quoted 4 lines

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

So combination zstd+btrfs+i386-pc never worked?

It works for mbr gap, however we used to encounter some system shipped by Microsoft Windows still having its mbr gap aligned to cylinder boundary, which is usually 63 sectors. Therefore we could only fallback to btrfs bootloader area if mbr gap is found to be too small and now that the solution is no longer appropriate.

Quoted 3 lines

Removing support for
zstd could lead to unbootable system, but if that has never worked before it'd be ok to make the build conditional.

So far didn't have the request to enable btrfs zstd compression for the root filesystem. But with new grub supporting it, the request might come up in the future and hope we could have solution for i386-pc at that time.

Quoted 3 lines

Looking at zstd code, does not seem to be easy to squeeze the asm to something like 64-56=8K.

Another solution from Vladmir is making the zstd support a separate module and loading on demand. I personally think it is better than reducing the size as the same situation could still come up in the future.

Thanks,
Michael

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#6 lore raw
In reply to Michael Chang

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 150 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +1 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY

grub-core/fs/btrfs.c +9 -0
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode),
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +5 -0
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +1 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +3 -1
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#7 lore raw
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

Quoted 3 lines

Not everybody uses btrfs
embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

The problem here is existing installation would fail to update to new grub version if it is using btrfs bootloader area. For new installation certainly we could recommend them to do so. or btrfs zstd support will be disabled if using a btrfs bootloader area.

Thanks,
Michael

Quoted 158 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 146 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +1 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY

grub-core/fs/btrfs.c +9 -0
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode),
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +5 -0
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +1 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +3 -1
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#8 lore raw
Quoted 10 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

We don't do a release right now, so I don't think it's justified. Also increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Quoted 190 lines
Quoted 4 lines

Not everybody uses btrfs
embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

The problem here is existing installation would fail to update to new grub version if it is using btrfs bootloader area. For new installation certainly we could recommend them to do so. or btrfs zstd support will be disabled if using a btrfs bootloader area.

Thanks,
Michael

Quoted 8 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 6 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

Quoted 7 lines
Quoted 7 lines

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too

close

Quoted 16 lines
Quoted 16 lines

to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +0 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
Quoted 3 lines

grub-core/fs/btrfs.c

Quoted 3 lines
grub-core/fs/btrfs.c +1 -0
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API,

which

Quoted 1 line

grub-core/fs/btrfs.c line 23

Quoted 1 line
grub-core/fs/btrfs.c +0 -0
  * aren't safe to use when linking against a dynamic library. We

vendor

Quoted 3 lines

grub-core/fs/btrfs.c line 25

Quoted 3 lines

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(),

to

Quoted 5 lines
Quoted 5 lines

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY
+#endif

Quoted 15 lines
Quoted 13 lines

#include <grub/err.h>
#include <grub/file.h>

grub-core/fs/btrfs.c +2 -0
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>

grub-core/fs/btrfs.c GRUB_MOD_LICENSE ("GPLv3+"); · line 61

Quoted 2 lines
grub-core/fs/btrfs.c +0 -0
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16)

+ 64

Quoted 17 lines
Quoted 17 lines
grub-core/fs/btrfs.c +5 -0
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
Quoted 3 lines

grub-core/fs/btrfs.c grub_btrfs_read_inode (struct grub_btrfs_data

Quoted 3 lines
grub-core/fs/btrfs.c +0 -0
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data

*data,

Quoted 1 line

grub-core/fs/btrfs.c grub_btrfs_read_inode (struct grub_btrfs_data · line 1237

Quoted 1 line
grub-core/fs/btrfs.c +0 -0
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof

(*inode),

Quoted 10 lines
Quoted 10 lines
grub-core/fs/btrfs.c +1 -0
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +0 -0
@@ -1318,6 +1327,7 @@ err:
 
Quoted 6 lines

grub-core/fs/btrfs.c err:

Quoted 6 lines
grub-core/fs/btrfs.c +1 -0
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t

off,

Quoted 15 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data

Quoted 15 lines
grub-core/fs/btrfs.c +4 -0
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +0 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data

*data,

Quoted 10 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data

Quoted 10 lines
grub-core/fs/btrfs.c +1 -0
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +0 -0
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data

*data,

Quoted 28 lines
Quoted 28 lines
grub-core/fs/btrfs.c +3 -1
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

Quoted 10 lines
Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#9 lore raw
Quoted 7 lines · Vladimir 'phcoder' Serbinenko
Quoted 5 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

Quoted 6 lines
Quoted 5 lines

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

We don't do a release right now, so I don't think it's justified.

Alright, then I'll keep it as downstream patch as it was found while we were doing a new release to latest stable grub.

Quoted 4 lines

Also
increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Here I'd like to share my thoughts to the unbootable issue.

The increased size will casue the new boot image failure to install thus old/obseleted one still remains in place. We should detect such failure and abort the grub-install process earlier before new grub modules copied to preboot directory and avoiding the symbol lookup error as the consequence of mixture of old and new grub versions.

Thanks,
Michael

Quoted 196 lines
Quoted 186 lines
Quoted 5 lines

Not everybody uses btrfs
embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

The problem here is existing installation would fail to update to new grub version if it is using btrfs bootloader area. For new installation certainly we could recommend them to do so. or btrfs zstd support will be disabled if using a btrfs bootloader area.

Thanks,
Michael

Quoted 8 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 6 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

Quoted 7 lines
Quoted 7 lines

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too

close

Quoted 16 lines
Quoted 16 lines

to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +0 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
Quoted 3 lines

grub-core/fs/btrfs.c

Quoted 3 lines
grub-core/fs/btrfs.c +1 -0
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API,

which

Quoted 1 line

grub-core/fs/btrfs.c line 23

Quoted 1 line
grub-core/fs/btrfs.c +0 -0
  * aren't safe to use when linking against a dynamic library. We

vendor

Quoted 3 lines

grub-core/fs/btrfs.c line 25

Quoted 3 lines

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(),

to

Quoted 5 lines
Quoted 5 lines

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY
+#endif

Quoted 15 lines
Quoted 13 lines

#include <grub/err.h>
#include <grub/file.h>

grub-core/fs/btrfs.c +2 -0
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>

grub-core/fs/btrfs.c GRUB_MOD_LICENSE ("GPLv3+"); · line 61

Quoted 2 lines
grub-core/fs/btrfs.c +0 -0
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16)

+ 64

Quoted 17 lines
Quoted 17 lines
grub-core/fs/btrfs.c +5 -0
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
Quoted 3 lines

grub-core/fs/btrfs.c grub_btrfs_read_inode (struct grub_btrfs_data

Quoted 3 lines
grub-core/fs/btrfs.c +0 -0
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data

*data,

Quoted 1 line

grub-core/fs/btrfs.c grub_btrfs_read_inode (struct grub_btrfs_data · line 1237

Quoted 1 line
grub-core/fs/btrfs.c +0 -0
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof

(*inode),

Quoted 10 lines
Quoted 10 lines
grub-core/fs/btrfs.c +1 -0
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +0 -0
@@ -1318,6 +1327,7 @@ err:
 
Quoted 6 lines

grub-core/fs/btrfs.c err:

Quoted 6 lines
grub-core/fs/btrfs.c +1 -0
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t

off,

Quoted 15 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data

Quoted 15 lines
grub-core/fs/btrfs.c +4 -0
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +0 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data

*data,

Quoted 10 lines

grub-core/fs/btrfs.c grub_btrfs_extent_read (struct grub_btrfs_data

Quoted 10 lines
grub-core/fs/btrfs.c +1 -0
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +0 -0
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data

*data,

Quoted 28 lines
Quoted 28 lines
grub-core/fs/btrfs.c +3 -1
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

Quoted 10 lines
Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#10 lore raw
In reply to Vladimir 'phcoder' Serbinenko
Quoted 12 lines · Vladimir 'phcoder' Serbinenko
Quoted 10 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

Quoted 4 lines

We don't do a release right now, so I don't think it's justified. Also increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Given grub 2.06 is on the horizon, is it time to re-evaluate this and disable something known to be badly broken?

Failing that, is anyone working on making it a separate module? :)

Signature · 2 lines
Eli Schwartz
Arch Linux Bug Wrangler and Trusted User
#11 lore raw
Quoted 14 lines · Eli Schwartz
Quoted 12 lines · Vladimir 'phcoder' Serbinenko
Quoted 10 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

Quoted 7 lines
Quoted 4 lines

We don't do a release right now, so I don't think it's justified. Also increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Given grub 2.06 is on the horizon, is it time to re-evaluate this and disable something known to be badly broken?

If I have read the thread correctly, this only breaks if you try to install grub in an undersized embedding area. This would really only affect new grub installs; existing ones can keep using the currently installed code.

I think it would be better to hold out for a real fix than to disable a feature for an entire platform where only a subset of configurations would not work.

#12 lore raw
Quoted 20 lines · Mike Gilbert
Quoted 19 lines · Eli Schwartz
Quoted 16 lines · Vladimir 'phcoder' Serbinenko
Quoted 10 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

We don't do a release right now, so I don't think it's justified. Also increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Quoted 7 lines
Quoted 2 lines

Given grub 2.06 is on the horizon, is it time to re-evaluate this and disable something known to be badly broken?

If I have read the thread correctly, this only breaks if you try to install grub in an undersized embedding area. This would really only affect new grub installs; existing ones can keep using the currently installed code.

"changing grub only affects people who newly run the grub-install command" therefore it's not worth considering whether to make sure grub-install completes without errors? I don't comprehend this logic. Who ever suggested we care about people who aren't the target audience of new grub versions to begin with?

But anyway this isn't true. There are valid reasons to reinstall grub on old systems, in which case you are most likely not benefiting from zstd support one way or another, but in this case, rerunning grub-install destroys the working bootloader code and fails to replace.

https://bugs.archlinux.org/task/63656 our infrastructure apparently somehow ended up mismatching core.img and /boot modules, and had to rerun grub-install (it's not clear what happened exactly, possibly grub-install accidentally got rerun in a broken manner?) and then this remote server would not boot, and could not be fixed without hunting for the right old version of grub via the rescue system, reinstalling it, and using it to restore the boring (working) old pre-zstd bootloader code, after a lot of painful debugging.

Quoted 3 lines

I think it would be better to hold out for a real fix than to disable a feature for an entire platform where only a subset of configurations would not work.

I'm astonished that "it's better to hold out for a real fix than to ensure users' systems actually work".

Why is the benefit of enabling zstd support in situations where it very often breaks everything, worth more than the disadvantage of, in fact, breaking everything? The feature was broken out of the gate, no proper fix seems imminent, let's bandage the wound before more people get hurt.

Better to not have a hot new feature than to have it, but be unable to boot.

Signature · 2 lines
Eli Schwartz
Bug Wrangler and Trusted User
#13 lore raw
Quoted 54 lines · Eli Schwartz
Quoted 20 lines · Mike Gilbert
Quoted 19 lines · Eli Schwartz
Quoted 16 lines · Vladimir 'phcoder' Serbinenko
Quoted 10 lines · On Wed, 6 Nov 2019, 20:55 Michael Chang, <[email protected]> wrote:
Quoted 2 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed.

I fully agree to work out a proper solution, but before that I think it is necessary to stop it from spread out going forward, as the running system upgrading to new version could be affected and the consequence is fatal - an unbootable system, qualified to be show stopper imho.

We don't do a release right now, so I don't think it's justified. Also increase in size can easily come from a compiler. If an increase in size can make system unbootable on upgrade, I'd rather be fixing this than just pepper over it.

Quoted 7 lines
Quoted 2 lines

Given grub 2.06 is on the horizon, is it time to re-evaluate this and disable something known to be badly broken?

If I have read the thread correctly, this only breaks if you try to install grub in an undersized embedding area. This would really only affect new grub installs; existing ones can keep using the currently installed code.

"changing grub only affects people who newly run the grub-install command" therefore it's not worth considering whether to make sure grub-install completes without errors? I don't comprehend this logic. Who ever suggested we care about people who aren't the target audience of new grub versions to begin with?

But anyway this isn't true. There are valid reasons to reinstall grub on old systems, in which case you are most likely not benefiting from zstd support one way or another, but in this case, rerunning grub-install destroys the working bootloader code and fails to replace.

https://bugs.archlinux.org/task/63656 our infrastructure apparently somehow ended up mismatching core.img and /boot modules, and had to rerun grub-install (it's not clear what happened exactly, possibly grub-install accidentally got rerun in a broken manner?) and then this remote server would not boot, and could not be fixed without hunting for the right old version of grub via the rescue system, reinstalling it, and using it to restore the boring (working) old pre-zstd bootloader code, after a lot of painful debugging.

Quoted 3 lines

I think it would be better to hold out for a real fix than to disable a feature for an entire platform where only a subset of configurations would not work.

Quoted 7 lines

I'm astonished that "it's better to hold out for a real fix than to ensure users' systems actually work".

Why is the benefit of enabling zstd support in situations where it very often breaks everything, worth more than the disadvantage of, in fact, breaking everything? The feature was broken out of the gate, no proper fix seems imminent, let's bandage the wound before more people get hurt.

If the feature is "broken", it should be reverted fully, not disabled on a single platform. That might encourage the original contributor to implement it properly next time.

#14 lore raw
In reply to Eli Schwartz
Quoted 4 lines · Eli Schwartz

But anyway this isn't true. There are valid reasons to reinstall grub on old systems, in which case you are most likely not benefiting from zstd support one way or another, but in this case, rerunning grub-install destroys the working bootloader code and fails to replace.

This sounds like a bug/defect in grub-install. Ideally, it would look at the size of the embedding area, and abort early if it is too small. This should happen before files are copied to /boot/grub, but currently it is done afterward.

#15 lore raw
Quoted 10 lines · Mike Gilbert
Quoted 4 lines · Eli Schwartz

But anyway this isn't true. There are valid reasons to reinstall grub on old systems, in which case you are most likely not benefiting from zstd support one way or another, but in this case, rerunning grub-install destroys the working bootloader code and fails to replace.

This sounds like a bug/defect in grub-install. Ideally, it would look at the size of the embedding area, and abort early if it is too small. This should happen before files are copied to /boot/grub, but currently it is done afterward.

Even though the system can stay bootable, it cannot receive any grub updates afterward, thus breaking the principle of backward compatibilty that new build couldn't run at all on an existing system running old grub.

Thanks,
Michael

Quoted 5 lines

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#16 lore raw
Quoted 13 lines · Michael Chang
Quoted 10 lines · Mike Gilbert
Quoted 4 lines · Eli Schwartz

But anyway this isn't true. There are valid reasons to reinstall grub on old systems, in which case you are most likely not benefiting from zstd support one way or another, but in this case, rerunning grub-install destroys the working bootloader code and fails to replace.

This sounds like a bug/defect in grub-install. Ideally, it would look at the size of the embedding area, and abort early if it is too small. This should happen before files are copied to /boot/grub, but currently it is done afterward.

Quoted 4 lines

Even though the system can stay bootable, it cannot receive any grub updates afterward, thus breaking the principle of backward compatibilty that new build couldn't run at all on an existing system running old grub.

There was some discussion earlier in the thread that proposed the documentation be updated to indicate that using a small embedding area is not well-supported.

I don't think 100% backward compatibility is a reasonable target, especially for configurations that have never been well-supported in the first place.

#17 lore raw
In reply to Vladimir 'phcoder' Serbinenko
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Daniel

Quoted 145 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 144 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +1 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY

grub-core/fs/btrfs.c +9 -0
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode),
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +5 -0
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +1 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +3 -1
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

#18 lore raw
Quoted 9 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

Quoted 147 lines

Daniel

Quoted 145 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 144 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +1 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY

grub-core/fs/btrfs.c +9 -0
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode),
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +5 -0
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +1 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +3 -1
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
 
--

2.16.4

#19 lore raw
Quoted 7 lines · Daniel Kiper
Quoted 6 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 5 lines
Quoted 3 lines

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

I don't think we are able to give up MBR gap support, simply because no other way out in creating the area for bootloader embedding. However we should consider to correct this claim in manual.

"You must ensure that the first partition starts at least 31 KiB (63 sectors) from the start of the disk"

to reflect the fact that 31 KiB is no longer applicable and the ideal size should be above 1MB (or such). You should go check with your disk tools to find the suitable parameter to fulfill the requirement, for eg, the partition alignment would mostly affect this.

Thanks,
Michael

Quoted 153 lines
Quoted 146 lines

Daniel

Quoted 143 lines · On Tue, 5 Nov 2019, 01:25 Michael Chang, <[email protected]> wrote:
Quoted 142 lines

The zstd support in btrfs has dependenciy to zstd module and core.img grows its size significantly to 75KB on my system. The resulted image cannot be installed into btrfs bootloader area in the size of 64KB and eventually fails with following message.

/usr/sbin/grub-install: warning: your core.img is unusually large. It won't fit in the embedding area.
/usr/sbin/grub-install: error: filesystem `btrfs' doesn't support blocklists.

The patch disabled the zstd support of btrfs in pc-bios platform to avoid the regression. The resulting size is 56KB, albeit a bit too close to the 64KB but works. This is simple workaround until a proper fix landed upstream.

---
 grub-core/fs/btrfs.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
grub-core/fs/btrfs.c +1 -0
index 48bd3d04a..8f98892d3 100644
@@ -17,6 +17,7 @@
  *  along with GRUB.  If not, see <http://www.gnu.org/licenses/>.
  */
 
+#ifndef GRUB_MACHINE_PCBIOS
 /*
  * Tell zstd to expose functions that aren't part of the stable API, which
  * aren't safe to use when linking against a dynamic library. We vendor

in a

grub-core/fs/btrfs.c +0 -0
@@ -24,6 +25,7 @@
  * functions to provide our own allocator, which uses grub_malloc(), to

zstd.
*/
#define ZSTD_STATIC_LINKING_ONLY

grub-core/fs/btrfs.c +9 -0
+#endif
 
 #include <grub/err.h>
 #include <grub/file.h>
@@ -35,7 +37,9 @@
 #include <grub/lib/crc.h>
 #include <grub/deflate.h>
 #include <minilzo.h>
+#ifndef GRUB_MACHINE_PCBIOS
 #include <zstd.h>
+#endif
 #include <grub/i18n.h>
 #include <grub/btrfs.h>
 #include <grub/crypto.h>
@@ -56,8 +60,10 @@ GRUB_MOD_LICENSE ("GPLv3+");
 #define GRUB_BTRFS_LZO_BLOCK_MAX_CSIZE (GRUB_BTRFS_LZO_BLOCK_SIZE + \
                                     (GRUB_BTRFS_LZO_BLOCK_SIZE / 16) + 64
+ 3)
 
+#ifndef GRUB_MACHINE_PCBIOS
 #define ZSTD_BTRFS_MAX_WINDOWLOG 17
 #define ZSTD_BTRFS_MAX_INPUT     (1 << ZSTD_BTRFS_MAX_WINDOWLOG)
+#endif
 
 typedef grub_uint8_t grub_btrfs_checksum_t[0x20];
 typedef grub_uint16_t grub_btrfs_uuid_t[8];
@@ -229,7 +235,9 @@ struct grub_btrfs_extent_data
 #define GRUB_BTRFS_COMPRESSION_NONE 0
 #define GRUB_BTRFS_COMPRESSION_ZLIB 1
 #define GRUB_BTRFS_COMPRESSION_LZO  2
+#ifndef GRUB_MACHINE_PCBIOS
 #define GRUB_BTRFS_COMPRESSION_ZSTD 3
+#endif
 
 #define GRUB_BTRFS_OBJECT_ID_CHUNK 0x100
 
@@ -1229,6 +1237,7 @@ grub_btrfs_read_inode (struct grub_btrfs_data *data,
   return grub_btrfs_read_logical (data, elemaddr, inode, sizeof (*inode),
 0);
 }
 
+#ifndef GRUB_MACHINE_PCBIOS
 static void *grub_zstd_malloc (void *state __attribute__((unused)),
 size_t size)
 {
   return grub_malloc (size);
grub-core/fs/btrfs.c +5 -0
@@ -1318,6 +1327,7 @@ err:
 
   return ret;
 }
+#endif
 
 static grub_ssize_t
 grub_btrfs_lzo_decompress(char *ibuf, grub_size_t isize, grub_off_t off,
@@ -1494,8 +1504,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
 
       if (data->extent->compression != GRUB_BTRFS_COMPRESSION_NONE
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZLIB
+#ifndef GRUB_MACHINE_PCBIOS
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO
          && data->extent->compression != GRUB_BTRFS_COMPRESSION_ZSTD)
+#else
+         && data->extent->compression != GRUB_BTRFS_COMPRESSION_LZO)
+#endif
        {
          grub_error (GRUB_ERR_NOT_IMPLEMENTED_YET,
                      "compression type 0x%x not supported",
grub-core/fs/btrfs.c +1 -0
@@ -1535,6 +1549,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#ifndef GRUB_MACHINE_PCBIOS
          else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
            {
              if (grub_btrfs_zstd_decompress (data->extent->inl,

data->extsize -

grub-core/fs/btrfs.c +3 -0
@@ -1544,6 +1559,7 @@ grub_btrfs_extent_read (struct grub_btrfs_data *data,
                  != (grub_ssize_t) csize)
                return -1;
            }
+#endif
          else
            grub_memcpy (buf, data->extent->inl + extoff, csize);
          break;
@@ -1581,10 +1597,12 @@ grub_btrfs_extent_read (struct grub_btrfs_data
 *data,
                ret = grub_btrfs_lzo_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#ifndef GRUB_MACHINE_PCBIOS
              else if (data->extent->compression ==
 GRUB_BTRFS_COMPRESSION_ZSTD)
                ret = grub_btrfs_zstd_decompress (tmp, zsize, extoff
                                    + grub_le_to_cpu64
 (data->extent->offset),
                                    buf, csize);
+#endif
              else
                ret = -1;
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.16.4

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel

#20 lore raw
Quoted 8 lines · Michael Chang
Quoted 7 lines · Daniel Kiper
Quoted 6 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

I don't think we are able to give up MBR gap support, simply because no other way out in creating the area for bootloader embedding. However we

Nope, I do not propose that...

Quoted 9 lines

should consider to correct this claim in manual.

"You must ensure that the first partition starts at least 31 KiB (63 sectors) from the start of the disk"

to reflect the fact that 31 KiB is no longer applicable and the ideal size should be above 1MB (or such). You should go check with your disk tools to find the suitable parameter to fulfill the requirement, for eg, the partition alignment would mostly affect this.

Exactly! However, there are some legacy systems which do not boot if MBR gap does not end at 63 sectors boundary. Hence, maybe we should suggest chainloading, using e.g. SYSLINUX, in a such cases. Anyway, may I ask you to prepare a patch for GRUB manual which describes the problem?

Daniel

#21 lore raw
Quoted 30 lines · Daniel Kiper
Quoted 8 lines · Michael Chang
Quoted 7 lines · Daniel Kiper
Quoted 6 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

I don't think we are able to give up MBR gap support, simply because no other way out in creating the area for bootloader embedding. However we

Nope, I do not propose that...

Quoted 9 lines

should consider to correct this claim in manual.

"You must ensure that the first partition starts at least 31 KiB (63 sectors) from the start of the disk"

to reflect the fact that 31 KiB is no longer applicable and the ideal size should be above 1MB (or such). You should go check with your disk tools to find the suitable parameter to fulfill the requirement, for eg, the partition alignment would mostly affect this.

Quoted 4 lines

Exactly! However, there are some legacy systems which do not boot if MBR gap does not end at 63 sectors boundary. Hence, maybe we should suggest chainloading, using e.g. SYSLINUX, in a such cases. Anyway, may I ask you to prepare a patch for GRUB manual which describes the problem?

Alright, sounds good to me. :)

Regards,
Michael

Quoted 2 lines

Daniel

#22 lore raw
Quoted 31 lines · Michael Chang
Quoted 30 lines · Daniel Kiper
Quoted 8 lines · Michael Chang
Quoted 7 lines · Daniel Kiper
Quoted 6 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

I don't think we are able to give up MBR gap support, simply because no other way out in creating the area for bootloader embedding. However we

Nope, I do not propose that...

Quoted 9 lines

should consider to correct this claim in manual.

"You must ensure that the first partition starts at least 31 KiB (63 sectors) from the start of the disk"

to reflect the fact that 31 KiB is no longer applicable and the ideal size should be above 1MB (or such). You should go check with your disk tools to find the suitable parameter to fulfill the requirement, for eg, the partition alignment would mostly affect this.

Quoted 6 lines
Quoted 4 lines

Exactly! However, there are some legacy systems which do not boot if MBR gap does not end at 63 sectors boundary. Hence, maybe we should suggest chainloading, using e.g. SYSLINUX, in a such cases. Anyway, may I ask you to prepare a patch for GRUB manual which describes the problem?

Alright, sounds good to me. :)

Michael, ping? I would like to see this limitation documented in 2.06. Could you do that?

Daniel

#23 lore raw
Quoted 37 lines · Daniel Kiper
Quoted 36 lines · Michael Chang
Quoted 34 lines · Daniel Kiper
Quoted 8 lines · Michael Chang
Quoted 7 lines · Daniel Kiper
Quoted 6 lines · Daniel Kiper
Quoted 4 lines · Vladimir 'phcoder' Serbinenko

Please don't do it this way. The right solution is to move it to separate module and include zstd module when needed. Not everybody uses btrfs embedded area. I recommend not to use it. Using mbr gap or BBP is the recommended way.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I will put the cat among the pigeons. Maybe we should finally stop pretending that the GRUB supports small MBR gaps. Otherwise we will be fighting with such issues endlessly.

Bumping the thread... Nobody objects? Hmmm...

I don't think we are able to give up MBR gap support, simply because no other way out in creating the area for bootloader embedding. However we

Nope, I do not propose that...

Quoted 9 lines

should consider to correct this claim in manual.

"You must ensure that the first partition starts at least 31 KiB (63 sectors) from the start of the disk"

to reflect the fact that 31 KiB is no longer applicable and the ideal size should be above 1MB (or such). You should go check with your disk tools to find the suitable parameter to fulfill the requirement, for eg, the partition alignment would mostly affect this.

Exactly! However, there are some legacy systems which do not boot if MBR gap does not end at 63 sectors boundary. Hence, maybe we should suggest chainloading, using e.g. SYSLINUX, in a such cases. Anyway, may I ask you to prepare a patch for GRUB manual which describes the problem?

Quoted 4 lines
Quoted 1 line

Alright, sounds good to me. :)

Michael, ping? I would like to see this limitation documented in 2.06. Could you do that?

Hi Daniel,

OK. I'll come up with a patch later this week. I am sorry for the delay.

Regards,
Michael

Quoted 7 lines

Daniel

_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel