kern/dl: Add module version check
v3 cover kern/dl: Add module version check
Hi,
This is the V3 of my patch.
V2 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00234.html
V1 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00213.html
[ TD;LR ]
1) The check is always enforced when GRUB is locked down, i.e. modules will be refused to load if they have mismatched version
2) If built with "--disable-modver-check", modules can always be loaded even if they have mismatched version, and no message will be displayed.
3) If built with "--enable-modver-check=audit", modules can always be loaded even if they have mismatched version, but if mismatch is found, a warning message will be displayed. This is the default.
4) If built with "--enable-modver-check=enforce", the behavior is same as 1)
[ Why this patch is useful, even for BIOS boot ]
Because it helps people diagnose broken (or improper) GRUB installations.
For example, if you google "452: out of range pointer", you will got a lot of results in 2022. I think the most of them are related to mismatched modules. However, these problem are often not properly diagnosed because they disappear magically, e.g. update whole system (which triggers grub reinstall). There are several people even suspect there are problems with their hard disk / BIOS. However, the root cause is 052e6068be62 ("mm: When adding a region, merge with region after as well as before") changed the layout of struct grub_mm_region, which is both used in main program and "relocator.mod", so the module reads the wrong field and crashes GRUB. Please the commit did nothing wrong because there is no API/ABI compatibility guarantees in GRUB.
If there are warning messages about mismatched modules, user will easily notice there are problems with their GRUB installation.
[ Why not enforce this check to prevent crashes ]
As Glenn & Pete said, most mismatched modules isn't harmful. At most times, GRUB with mismatched modules can boot Linux happily, even if these modules come from another Linux distribution. This enables user to fix his/her GRUB installation without using a boot/rescue disk, because the user can boot the existing Linux using the existing (but improperly installed) GRUB.
[ Why warning can be disabled ]
Some tools like Rufus relies on mismatched modules. Some advanced users also doesn't like redundant warnings for their existing known-to-work configurations.
However, it's highly unrecommended to disable this warning.
[ Why this patch is a prerequisite for external signed module support ]
Consider this scenario:
1) GRUB 2.XX is free of vulnerabilities
2) GRUB 2.YY is also free of vulnerabilities
3) So GRUB 2.XX shares same SBAT numbers with GRUB 2.YY, therefore SBAT can't help in version check
4) If there is no version check, it's possible to load GRUB 2.YY modules into GRUB 2.XX (and vice versa)
5) However, due to some changes in API or ABI, although unlikely, there is possibility that there are vulnerabilities when using GRUB 2.YY modules with GRUB 2.XX (and vice versa)
6) So we must enforce version check to prevent this from happening
However, because version string is only consisted of PACKAGE_VERSION, it must be unique for one given vendor (signer). For example, version string need to be different for Debian 10 and Debian 11 even they both use GRUB 2.06, and no two build in Debian 10 (or Debian 11) have same version string.
Best Regards,
Zhang Boyang
RFC PATCH v3 1/1 kern/dl: Add module version check
This patch add version information to GRUB modules. Specifically, PACKAGE_VERSION is embedded as null-terminated string in .modver section. This string is checked at module loading time.
If GRUB is locked down, modules with mismatched version will be rejected. This is necessary for implementing external signed modules securely (in future).
If GRUB is not locked down, the behavior is configurable at build time by "--enable-modver-check={audit|enforce}". If the option is set to "enforce", the behavior is same as lockdown mode. If the option is set to "audit" (this is the default), modules can still be loaded even if they have mismatched version, and a warning message will be emited. If the option is disabled by "--disable-modver-check", modules can always be loaded and no warning message will be generated. Please note this option doesn't have any effect if GRUB is locked down.
It should be pointed out that the version string is only consisted of PACKAGE_VERSION. Any difference not reflected in PACKAGE_VERSION is not noticed by version check (e.g. configure options).
--- config.h.in | 5 +++++ configure.ac | 12 ++++++++++ grub-core/Makefile.am | 2 +- grub-core/genmod.sh.in | 28 ++++++++++++++--------- grub-core/kern/dl.c | 42 +++++++++++++++++++++++++++++++++++ util/grub-module-verifierXX.c | 10 +++++++++ 6 files changed, 87 insertions(+), 12 deletions(-)
config.h.in +5 -0
@@ -13,6 +13,11 @@ #define MM_DEBUG @MM_DEBUG@ #endif +#define MODVER_CHECK @MODVER_CHECK@ +#define MODVER_CHECK_IGNORE 0 +#define MODVER_CHECK_AUDIT 1 +#define MODVER_CHECK_ENFORCE 2 + /* Define to 1 to enable disk cache statistics. */ #define DISK_CACHE_STATS @DISK_CACHE_STATS@ #define BOOT_TIME_STATS @BOOT_TIME_STATS@
configure.ac +12 -0
@@ -1521,6 +1521,18 @@ else fi AC_SUBST([MM_DEBUG]) +AC_ARG_ENABLE([modver-check], + AS_HELP_STRING([--enable-modver-check={audit|enforce}], + [enable module version check for non-lockdown mode (default=audit)])) +if test "x$enable_modver_check" = xno; then + MODVER_CHECK=0 +elif test "x$enable_modver_check" = xenforce; then + MODVER_CHECK=2 +else + MODVER_CHECK=1 +fi +AC_SUBST([MODVER_CHECK]) + AC_ARG_ENABLE([cache-stats], AS_HELP_STRING([--enable-cache-stats], [enable disk cache statistics collection]))
grub-core/Makefile.am +1 -1
@@ -40,7 +40,7 @@ gentrigtables$(BUILD_EXEEXT): gentrigtables.c CLEANFILES += gentrigtables$(BUILD_EXEEXT) build-grub-module-verifier$(BUILD_EXEEXT): $(top_srcdir)/util/grub-module-verifier.c $(top_srcdir)/util/grub-module-verifier32.c $(top_srcdir)/util/grub-module-verifier64.c $(top_srcdir)/grub-core/kern/emu/misc.c $(top_srcdir)/util/misc.c - $(BUILD_CC) -o $@ -I$(top_srcdir)/include $(BUILD_CFLAGS) $(BUILD_CPPFLAGS) $(BUILD_LDFLAGS) -DGRUB_BUILD=1 -DGRUB_UTIL=1 -DGRUB_BUILD_PROGRAM_NAME=\"build-grub-module-verifier\" $^ + $(BUILD_CC) -o $@ -I$(top_srcdir)/include $(BUILD_CFLAGS) $(BUILD_CPPFLAGS) $(BUILD_LDFLAGS) -DGRUB_BUILD=1 -DGRUB_UTIL=1 -DGRUB_BUILD_PROGRAM_NAME=\"build-grub-module-verifier\" -DPACKAGE_VERSION=\"$(PACKAGE_VERSION)\" $^ CLEANFILES += build-grub-module-verifier$(BUILD_EXEEXT) # trigtables.c
grub-core/genmod.sh.in +17 -11
@@ -36,22 +36,25 @@ deps=`grep ^$modname: $moddep | sed s@^.*:@@` rm -f $tmpfile $outfile if test x@TARGET_APPLE_LINKER@ != x1; then - # stripout .modname and .moddeps sections from input module - @TARGET_OBJCOPY@ -R .modname -R .moddeps $infile $tmpfile + # stripout .modname and .moddeps and .modver sections from input module + @TARGET_OBJCOPY@ -R .modname -R .moddeps -R .modver $infile $tmpfile - # Attach .modname and .moddeps sections + # Attach .modname and .moddeps and .modver sections t1=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1 printf "$modname\0" >$t1 t2=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1 for dep in $deps; do printf "$dep\0" >> $t2; done + t3=`mktemp "${TMPDIR:-/tmp}/tmp.XXXXXXXXXX"` || exit 1 + printf "@PACKAGE_VERSION@\0" >$t3 + if test -n "$deps"; then - @TARGET_OBJCOPY@ --add-section .modname=$t1 --add-section .moddeps=$t2 $tmpfile + @TARGET_OBJCOPY@ --add-section .modname=$t1 --add-section .moddeps=$t2 --add-section .modver=$t3 $tmpfile else - @TARGET_OBJCOPY@ --add-section .modname=$t1 $tmpfile + @TARGET_OBJCOPY@ --add-section .modname=$t1 --add-section .modver=$t3 $tmpfile fi - rm -f $t1 $t2 + rm -f $t1 $t2 $t3 if test x@platform@ != xemu; then @TARGET_STRIP@ --strip-unneeded \
@@ -71,23 +74,26 @@ else tmpfile2=${outfile}.tmp2 t1=${outfile}.t1.c t2=${outfile}.t2.c + t3=${outfile}.t3.c # remove old files if any - rm -f $t1 $t2 + rm -f $t1 $t2 $t3 cp $infile $tmpfile - # Attach .modname and .moddeps sections + # Attach .modname and .moddeps and .modver sections echo "char modname[] __attribute__ ((section(\"_modname, _modname\"))) = \"$modname\";" >$t1 for dep in $deps; do echo "char moddep_$dep[] __attribute__ ((section(\"_moddeps, _moddeps\"))) = \"$dep\";" >>$t2; done + echo "char modver[] __attribute__ ((section(\"_modver, _modver\"))) = \"@PACKAGE_VERSION@\";" >$t3 + if test -n "$deps"; then - @TARGET_CC@ @TARGET_LDFLAGS@ -ffreestanding -nostdlib -o $tmpfile2 $t1 $t2 $tmpfile -Wl,-r + @TARGET_CC@ @TARGET_LDFLAGS@ -ffreestanding -nostdlib -o $tmpfile2 $t1 $t2 $t3 $tmpfile -Wl,-r else - @TARGET_CC@ @TARGET_LDFLAGS@ -ffreestanding -nostdlib -o $tmpfile2 $t1 $tmpfile -Wl,-r + @TARGET_CC@ @TARGET_LDFLAGS@ -ffreestanding -nostdlib -o $tmpfile2 $t1 $t3 $tmpfile -Wl,-r fi - rm -f $t1 $t2 $tmpfile + rm -f $t1 $t2 $t3 $tmpfile mv $tmpfile2 $tmpfile cp $tmpfile $tmpfile.bin
grub-core/kern/dl.c +42 -0
60 lines not on this page · raw message
util/grub-module-verifierXX.c +10 -0
@@ -491,8 +491,18 @@ SUFFIX(grub_module_verify) (const char * const filename,check_license (filename, arch, e);Elf_Shdr *s; +const char *modver;const char *modname; + s =find_section (arch, e,".modver" ); +if (!s) +grub_util_error ("%s: no module version found" , filename); + + modver = (const char *) e +grub_target_to_host (s->sh_offset); + +if (strcmp (modver,PACKAGE_VERSION ) !=0 ) +grub_util_error ("%s: bad module version" , filename); + s =find_section (arch, e,".modname" );if (!s)grub_util_error ("%s: no module name found" , filename);
Signature · 1 line
2.30.2
I'm sorry but that's NO_GO for me.
This is a major step back from v2, where the check was never applicable for BIOS and I believe I explained why a module check that can be enforced for BIOS is going to create a major headache for end-users.
It seems pretty obvious that, if distro maintainers start to use --enable-modver-check=enforce for UEFI they are also going to use the same option for BIOS (because why should they use a different compilation options if it applies for both?), and you *ARE* going to leave Windows users stranded through them not trying to boot GRUB based media because they will be forced to use DD to create a workable media (even though, and this is the major point here, in the very vast majority of cases there is not issue with mixing versions if you're on the same major/minor base for GRUB), and consider that, because they cannot see its content, or can only see the ESP, their media was not properly created and is not worth trying to boot.
This is a *VERY REAL* problem, for which I have provided many examples (see [1]), and could provide even more, which is why utilities like Rufus try to steer away from using DD mode where possible.
However, if people can use --enable-modver-check=enforce for BIOS, then, under the aim of solving one minor issue (that from looking the Google links for "452: out of range pointer" seem to be a problem that is mostly encountered in UEFI environments rather than BIOS, or at least is not a BIOS-only issue, and therefore something that will be sorted out for the majority of folks experiencing it *even* if --enable-modver-check cannot apply for BIOS), you will be adding to an already existing a major one, and you are going to leave people, and especially Windows users who are trying to switch to Linux, stranded.
In short, GRUB would be doing a major disservice to its end users if it is to allow module version checks to be enforceable for BIOS, and, considering how there is little benefit in enforcing this in an environment that cannot be secured in the first place, it's hard to see a justification for what is essentially a *MAJOR* design regression from the v2 of this patch. Or at the very least, a better approach is to introduce it for UEFI only, then, once that has been integrated with UEFI for a while, see if BIOS users are actually reporting issues due to module version mismatch, and reconvene on whether we really want to allow this check to also apply to BIOS.
Thus, in my view, the proper approach for the sake of the end users is leave default BIOS boot behaviour as it currently stands, with no warning and no possibility of failing the boot process on version mismatch (because, and I am well placed to know, both from experience as well as the feedback I get from Rufus users, that in the very vast majority of cases, as long as you use modules and core.img based on the same major+minor, there is no issue, and producing a warning that hints that there may be is only going to be detrimental to the end user).
And, once again, I have to ask this mailing list to please try not to consider issues only from a Linux/UNIX based maintainer's perspective, but also consider them from the end-users who are actually going to be creating GRUB based media on Windows, where using ISOHybrid as DD is far from the panacea it's supposed to be.
Finally, if you are interested in seeing how end users actually struggle with boot media and boot media creation, I *strongly* invite you to spend some time on reddit, askubuntu, superuser and other end-user places, where you should be able to validate that, as I have pointed out, Windows end users really do struggle with ISOHybrdids that are written in DD mode, which will hopefully help you reach the conclusion that, if this v3 proposal gets integrated, it will cause more harm for BIOS users than it will benefit them on account that distro maintainers are bound to start using --enable-modver-check=enforce everywhere, if it can apply to both BIOS and UEFI.
Regards,
/Pete
[1] https://lists.gnu.org/archive/html/grub-devel/2022-06/msg00024.html
Quoted 64 lines · Zhang Boyang
Hi,
This is the V3 of my patch.
V2 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00234.htmlV1 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00213.html[ TD;LR ]
1) The check is always enforced when GRUB is locked down, i.e. modules will be refused to load if they have mismatched version
2) If built with "--disable-modver-check", modules can always be loaded even if they have mismatched version, and no message will be displayed.
3) If built with "--enable-modver-check=audit", modules can always be loaded even if they have mismatched version, but if mismatch is found, a warning message will be displayed. This is the default.
4) If built with "--enable-modver-check=enforce", the behavior is same as 1)
[ Why this patch is useful, even for BIOS boot ]
Because it helps people diagnose broken (or improper) GRUB installations.
For example, if you google "452: out of range pointer", you will got a lot of results in 2022. I think the most of them are related to mismatched modules. However, these problem are often not properly diagnosed because they disappear magically, e.g. update whole system (which triggers grub reinstall). There are several people even suspect there are problems with their hard disk / BIOS. However, the root cause is
052e6068be62("mm: When adding a region, merge with region after as well as before") changed the layout ofstruct grub_mm_region, which is both used in main program and "relocator.mod", so the module reads the wrong field and crashes GRUB. Please the commit did nothing wrong because there is no API/ABI compatibility guarantees in GRUB.If there are warning messages about mismatched modules, user will easily notice there are problems with their GRUB installation.
[ Why not enforce this check to prevent crashes ]
As Glenn & Pete said, most mismatched modules isn't harmful. At most times, GRUB with mismatched modules can boot Linux happily, even if these modules come from another Linux distribution. This enables user to fix his/her GRUB installation without using a boot/rescue disk, because the user can boot the existing Linux using the existing (but improperly installed) GRUB.
[ Why warning can be disabled ]
Some tools like Rufus relies on mismatched modules. Some advanced users also doesn't like redundant warnings for their existing known-to-work configurations.
However, it's highly unrecommended to disable this warning.
[ Why this patch is a prerequisite for external signed module support ]
Consider this scenario:
1) GRUB 2.XX is free of vulnerabilities
2) GRUB 2.YY is also free of vulnerabilities
3) So GRUB 2.XX shares same SBAT numbers with GRUB 2.YY, therefore SBAT can't help in version check
4) If there is no version check, it's possible to load GRUB 2.YY modules into GRUB 2.XX (and vice versa)
5) However, due to some changes in API or ABI, although unlikely, there is possibility that there are vulnerabilities when using GRUB 2.YY modules with GRUB 2.XX (and vice versa)
6) So we must enforce version check to prevent this from happening
However, because version string is only consisted of
PACKAGE_VERSION, it must be unique for one given vendor (signer). For example, version string need to be different for Debian 10 and Debian 11 even they both use GRUB 2.06, and no two build in Debian 10 (or Debian 11) have same version string.Best Regards,
Zhang Boyang
Hi,
I think there are some misunderstandings.
This feature is implemented in kern/dl.c in core.img, which is UNDER YOUR CONTROL.
Let me analyze the worst case from your perspective:
1) Every distro is enforcing version check, even in BIOS mode.
2) Because this check is in kern/dl.c, there is no such code in any module (.mod) files. However, this modules are little fatter because they have version information (.modver section), which is harmless.
3) You can still build your own copy of core.img with this check disabled (or just revert this patch completely), then ship that core.img with Rufus.
4) Since there is no check in your core.img, it will load any module happily, regardless whether it has mismatched version information.
Therefore, there is no problem at all. The only thing you need to do is build your core.img with this patch reverted in the future. Then things will "work" as usual.
Let me analyze why I want to add version check to BIOS mode:
1) Because you can always remove this patch from your build for Rufus, there is no harm in having this patch in vanilla GRUB.
2) This patch can provide benefits to BIOS users, because it can diagnose improper installations. For example, user runs grub-install /dev/sdb to update GRUB but machine boot from /dev/sda.
So there are benefits without any harm, why not have it in BIOS mode?
By the way, I highly recommend you to test latest git GRUB (as a preview of upcoming 2.12 release) with Rufus and it will almost certainly break your existing Rufus build. e.g. use Rufus (with GRUB2.06) to process ISOs (with GRUB2.12); and the reverse, use Rufus (with GRUB2.12) to process ISOs (with GRUB2.06). You probably need to ship another bunch of core.img with Rufus.
Also, I'd also like to advice you to create a patchset/proposal, which make it easy to convert the bootcode in ISO to the bootcode in USB. I think you are expert in this area. If that patch finally got merged (e.g. before 2.12 release window), you life will be definitely easier because you are free from shipping another bunch of core.img. However, since I'm not a expert of this area, please point out if I said something wrong.
Best Regards,
Zhang Boyang
Quoted 94 lines · Pete Batard
I'm sorry but that's
NO_GOfor me.This is a major step back from v2, where the check was never applicable for BIOS and I believe I explained why a module check that can be enforced for BIOS is going to create a major headache for end-users.
It seems pretty obvious that, if distro maintainers start to use
--enable-modver-check=enforce for UEFI they are also going to use the
same option for BIOS (because why should they use a different compilation options if it applies for both?), and you *ARE* going to leave Windows users stranded through them not trying to boot GRUB based media because they will be forced to use DD to create a workable media (even though, and this is the major point here, in the very vast majority of cases there is not issue with mixing versions if you're on the same major/minor base for GRUB), and consider that, because they cannot see its content, or can only see the ESP, their media was not properly created and is not worth trying to boot.This is a *VERY REAL* problem, for which I have provided many examples (see [1]), and could provide even more, which is why utilities like Rufus try to steer away from using DD mode where possible.
However, if people can use --enable-modver-check=enforce for BIOS, then, under the aim of solving one minor issue (that from looking the Google links for "452: out of range pointer" seem to be a problem that is mostly encountered in UEFI environments rather than BIOS, or at least is not a BIOS-only issue, and therefore something that will be sorted out for the majority of folks experiencing it *even* if --enable-modver-check cannot apply for BIOS), you will be adding to an
already existing a major one, and you are going to leave people, and especially Windows users who are trying to switch to Linux, stranded.In short, GRUB would be doing a major disservice to its end users if it is to allow module version checks to be enforceable for BIOS, and, considering how there is little benefit in enforcing this in an environment that cannot be secured in the first place, it's hard to see a justification for what is essentially a *MAJOR* design regression from the v2 of this patch. Or at the very least, a better approach is to introduce it for UEFI only, then, once that has been integrated with UEFI for a while, see if BIOS users are actually reporting issues due to module version mismatch, and reconvene on whether we really want to allow this check to also apply to BIOS.
Thus, in my view, the proper approach for the sake of the end users is leave default BIOS boot behaviour as it currently stands, with no warning and no possibility of failing the boot process on version mismatch (because, and I am well placed to know, both from experience as well as the feedback I get from Rufus users, that in the very vast majority of cases, as long as you use modules and core.img based on the same major+minor, there is no issue, and producing a warning that hints that there may be is only going to be detrimental to the end user).
And, once again, I have to ask this mailing list to please try not to consider issues only from a Linux/UNIX based maintainer's perspective, but also consider them from the end-users who are actually going to be creating GRUB based media on Windows, where using ISOHybrid as DD is far from the panacea it's supposed to be.
Finally, if you are interested in seeing how end users actually struggle with boot media and boot media creation, I *strongly* invite you to spend some time on reddit, askubuntu, superuser and other end-user places, where you should be able to validate that, as I have pointed out, Windows end users really do struggle with ISOHybrdids that are written in DD mode, which will hopefully help you reach the conclusion that, if this v3 proposal gets integrated, it will cause more harm for BIOS users than it will benefit them on account that distro maintainers are bound to start using --enable-modver-check=enforce everywhere, if it can apply to both BIOS and UEFI.
Regards,
/Pete
[1] https://lists.gnu.org/archive/html/grub-devel/2022-06/msg00024.html
Quoted 64 lines · Zhang Boyang
Hi,
This is the V3 of my patch.
V2 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00234.htmlV1 is at:
https://lists.gnu.org/archive/html/grub-devel/2022-12/msg00213.html[ TD;LR ]
1) The check is always enforced when GRUB is locked down, i.e. modules will be refused to load if they have mismatched version
2) If built with "--disable-modver-check", modules can always be loaded even if they have mismatched version, and no message will be displayed.
3) If built with "--enable-modver-check=audit", modules can always be loaded even if they have mismatched version, but if mismatch is found, a warning message will be displayed. This is the default.
4) If built with "--enable-modver-check=enforce", the behavior is same as 1)
[ Why this patch is useful, even for BIOS boot ]
Because it helps people diagnose broken (or improper) GRUB installations.
For example, if you google "452: out of range pointer", you will got a lot of results in 2022. I think the most of them are related to mismatched modules. However, these problem are often not properly diagnosed because they disappear magically, e.g. update whole system (which triggers grub reinstall). There are several people even suspect there are problems with their hard disk / BIOS. However, the root cause is
052e6068be62("mm: When adding a region, merge with region after as well as before") changed the layout ofstruct grub_mm_region, which is both used in main program and "relocator.mod", so the module reads the wrong field and crashes GRUB. Please the commit did nothing wrong because there is no API/ABI compatibility guarantees in GRUB.If there are warning messages about mismatched modules, user will easily notice there are problems with their GRUB installation.
[ Why not enforce this check to prevent crashes ]
As Glenn & Pete said, most mismatched modules isn't harmful. At most times, GRUB with mismatched modules can boot Linux happily, even if these modules come from another Linux distribution. This enables user to fix his/her GRUB installation without using a boot/rescue disk, because the user can boot the existing Linux using the existing (but improperly installed) GRUB.
[ Why warning can be disabled ]
Some tools like Rufus relies on mismatched modules. Some advanced users also doesn't like redundant warnings for their existing known-to-work configurations.
However, it's highly unrecommended to disable this warning.
[ Why this patch is a prerequisite for external signed module support ]
Consider this scenario:
1) GRUB 2.XX is free of vulnerabilities
2) GRUB 2.YY is also free of vulnerabilities
3) So GRUB 2.XX shares same SBAT numbers with GRUB 2.YY, therefore SBAT can't help in version check
4) If there is no version check, it's possible to load GRUB 2.YY modules into GRUB 2.XX (and vice versa)
5) However, due to some changes in API or ABI, although unlikely, there is possibility that there are vulnerabilities when using GRUB 2.YY modules with GRUB 2.XX (and vice versa)
6) So we must enforce version check to prevent this from happening
However, because version string is only consisted of
PACKAGE_VERSION, it must be unique for one given vendor (signer). For example, version string need to be different for Debian 10 and Debian 11 even they both use GRUB 2.06, and no two build in Debian 10 (or Debian 11) have same version string.Best Regards,
Zhang Boyang
Quoted 6 lines · Zhang Boyang
This feature is implemented in
kern/dl.cin core.img, which is UNDER YOUR CONTROL.Let me analyze the worst case from your perspective:
1) Every distro is enforcing version check, even in BIOS mode.
Quoted 7 lines
2) Because this check is in
kern/dl.c, there is no such code in any module (.mod) files. However, this modules are little fatter because they have version information (.modver section), which is harmless.3) You can still build your own copy of core.img with this check disabled (or just revert this patch completely), then ship that core.img with Rufus.
4) Since there is no check in your core.img, it will load any module happily, regardless whether it has mismatched version information.
Therefore, there is no problem at all. The only thing you need to do is build your core.img with this patch reverted in the future. Then things will "work" as usual.
I need to properly test this, but I think that should work, yes.
I should have looked a bit more closely at your proposal, because I agree that it should address my concerns, though I do have a small worry about the possibility of transitive checks in the future (i.e. modules that perform their own loading of submodules for whatever reason, with a version check being introduced there).
I'm not planning to try to hold your patch on that, but I sure hope that transitive version checks never become a thing.
Quoted 3 lines
Let me analyze why I want to add version check to BIOS mode:
1) Because you can always remove this patch from your build for Rufus, there is no harm in having this patch in vanilla GRUB.
As long as the version check remains localised inside core.img, yes.
Quoted 5 lines
2) This patch can provide benefits to BIOS users, because it can diagnose improper installations. For example, user runs
grub-install /dev/sdbto update GRUB but machine boot from/dev/sda.So there are benefits without any harm, why not have it in BIOS mode?
By the way, I highly recommend you to test latest git GRUB (as a preview of upcoming 2.12 release) with Rufus and it will almost certainly break your existing Rufus build. e.g. use Rufus (with GRUB2.06) to process ISOs (with GRUB2.12); and the reverse, use Rufus (with GRUB2.12) to process ISOs (with GRUB2.06). You probably need to ship another bunch of core.img with Rufus.
That's already been the case for previous releases, e.g. trying to use core.img from 2.0(x) with 2.0(x-1) is usually asking for trouble, and I do have a mechanism in place to address just that, where users don't even need to update Rufus as we can always download a new 'core.img' server from our repository. This is also the mechanism that helps us address the rare case where distro maintainers have applied breaking pacthes onto a dot release that affect core.img.
So, once distros start using GRUB 2.12 (because I need to be able to validate the core.img I build against an actual image), I'll have a core.img for GRUB 2.12 that users can automatically use in Rufus.
As I pointed out before, the main issue is that even when using the same base version, and because GRUB releases are far and in-between, distro maintainers cherry pick patches to apply on top of the dot release, and (usually, which is the proper thing to do) suffix their version id, which of course, would make a version check fail.
But we're still working with the same dot release base in Rufus always, which is why the core.img replacement (barring a few exceptions) always works.
Quoted 1 line
Also, I'd also like to advice you to create a patchset/proposal, which make it easy to convert the bootcode in ISO to the bootcode in USB.
I considered it a long time ago, but the main issue is that it would only help with future GRUB based images, and not the many many current ones I needed to support.
Thus, all it would accomplish then, is force me to have two different methods of solving a problem, where one can most certainly do, and hope that, in 10 years time (at best, because you bet some people will still want to use 10-years old ISOs then, especially if they are booting old BIOS-based computers) I *might* be able to remove one of them.
I hope you can appreciate how this approach makes very little sense then when one method can accomplish all we need.
Plus, trying to reuse the files from El-Torito would be a major challenge, as, even if we were to somehow force include part_msdos, biosdisk, fat and ntfs always (which are completely unneeded for optical boot) and recover them, we'd still need to alter the prefix directory, and that resides in a part that is usually compressed.
So, the only viable alternative, would be to force optical media creators to generate a core.img that can work for USB boot (which means silent invocation of grub-mkimage behind the scenes), and have that somehow find its way on the resulting ISO media.
All in all, these are fairly heavy alterations to the GRUB optical boot generation process, especially again for people who (unlike what is the case for our UEFI File System Transposition proposal), are not guaranteed to care about ISOHybrid in the first place and will be wondering why they are suddenly forced to cater for it (which I wouldn't personally mind imposing on them, but I don't think it's the proper way to go about it).
Thus, even today, the better solution in my view is to just have a mechanism where we recompile and use our own core.img, with the modules and prefix we need, especially as experience shows that, if using the same dot release, this is seldom an issue.
Quoted 1 line
I think you are expert in this area. If that patch finally got merged (e.g. before 2.12 release window), you life will be definitely easier
Again, that would only potentially make my life easier in 10 years time, at best (while still probably leaving users who are trying to use a 10 yo ISO image stranded then, if this is when I choose to remove the old way of using our own core.img).
As with anything GRUB (and this is no criticism of the project), it usually takes years for any new feature to make its way to the end users, and I will need to support GRUB based images that have been generated with older versions of GRUB for a long long time in Rufus.
In short, please be mindful not to apply the perspective of a Linux distro maintainer (i.e. "We can just update to latest GRUB when we release our new distro and use the new feature!") to a problem that must encompass older GRUB versions along with newer ones.
Quoted 1 line
because you are free from shipping another bunch of core.img. However, since I'm not a expert of this area, please point out if I said something wrong.
I hope that what I've clarified above makes sense.
I'll be testing your proposal as soon as I get a chance, to validate that there are no hidden gotchas for my usage scenario, but it will probably be a few days. I'll report tot this list when I have done so.
Regards,
/Pete
Hi,
Quoted 9 lines · Pete Batard
Quoted 8 lines · Zhang Boyang
This feature is implemented in
kern/dl.cin core.img, which is UNDER YOUR CONTROL.Let me analyze the worst case from your perspective:
1) Every distro is enforcing version check, even in BIOS mode.
2) Because this check is in
kern/dl.c, there is no such code in any module (.mod) files. However, this modules are little fatter because they have version information (.modver section), which is harmless.
Quoted 8 lines
Quoted 5 lines
3) You can still build your own copy of core.img with this check disabled (or just revert this patch completely), then ship that core.img with Rufus.
4) Since there is no check in your core.img, it will load any module happily, regardless whether it has mismatched version information.
Therefore, there is no problem at all. The only thing you need to do is build your core.img with this patch reverted in the future. Then things will "work" as usual.
I need to properly test this, but I think that should work, yes.
I think the result will be OK. If so, it's good news for both of us.
Quoted 3 lines
I should have looked a bit more closely at your proposal, because I agree that it should address my concerns, though I do have a small worry about the possibility of transitive checks in the future (i.e. modules that perform their own loading of submodules for whatever reason, with a version check being introduced there).
I'm not planning to try to hold your patch on that, but I sure hope that transitive version checks never become a thing.
By the way, I'm planning to drop V3 and reuse V2 as the base of V4 (if Glenn's concerns is solved). So there will be no "enforce" for BIOS at all in V4, thus the "small worry" is eliminated. However, I will probably add sleep(100ms) to warning message. But don't worry, you can patch your own build like this
- || grub_dl_check_version (mod, e) + || (1||grub_dl_check_version (mod, e))
to completely disable the check.
Quoted 12 lines
Quoted 3 lines
Let me analyze why I want to add version check to BIOS mode:
1) Because you can always remove this patch from your build for Rufus, there is no harm in having this patch in vanilla GRUB.
As long as the version check remains localised inside core.img, yes.
Quoted 5 lines
2) This patch can provide benefits to BIOS users, because it can diagnose improper installations. For example, user runs
grub-install /dev/sdbto update GRUB but machine boot from/dev/sda.So there are benefits without any harm, why not have it in BIOS mode?
By the way, I highly recommend you to test latest git GRUB (as a preview of upcoming 2.12 release) with Rufus and it will almost certainly break your existing Rufus build. e.g. use Rufus (with GRUB2.06) to process ISOs (with GRUB2.12); and the reverse, use Rufus (with GRUB2.12) to process ISOs (with GRUB2.06). You probably need to ship another bunch of core.img with Rufus.
OK, That's great news.
Quoted 8 lines
That's already been the case for previous releases, e.g. trying to use core.img from 2.0(x) with 2.0(x-1) is usually asking for trouble, and I do have a mechanism in place to address just that, where users don't even need to update Rufus as we can always download a new 'core.img' server from our repository. This is also the mechanism that helps us address the rare case where distro maintainers have applied breaking pacthes onto a dot release that affect core.img.
So, once distros start using GRUB 2.12 (because I need to be able to validate the core.img I build against an actual image), I'll have a core.img for GRUB 2.12 that users can automatically use in Rufus.
As I pointed out before, the main issue is that even when using the same base version, and because GRUB releases are far and in-between, distro maintainers cherry pick patches to apply on top of the dot release, and (usually, which is the proper thing to do) suffix their version id, which of course, would make a version check fail.
But we're still working with the same dot release base in Rufus always, which is why the core.img replacement (barring a few exceptions) always works.
FYI, Ubuntu, Debian, and some other distros are likely to update their GRUB version (dot release) during life cycle, mostly because of secure boot related fixes. For example, https://tracker.debian.org/news/1352218/accepted-grub2-206-3deb10u1-source-into-oldstable-proposed-updates-oldstable-new-oldstable-proposed-updates/
Personally speaking, I guess the number of exceptions might be underestimated. Because most ISOs were using ISOLINUX instead of GRUB2 in the past, and there seems a trend of replacing ISOLINUX with GRUB2. For example, Ubuntu 20.04 uses ISOLINUX and Ubuntu 20.10 uses GRUB2, so there is only 2 years of experience of using mismatched GRUB2 for Ubuntu. So I guess problem might arise when more and more distros switch to GRUB2. However, this is only a guess, and we both hope it's not true.
Quoted 4 lines
Quoted 1 line
Also, I'd also like to advice you to create a patchset/proposal, which make it easy to convert the bootcode in ISO to the bootcode in USB.
I considered it a long time ago, but the main issue is that it would only help with future GRUB based images, and not the many many current ones I needed to support.
May be the window of GRUB 2.12 is a good chance to change future. I really think Thomas's idea is a good start point.
Quoted 22 lines
Thus, all it would accomplish then, is force me to have two different methods of solving a problem, where one can most certainly do, and hope that, in 10 years time (at best, because you bet some people will still want to use 10-years old ISOs then, especially if they are booting old BIOS-based computers) I *might* be able to remove one of them.
I hope you can appreciate how this approach makes very little sense then when one method can accomplish all we need.
Plus, trying to reuse the files from El-Torito would be a major challenge, as, even if we were to somehow force include
part_msdos, biosdisk, fat and ntfs always (which are completely unneeded for optical boot) and recover them, we'd still need to alter the prefix directory, and that resides in a part that is usually compressed.So, the only viable alternative, would be to force optical media creators to generate a core.img that can work for USB boot (which means silent invocation of grub-mkimage behind the scenes), and have that somehow find its way on the resulting ISO media.
All in all, these are fairly heavy alterations to the GRUB optical boot generation process, especially again for people who (unlike what is the case for our UEFI File System Transposition proposal), are not guaranteed to care about ISOHybrid in the first place and will be wondering why they are suddenly forced to cater for it (which I wouldn't personally mind imposing on them, but I don't think it's the proper way to go about it).
Thus, even today, the better solution in my view is to just have a mechanism where we recompile and use our own core.img, with the modules and prefix we need, especially as experience shows that, if using the same dot release, this is seldom an issue.
Quoted 1 line
I think you are expert in this area. If that patch finally got merged (e.g. before 2.12 release window), you life will be definitely easier
Again, that would only potentially make my life easier in 10 years time, at best (while still probably leaving users who are trying to use a 10 yo ISO image stranded then, if this is when I choose to remove the old way of using our own core.img).
As with anything GRUB (and this is no criticism of the project), it usually takes years for any new feature to make its way to the end users, and I will need to support GRUB based images that have been generated with older versions of GRUB for a long long time in Rufus.
In short, please be mindful not to apply the perspective of a Linux distro maintainer (i.e. "We can just update to latest GRUB when we release our new distro and use the new feature!") to a problem that must encompass older GRUB versions along with newer ones.
Quoted 1 line
because you are free from shipping another bunch of core.img. However, since I'm not a expert of this area, please point out if I said something wrong.
Quoted 7 lines
I hope that what I've clarified above makes sense.
I'll be testing your proposal as soon as I get a chance, to validate that there are no hidden gotchas for my usage scenario, but it will probably be a few days. I'll report tot this list when I have done so.
Regards,
/Pete
Best Regards,
Zhang Boyang
Quoted 2 lines · Zhang Boyang
- || grub_dl_check_version (mod, e) + || (1||grub_dl_check_version (mod, e))
Oh, there is a small error, it should be:
- || grub_dl_check_version (mod, e) + || (0 && grub_dl_check_version (mod, e))
Best Regards,
Zhang Boyang