grub-shell: Add flexibility in QEMU firmware handling
PATCH 1/1 grub-shell: Add flexibility in QEMU firmware handling
The current qemu firmware paths for arm-efi and arm64-efi are not available on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware files in the source directory and if not found, look for them in locations where Debian installs them. Prefer to use the firmware file usable with the QEMU '-bios' option and if not found use the newer pflash firmware files. This allows supporting older systems more easily. By looking for files in the source directory first, system firmware files can be overridden and it can be ensured that the tests can be run regardless of the distro and where it stores firmware files. If no firmware files are found, print an error message telling the user that those files must exist and exit with error.
Do not load the system 32-bit ARM firmware VARS file because it must be writable to prevent a data exception and boot failure. So in order to use the VARS file, it must be copied to a writable location, but its quite large at 64M and is not needed to boot successfully.
--- tests/util/grub-shell.in | 105 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 101 insertions(+), 4 deletions(-)
tests/util/grub-shell.in +101 -4
@@ -181,21 +181,92 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in boot=cd console=console trim=1 - qemuopts="-bios OVMF-ia32.fd $qemuopts" + bios=${srcdir}/OVMF32.fd + pflash_code=${srcdir}/OVMF32_CODE.fd + pflash_vars=${srcdir}/OVMF32_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + if [ -f "$pflash_vars" ]; then + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + fi + else + bios=/usr/share/qemu/OVMF32.fd + pflash_code=/usr/share/OVMF/OVMF32_CODE_4M.secboot.fd + pflash_vars=/usr/share/OVMF/OVMF32_VARS_4M.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + else + echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2 + exit 1 + fi + fi + qemuopts="-machine q35,accel=tcg $qemuopts" ;; x86_64-efi) qemu=qemu-system-x86_64 boot=cd console=console trim=1 - qemuopts="-bios OVMF.fd $qemuopts" + bios=${srcdir}/OVMF.fd + pflash_code=${srcdir}/OVMF_CODE.fd + pflash_vars=${srcdir}/OVMF_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + if [ -f "$pflash_vars" ]; then + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + fi + else + bios=/usr/share/qemu/OVMF.fd + pflash_code=/usr/share/OVMF/OVMF_CODE.fd + pflash_vars=/usr/share/OVMF/OVMF_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + else + echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2 + exit 1 + fi + fi ;; arm64-efi) qemu=qemu-system-aarch64 boot=hd console=console trim=1 - qemuopts="-machine virt -cpu cortex-a57 -bios /usr/share/qemu-efi/QEMU_EFI.fd $qemuopts" + bios=${srcdir}/AAVMF.fd + pflash_code=${srcdir}/AAVMF_CODE.fd + pflash_vars=${srcdir}/AAVMF_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + if [ -f "$pflash_vars" ]; then + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + fi + else + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + pflash_code=/usr/share/AAVMF/AAVMF_CODE.fd + pflash_vars=/usr/share/AAVMF/AAVMF_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + qemuopts="-drive if=pflash,format=raw,unit=1,readonly=on,file=$pflash_vars $qemuopts" + else + echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2 + exit 1 + fi + fi + qemuopts="-machine virt -cpu cortex-a57 $qemuopts" disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file=" serial_port= ;;
@@ -204,7 +275,33 @@ case "${grub_modinfo_target_cpu}-${grub_modinfo_platform}" in boot=hd console=console trim=1 - qemuopts="-machine virt -bios /usr/share/ovmf-arm/QEMU_EFI.fd $qemuopts" + bios=${srcdir}/AAVMF32.fd + pflash_code=${srcdir}/AAVMF32_CODE.fd + pflash_vars=${srcdir}/AAVMF32_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + if [ -f "$pflash_vars" ]; then + qemuopts="-drive if=pflash,format=raw,unit=1,file=$pflash_vars $qemuopts" + fi + else + bios=/usr/share/AAVMF/AAVMF32.fd + pflash_code=/usr/share/AAVMF/AAVMF32_CODE.fd + pflash_vars=/usr/share/AAVMF/AAVMF32_VARS.fd + if [ -f "$bios" ]; then + qemuopts="-bios $bios $qemuopts" + elif [ -f "$pflash_code" ]; then + # NOTE: Do not use the pflash VARS file because it cannot be + # used in readonly. So it would need to be copied to a writable + # path, but its large at 64M and not needed for running tests. + qemuopts="-drive if=pflash,format=raw,unit=0,readonly=on,file=$pflash_code $qemuopts" + else + echo "Firmware not found, please make sure $bios or both $pflash_code and $pflash_vars exist." >&2 + exit 1 + fi + fi + qemuopts="-machine virt $qemuopts" disk="device virtio-blk-device,drive=hd1 -drive if=none,id=hd1,file=" serial_port=efi0 ;;
Signature · 1 line
2.34.1
Quoted 4 lines · Glenn Washburn
The current qemu firmware paths for arm-efi and arm64-efi are not available on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware files in the source directory and if not found, look for them in locations where Debian installs them.
I'd suggest to inspect the *.json files in /usr/share/qemu/firmware/ to find distro-installed firmware files.
Quoted 4 lines
Do not load the system 32-bit ARM firmware VARS file because it must be writable to prevent a data exception and boot failure. So in order to use the VARS file, it must be copied to a writable location, but its quite large at 64M and is not needed to boot successfully.
You can load the VARS file with snapshot=on (and drop readonly=on) to make things work without copying the file.
take care,
Gerd
Quoted 8 lines · On Mon, 23 Jan 2023 10:28:09 +0100 Gerd Hoffmann <[email protected]> wrote:
Quoted 4 lines · Glenn Washburn
The current qemu firmware paths for arm-efi and arm64-efi are not available on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware files in the source directory and if not found, look for them in locations where Debian installs them.
I'd suggest to inspect the *.json files in
/usr/share/qemu/firmware/to find distro-installed firmware files.
Yes, I know about this, but decided against it so as not to do real or hacked up json parsing and add another dependency (eg. jq). I think right now the unstated GRUB policy is that these tests are only officially supported to run on (newer) debian systems, so I felt that hard coding was reasonable. It occurs to me that its possible (though I suspect very improbable), that Redhat based distros run the tests when building the official RPM. Is there a reason that redhat would be very interested in the change you're suggesting?
Since building GRUB already has a dependency on Python, a tiny script could be used to extract needed info from the json file. But I still find that kinda ugly cause the grub-shell script is a shell script. I'm inclined to say this is good enough as is for now, and if anyone wants to submit a patch to change it I'm open to endorsing it.
Quoted 8 lines
Quoted 5 lines
Do not load the system 32-bit ARM firmware VARS file because it must be writable to prevent a data exception and boot failure. So in order to use the VARS file, it must be copied to a writable location, but its quite large at 64M and is not needed to boot successfully.
You can load the VARS file with snapshot=on (and drop readonly=on) to make things work without copying the file.
Good to know. Do you know of any benefit to doing this (ie. using the installed VARS file)?
Thanks for the feedback,
Glenn
Hi,
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Do not load the system 32-bit ARM firmware VARS file because it must be writable to prevent a data exception and boot failure. So in order to use the VARS file, it must be copied to a writable location, but its quite large at 64M and is not needed to boot successfully.
You can load the VARS file with snapshot=on (and drop readonly=on) to make things work without copying the file.
Good to know. Do you know of any benefit to doing this (ie. using the installed VARS file)?
Well, the firmware expects it can write to VARS flash. ovmf has for historical reasons code to cope with non-flash or readonly VARS (so -bios works), but armvirt has not. Which is why you see the failures.
snapshot=on allows the guest write to a in-RAM throwaway snapshot, which is the same as using a writable temporary copy of the vars file, but without the need for a temporary file.
take care,
Gerd
Quoted 11 lines · On Tue, 24 Jan 2023 09:16:26 +0100 Gerd Hoffmann <[email protected]> wrote:
Hi,
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Do not load the system 32-bit ARM firmware VARS file because it must be writable to prevent a data exception and boot failure. So in order to use the VARS file, it must be copied to a writable location, but its quite large at 64M and is not needed to boot successfully.
You can load the VARS file with snapshot=on (and drop readonly=on) to make things work without copying the file.
Quoted 6 lines
Quoted 2 lines
Good to know. Do you know of any benefit to doing this (ie. using the installed VARS file)?
Well, the firmware expects it can write to VARS flash. ovmf has for historical reasons code to cope with non-flash or readonly VARS (so -bios works), but armvirt has not. Which is why you see the failures.
Well armvirt seems to cope fine if there's no VARS flash provided, it just dies when there is one provided which is readonly.
Quoted 3 lines
snapshot=on allows the guest write to a in-RAM throwaway snapshot, which is the same as using a writable temporary copy of the vars file, but without the need for a temporary file.
Yeah, I get that, but armvirt doesn't even need the VARS file to run. You'll see in this patch that for 32-bit arm, no VARS flash is created when loading the firmware code from the system path or when loading the firmware code from the grub source directory and the VARS file doesn't exist. So what benefit is there to providing the system VARS as a flash device? I can't think of a benefit to using the VARS file if the tests all pass fine without it.
I'd prefer to have as few dependencies on files not in the GRUB source as possible. So I like that I don't need to use the system VARS file on 32-bit arm. I wish I could do the same on the other platforms.
This patch does no temporary copying of VARS files on any platform and all the QEMU tests run as expected. So using "snapshot=on" won't prevent any copying, which is the main benefit that I see. I do appreciate this suggestion as it might be useful in other work with QEMU, I just don't see how its beneficial here.
Glenn
Quoted 12 lines · Glenn Washburn
Quoted 8 lines · On Mon, 23 Jan 2023 10:28:09 +0100 Gerd Hoffmann <[email protected]> wrote:
Quoted 4 lines · Glenn Washburn
The current qemu firmware paths for arm-efi and arm64-efi are not available on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware files in the source directory and if not found, look for them in locations where Debian installs them.
I'd suggest to inspect the *.json files in
/usr/share/qemu/firmware/to find distro-installed firmware files.
Quoted 8 lines
Yes, I know about this, but decided against it so as not to do real or hacked up json parsing and add another dependency (eg. jq). I think right now the unstated GRUB policy is that these tests are only officially supported to run on (newer) debian systems, so I felt that hard coding was reasonable. It occurs to me that its possible (though I suspect very improbable), that Redhat based distros run the tests when building the official RPM. Is there a reason that redhat would be very interested in the change you're suggesting?
We don't run the tests during builds. That has more to do with buildtimes and historical considerations than the tests themselves.
Personally, I would write this in such a way that we could, e.g., check the debian locations, and fallback to the fedora locations if they don't exist, and so forth for other distros. That said, grub has not been in favor of handling compatibility in this way in the past and seems to prefer pushing that burden onto the distros and their users.
Be well,
--Robbie
Quoted 24 lines · On Tue, 24 Jan 2023 16:21:34 -0500 Robbie Harwood <[email protected]> wrote:
Quoted 12 lines · Glenn Washburn
Quoted 8 lines · On Mon, 23 Jan 2023 10:28:09 +0100 Gerd Hoffmann <[email protected]> wrote:
Quoted 4 lines · Glenn Washburn
The current qemu firmware paths for arm-efi and arm64-efi are not available on Ubuntu/Debian but are hardcoded. Switch to first looking for firmware files in the source directory and if not found, look for them in locations where Debian installs them.
I'd suggest to inspect the *.json files in
/usr/share/qemu/firmware/to find distro-installed firmware files.Quoted 9 lines
Yes, I know about this, but decided against it so as not to do real or hacked up json parsing and add another dependency (eg. jq). I think right now the unstated GRUB policy is that these tests are only officially supported to run on (newer) debian systems, so I felt that hard coding was reasonable. It occurs to me that its possible (though I suspect very improbable), that Redhat based distros run the tests when building the official RPM. Is there a reason that redhat would be very interested in the change you're suggesting?
Quoted 2 lines
We don't run the tests during builds. That has more to do with buildtimes and historical considerations than the tests themselves.
I don't understand the last sentence, but I'm gathering that you guys don't ever run these tests.
Quoted 6 lines
Personally, I would write this in such a way that we could, e.g., check the debian locations, and fallback to the fedora locations if they don't exist, and so forth for other distros. That said, grub has not been in favor of handling compatibility in this way in the past and seems to prefer pushing that burden onto the distros and their users.
Yeah, I guess I don't personally care enough, even if its trivial. Is the point that it would make it easier for someone (not me) to send a patch adding Redhat support? If so, then they could add what your proposing in that patch. If other distros want to patch their source so that users getting the source ball can run the tests without having to manually change the paths, this current implementation makes that easy, just change the paths in a patch. I'm not completely opposed to the idea if someone else wants to do it, but I'm not personally interested in making it uglier and more complex than it already is. Despite this, I appreciate your input.
Glenn