Skip to contentSkip to conversation

roms: Rename the EFIROM variable and let it be overridable

PATCH Superseded Philippe Mathieu-Daudé 3 participants 9 messages Apr 5, 2019 – Apr 10, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover roms: Rename the EFIROM variable and let it be overridable

Hi,

Two trivial fixes to avoid the latest EDK2 testing series to cause trouble to downstream distributions (in particular if they have PIE enforced).

Regards,

Phil.

Philippe Mathieu-Daudé (2):
roms: Rename the EFIROM variable to avoid clashing with iPXE
roms: Allow the EDK2_EFIROM variable to be overridden

 roms/Makefile | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/2 roms: Rename the EFIROM variable to avoid clashing with iPXE

The iPXE project already uses the EFIROM for a tool named 'efirom' which is not the Intel EfiRom used by the EDK2 project. To make the difference obvious, rename the variable.

This fixes a long standing issue which is now masked since commit f590a812c21 "roms: build the EfiRom utility from the roms/edk2 submodule" (but would reappear in the next commit):

  $ git checkout f590a812c21~
  $ make -C roms clean EFIROM=$(type -P EfiRom)
  make: Entering directory '/source/qemu/roms'
  [...]
  make -C ipxe/src veryclean
  make[1]: Entering directory '/source/qemu/roms/ipxe/src'
  rm -f bin{,-*}/*.* bin{,-*}/.certificate.* bin{,-*}/.certificates.* bin{,-*}/.private_key.* bin{,-*}/errors bin{,-*}/NIC ./util/zbin ./util/elf2efi32 ./util/elf2efi64 /usr/bin/EfiRom ./util/efifatbin ./util/iccfix ./util/einfo TAGS bin{,-*}/symtab
  rm: cannot remove '/usr/bin/EfiRom': Permission denied
  make[1]: *** [Makefile.housekeeping:1564: clean] Error 1
  make[1]: Leaving directory '/source/qemu/roms/ipxe/src'
  make: *** [Makefile:152: clean] Error 2
  make: Leaving directory '/source/qemu/roms'
---
 roms/Makefile | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
roms/Makefile +4 -4
index 78d5dd18c30..d28252dafdf 100644
@@ -47,7 +47,7 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 # We need that to combine multiple images (legacy bios,
 # efi ia32, efi x64) into a single rom binary.
 #
-EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
+EDK2_EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
 
 default:
 	@echo "nothing is build by default"
@@ -102,8 +102,8 @@ pxe-rom-%: build-pxe-roms
 
 efirom: $(patsubst %,efi-rom-%,$(pxerom_variants))
 
-efi-rom-%: build-pxe-roms build-efi-roms $(EFIROM)
-	$(EFIROM) -f "0x$(VID)" -i "0x$(DID)" -l 0x02 \
+efi-rom-%: build-pxe-roms build-efi-roms $(EDK2_EFIROM)
+	$(EDK2_EFIROM) -f "0x$(VID)" -i "0x$(DID)" -l 0x02 \
 		-b ipxe/src/bin/$(VID)$(DID).rom \
 		-ec ipxe/src/bin-i386-efi/$(VID)$(DID).efidrv \
 		-ec ipxe/src/bin-x86_64-efi/$(VID)$(DID).efidrv \
@@ -120,7 +120,7 @@ build-efi-roms: build-pxe-roms
 		$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \
 		$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))
 
-$(EFIROM):
+$(EDK2_EFIROM):
 	$(MAKE) -C edk2/BaseTools
 
 slof:
Signature · 1 line
2.20.1
Reviewed-by on v2 1/2
#2 lore raw

Re: [Qemu-devel] [PATCH for-4.0 v2 1/2] roms: Rename the EFIROM variable to avoid clashing with iPXE

Quoted 7 lines · Philippe Mathieu-Daudé

The iPXE project already uses the EFIROM for a tool named 'efirom' which is not the Intel EfiRom used by the EDK2 project. To make the difference obvious, rename the variable.

This fixes a long standing issue which is now masked since commit f590a812c21 "roms: build the EfiRom utility from the roms/edk2 submodule" (but would reappear in the next commit):

[*]

Quoted 46 lines
  $ git checkout f590a812c21~
  $ make -C roms clean EFIROM=$(type -P EfiRom)
  make: Entering directory '/source/qemu/roms'
  [...]
  make -C ipxe/src veryclean
  make[1]: Entering directory '/source/qemu/roms/ipxe/src'
  rm -f bin{,-*}/*.* bin{,-*}/.certificate.* bin{,-*}/.certificates.* bin{,-*}/.private_key.* bin{,-*}/errors bin{,-*}/NIC ./util/zbin ./util/elf2efi32 ./util/elf2efi64 /usr/bin/EfiRom ./util/efifatbin ./util/iccfix ./util/einfo TAGS bin{,-*}/symtab
  rm: cannot remove '/usr/bin/EfiRom': Permission denied
  make[1]: *** [Makefile.housekeeping:1564: clean] Error 1
  make[1]: Leaving directory '/source/qemu/roms/ipxe/src'
  make: *** [Makefile:152: clean] Error 2
  make: Leaving directory '/source/qemu/roms'
---
 roms/Makefile | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
roms/Makefile +3 -3
index 78d5dd18c30..d28252dafdf 100644
@@ -47,7 +47,7 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 # We need that to combine multiple images (legacy bios,
 # efi ia32, efi x64) into a single rom binary.
 #
-EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
+EDK2_EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
 
 default:
 	@echo "nothing is build by default"
@@ -102,8 +102,8 @@ pxe-rom-%: build-pxe-roms
 
 efirom: $(patsubst %,efi-rom-%,$(pxerom_variants))
 
-efi-rom-%: build-pxe-roms build-efi-roms $(EFIROM)
-	$(EFIROM) -f "0x$(VID)" -i "0x$(DID)" -l 0x02 \
+efi-rom-%: build-pxe-roms build-efi-roms $(EDK2_EFIROM)
+	$(EDK2_EFIROM) -f "0x$(VID)" -i "0x$(DID)" -l 0x02 \
 		-b ipxe/src/bin/$(VID)$(DID).rom \
 		-ec ipxe/src/bin-i386-efi/$(VID)$(DID).efidrv \
 		-ec ipxe/src/bin-x86_64-efi/$(VID)$(DID).efidrv \

roms/Makefile build-efi-roms: build-pxe-roms · line 126

Quoted 8 lines
roms/Makefile +1 -1
@@ -120,7 +120,7 @@ build-efi-roms: build-pxe-roms
 		$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \
 		$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))
 
-$(EFIROM):
+$(EDK2_EFIROM):
 	$(MAKE) -C edk2/BaseTools
 
 slof:

Upon reading your and Olaf's comments in the v1 thread:

[PATCH for-4.0] roms: Allow the EFIROM variable to be overridden

I've changed my mind on this series.

Please insert a paragraph at [*] in the commit message:

"iPXE's 'veryclean' recipe removes $(EFIROM) even if the EFIROM macro originates from elsewhere."

With that:

(I'll comment under patch#2 separately.)

Thank you for picking this up.
Laszlo

#3 lore raw

PATCH v2 2/2 roms: Allow the EDK2_EFIROM variable to be overridden

In reply to Philippe Mathieu-Daudé

Since commit f590a812c210 we build the EDK2 EfiRom utility unconditionally. This has been tested on all the Linux distribution providing continuous integration (namely Debian and Fedora). Not all distributions are able to build the EfiRom without specific patches (In particular SUSE which enforces the PIE protection, see [*]).

Restore the possibility to other distributions to override the EDK2_EFIROM variable.

[*] https://lists.opensuse.org/opensuse-factory/2017-06/msg00403.html

---
 roms/Makefile | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
roms/Makefile +8 -1
index d28252dafdf..ea19aa9b33c 100644
@@ -47,7 +47,11 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 # We need that to combine multiple images (legacy bios,
 # efi ia32, efi x64) into a single rom binary.
 #
-EDK2_EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
+# By default we build the latest EDK2 stable EfiRom utility.
+# If you have to use another one, you can also pass the location on
+# the command line, i.e. "make EDK2_EFIROM=$(type -P EfiRom) efirom"
+#
+EDK2_EFIROM ?= edk2/BaseTools/Source/C/bin/EfiRom
 
 default:
 	@echo "nothing is build by default"
@@ -120,8 +124,11 @@ build-efi-roms: build-pxe-roms
 		$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \
 		$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))
 
+# Do not compile $(EDK2_EFIROM) if the variable is overridden
+ifeq "$(origin EDK2_EFIROM)" "file"
 $(EDK2_EFIROM):
 	$(MAKE) -C edk2/BaseTools
+endif
 
 slof:
 	$(MAKE) -C SLOF CROSS=$(powerpc64_cross_prefix) qemu
Signature · 1 line
2.20.1
#4 lore raw

Re: [Qemu-devel] [PATCH for-4.0 v2 2/2] roms: Allow the EDK2_EFIROM variable to be overridden

Quoted 38 lines · Philippe Mathieu-Daudé

Since commit f590a812c210 we build the EDK2 EfiRom utility unconditionally. This has been tested on all the Linux distribution providing continuous integration (namely Debian and Fedora). Not all distributions are able to build the EfiRom without specific patches (In particular SUSE which enforces the PIE protection, see [*]).

Restore the possibility to other distributions to override the EDK2_EFIROM variable.

[*] https://lists.opensuse.org/opensuse-factory/2017-06/msg00403.html

---
 roms/Makefile | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
roms/Makefile +5 -1
index d28252dafdf..ea19aa9b33c 100644
@@ -47,7 +47,11 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 # We need that to combine multiple images (legacy bios,
 # efi ia32, efi x64) into a single rom binary.
 #
-EDK2_EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
+# By default we build the latest EDK2 stable EfiRom utility.
+# If you have to use another one, you can also pass the location on
+# the command line, i.e. "make EDK2_EFIROM=$(type -P EfiRom) efirom"
+#
+EDK2_EFIROM ?= edk2/BaseTools/Source/C/bin/EfiRom
 
 default:
 	@echo "nothing is build by default"
@@ -120,8 +124,11 @@ build-efi-roms: build-pxe-roms
 		$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \

roms/Makefile build-efi-roms: build-pxe-roms · line 134

Quoted 10 lines
roms/Makefile +3 -0
 		$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))
 
+# Do not compile $(EDK2_EFIROM) if the variable is overridden
+ifeq "$(origin EDK2_EFIROM)" "file"
 $(EDK2_EFIROM):
 	$(MAKE) -C edk2/BaseTools
+endif
 
 slof:
 	$(MAKE) -C SLOF CROSS=$(powerpc64_cross_prefix) qemu

I agree with the problem statement, from Olaf's message in the

[PATCH for-4.0] roms: Allow the EFIROM variable to be overridden

thread: "roms/ seems to be unable to receive configure options".

That's the issue in need of a fix.

The present patch would break what was fixed by f590a812c210. In other words, we should stick with using a single EfiRom for all purposes (of roms/Makefile), but we should let the caller easily inject compiler & linker options.

So I think the recipe should do something like:

	$(MAKE) -C edk2/BaseTools \
		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
		EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'

Then build scripts could be updated to invoke:

  make -C roms \
    EDK2_BASETOOLS_OPTFLAGS='...' \
    EDK2_BASETOOLS_LDFLAGS='...' \
    efirom

Thanks,
Laszlo

#5 lore raw

Re: [Qemu-devel] [PATCH for-4.0 v2 2/2] roms: Allow the EDK2_EFIROM variable to be overridden

Am Mon, 8 Apr 2019 13:05:07 +0200
schrieb Laszlo Ersek <[email protected]>:

Quoted 6 lines

Then build scripts could be updated to invoke:

  make -C roms \
    EDK2_BASETOOLS_OPTFLAGS='...' \
    EDK2_BASETOOLS_LDFLAGS='...' \
    efirom

The question remains: 'But why?'.
Why can edk2 not be built with '-fno-pie' right away?
Did noone approach the edk2 developers yet that their buildsystem is broken in that regard?

Olaf

#6 lore raw

Re: [Qemu-devel] [PATCH for-4.0 v2 2/2] roms: Allow the EDK2_EFIROM variable to be overridden

Quoted 11 lines · Olaf Hering

Am Mon, 8 Apr 2019 13:05:07 +0200
schrieb Laszlo Ersek <[email protected]>:

Quoted 6 lines

Then build scripts could be updated to invoke:

  make -C roms \
    EDK2_BASETOOLS_OPTFLAGS='...' \
    EDK2_BASETOOLS_LDFLAGS='...' \
    efirom

The question remains: 'But why?'.

Because it lets you pass "-fno-pie" & friends.

Quoted 1 line

Why can edk2 not be built with '-fno-pie' right away?

Those flags are not universal across gcc/binutils versions.

Some gcc/binutils versions don't enable PIE to begin with.

Some gcc/binutils versions take different PIE-disablement flags (considering both the compiler and the linker) from other gcc/binutils versions.

For example, please refer to the following *iPXE* commit:

Quoted 10 lines

commit 7c395b0e21806b946fe944a27fc273407f357ea1

Author: Michael Brown <[email protected]>
Date:   Wed Jun 14 12:33:16 2017 +0100

[build] Use -no-pie on newer versions of gcc

Some distributions patch gcc to generate position independent executables by default. We currently include a workaround to check for this and to add -fno-PIE -nopie to CFLAGS if required.

Quoted 6 lines

Newer patched versions of gcc require -fno-PIE -no-pie instead. Check for both variants.

(I remember this commit because the logic it had added actually failed on a system that I used once for building iPXE -- it was an x86_64 system with both a native gcc, and a *different version* x86_64-to-x86_64 *cross* gcc. The selection logic determined the flags for the one compiler toolchain, but then passed the flags to the other compiler toolchain. The build broke. I narrowed it down to the above commit, and then shrugged it off; it wasn't important enough to spend more time on it.)

I also refer you to the following *edk2* commit:

Quoted 13 lines

commit 11d0cd23dd1bc15a6e6a1598250ea2e0c4c36e9a

Author: Ard Biesheuvel <[email protected]>
Date:   Mon Jun 18 10:23:49 2018 +0200

BaseTools/tools_def IA32: drop -no-pie linker option for GCC49

As reported by Liming, GCC 4.9.2 does not support the -no-pie linker option that we added to the GCC49 and GCC5 toolchain profiles in commit c25d3905523a ("BaseTools/tools_def IA32:
disable PIE code generation explicitly") to work around issues with recent distro toolchains that enable PIE code generation by default.

Quoted 5 lines

So rollback the changes for GCC49 but preserve them for GCC5

Contributed-under: TianoCore Contribution Agreement 1.1

The above facility will let you stuff the options into the edk2 BaseTools build, in your downstream qemu.spec file, that match your downstream gcc/binutils version.

Most build hosts will need no flags.

Quoted 2 lines

Did noone approach the edk2 developers yet that their buildsystem is broken in that regard?

Well, have you?

I don't remember anyone else reporting such an issue yet, on edk2-devel or elsewhere, with building BaseTools. I certainly remember collaborating with Gary Lin from SUSE on various stuff, but not this.

You are welcome to file a bug at <https://bugzilla.tianocore.org/>.
Please pick "EDK2" for "Product", "Code" for "Component", and "BaseTools" for "Package".

... Please don't try to force a "zero build-interface changes" policy on upstream, just so you can avoid a small tweak in a downstream build script when you rebase. Not even the gcc/binutils command lines conform to that. We all hope downstream rebases to be painless, and upstreams do strive to help with that, but the downstream rebase experience is never *entirely* painless (or automated).

Thanks,
Laszlo

#7 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 5 lines · Philippe Mathieu-Daudé

Hi,

Two trivial fixes to avoid the latest EDK2 testing series to cause trouble to downstream distributions (in particular if they have PIE enforced).

I disgree with this.

(1) In the first commit message, you say,

"The iPXE project already uses the EFIROM for a tool named 'efirom' which is not the Intel EfiRom used by the EDK2 project".

That's wrong. For building the combined (UEFI+BIOS) iPXE oprom binaries, the efirom tool that is invoked is *most definitely* the tool from edk2.

What changes is that we now build efirom directly from the edk2 submodule, rather than using a binary that could possibly be found on a GNU/Linux system from another package.

This is entirely aligned with the addition of the edk2 submodule. The source for the efirom tool is now directly available, so in a *maintainer* build -- i.e., when you decide to rebuild iPXE binaries -- we should certainly prefer to build everything from source.

Again, this is a *maintainer* build (which also covers downstream package builds), not end-user build. If you decide to rebuild artifacts that are otherwise offered in binary form to end-users, you commit to building everything from source that goes into (or is necessary for) producing those artifacts.

In the thread "edk2 fails to compile in v4.0.0-rc2", Olaf wrote,

"It is in ovmf-tools.rpm, which comes from ovmf."

That only proves my point.

(2) For a while now, it has been possible for downstream build scripts to inject their preferred build flags into the BaseTools build recipes (makefiles) themselves. Please see
<https://bugzilla.redhat.com/show_bug.cgi?id=1540244>. This is the relevant upstream commit list:

1 67983484a443 BaseTools/footer.makefile: expand BUILD_CFLAGS last
for C files too
2 03252ae287c4 BaseTools/header.makefile: remove "-c" from
BUILD_CFLAGS
3 b8a661702643 BaseTools/Source/C: split "-O2" to BUILD_OPTFLAGS 4 b0ca5dae78ff BaseTools/Source/C: take EXTRA_OPTFLAGS from the
caller
5 81502cee20ac BaseTools/Source/C: take EXTRA_LDFLAGS from the caller 6 aa4e0df1f0c7 BaseTools/VfrCompile: honor EXTRA_LDFLAGS

Build BaseTools as follows:

make -C "$EDK_TOOLS_PATH" EXTRA_OPTFLAGS="..." EXTRA_LDFLAGS="..."

If you need to inject PIC/PIE-related flags into the BaseTools compilation/linking, please use the above facility.

I think it's pretty usual that new upstream releases (of any open source project) bring some changes for downstream packaging scripts.

Thanks
Laszlo

#8 lore raw
Quoted 32 lines · Laszlo Ersek
Quoted 5 lines · Philippe Mathieu-Daudé

Hi,

Two trivial fixes to avoid the latest EDK2 testing series to cause trouble to downstream distributions (in particular if they have PIE enforced).

I disgree with this.

(1) In the first commit message, you say,

"The iPXE project already uses the EFIROM for a tool named 'efirom' which is not the Intel EfiRom used by the EDK2 project".

That's wrong. For building the combined (UEFI+BIOS) iPXE oprom binaries, the efirom tool that is invoked is *most definitely* the tool from edk2.

What changes is that we now build efirom directly from the edk2 submodule, rather than using a binary that could possibly be found on a GNU/Linux system from another package.

This is entirely aligned with the addition of the edk2 submodule. The source for the efirom tool is now directly available, so in a *maintainer* build -- i.e., when you decide to rebuild iPXE binaries -- we should certainly prefer to build everything from source.

Again, this is a *maintainer* build (which also covers downstream package builds), not end-user build. If you decide to rebuild artifacts that are otherwise offered in binary form to end-users, you commit to building everything from source that goes into (or is necessary for) producing those artifacts.

Quoted 5 lines

In the thread "edk2 fails to compile in v4.0.0-rc2", Olaf wrote,

"It is in ovmf-tools.rpm, which comes from ovmf."

That only proves my point.

Of course, ipxe.git itself has a macro called EFIROM:

src/Makefile:EFIROM := ./util/efirom

but that has *zero* relevance for QEMU commit f590a812c210 ("roms: build the EfiRom utility from the roms/edk2 submodule", 2019-02-21).

Because, the "efirom" utility from inside ipxe.git is not used for anything at all in the *QEMU recipe* for building the combined iPXE option ROMs.

Look at the diff in f590a812c210:

-EFIROM ?= $(shell which EfiRom 2>/dev/null)
+EFIROM = edk2/BaseTools/Source/C/bin/EfiRom

If the way we build the combined iPXE oproms, from QEMU's ipxe.git submodule, using "roms/Makefile", had *anything* to do with iPXE's own "efirom", then the pre-patch state would have been wrong *already*.

Laszlo

Quoted 29 lines

(2) For a while now, it has been possible for downstream build scripts to inject their preferred build flags into the BaseTools build recipes (makefiles) themselves. Please see
<https://bugzilla.redhat.com/show_bug.cgi?id=1540244>. This is the relevant upstream commit list:

1 67983484a443 BaseTools/footer.makefile: expand BUILD_CFLAGS last
for C files too
2 03252ae287c4 BaseTools/header.makefile: remove "-c" from
BUILD_CFLAGS
3 b8a661702643 BaseTools/Source/C: split "-O2" to BUILD_OPTFLAGS 4 b0ca5dae78ff BaseTools/Source/C: take EXTRA_OPTFLAGS from the
caller
5 81502cee20ac BaseTools/Source/C: take EXTRA_LDFLAGS from the caller 6 aa4e0df1f0c7 BaseTools/VfrCompile: honor EXTRA_LDFLAGS

Build BaseTools as follows:

make -C "$EDK_TOOLS_PATH" EXTRA_OPTFLAGS="..." EXTRA_LDFLAGS="..."

If you need to inject PIC/PIE-related flags into the BaseTools compilation/linking, please use the above facility.

I think it's pretty usual that new upstream releases (of any open source project) bring some changes for downstream packaging scripts.

Thanks
Laszlo