Skip to contentSkip to conversation

Fix tarball builds of UEFI/EDK2 firmware

PATCH In review Michael Roth 3 participants 7 messages Sep 12, 2019 – Sep 24, 2019

lore
Share & tools
lore
#0 lore raw

cover Fix tarball builds of UEFI/EDK2 firmware

Bruce noticed that we cannot build make efi target from the v4.1.0 tarball. This is due to a failure on the part of the make-release script to pull in submodules nested under other submodules, as well as Makefile.edk2's assumptions about being in a git tree.

Suggestions for more robust solutions are definitely welcome, but for now this series takes what seems to be the most direct approach.

 roms/Makefile.edk2   | 7 ++++++-
 scripts/make-release | 8 ++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
#1 lore raw

PATCH 1/2 make-release: pull in edk2 submodules so we can build it from tarballs

The make efi target added by 536d2173 is built from the roms/edk2 submodule, which in turn relies on additional submodules nested under roms/edk2.

The make-release script currently only pulls in top-level submodules, so these nested submodules are missing in the resulting tarball.

We could try to address this situation more generally by recursively pulling in all submodules, but this doesn't necessarily ensure the end-result will build properly (this case also required other changes).

Additionally, due to the nature of submodules, we may not always have control over how these sorts of things are dealt with, so for now we continue to handle it on a case-by-case in the make-release script.

---
 scripts/make-release | 8 ++++++++
 1 file changed, 8 insertions(+)
scripts/make-release +8 -0
index b4af9c9e52..a2a8cda33c 100755
@@ -20,6 +20,14 @@ git checkout "v${version}"
 git submodule update --init
 (cd roms/seabios && git describe --tags --long --dirty > .version)
 (cd roms/skiboot && ./make_version.sh > .version)
+# Fetch edk2 submodule's submodules, since it won't have access to them via
+# the tarball later.
+#
+# A more uniform way to handle this sort of situation would be nice, but we
+# don't necessarily have much control over how a submodule handles its
+# submodule dependencies, so we continue to handle these on a case-by-case
+# basis for now.
+(cd roms/edk2 && git submodule update --init)
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
Signature · 1 line
2.17.1
#2 lore raw

PATCH 2/2 roms/Makefile.edk2: don't pull in submodules when building from tarball

In reply to Michael Roth

Currently the make efi target pulls submodules nested under the roms/edk2 submodule as dependencies. However, when we attempt to build from a tarball this fails since we are no longer in a git tree.

A preceding patch will pre-populate these submodules in the tarball, so assume this build dependency is only needed when building from a git tree.

---
 roms/Makefile.edk2 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
roms/Makefile.edk2 +6 -1
index c2f2ff59d5..33a074d3a4 100644
@@ -46,8 +46,13 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # files.
 .INTERMEDIATE: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd)
 
+# Fetch edk2 submodule's submodules. If it is not in a git tree, assume
+# we're building from a tarball and that they've already been fetched by
+# make-release/tarball scripts.
 submodules:
-	cd edk2 && git submodule update --init --force
+	if test -d edk2/.git; then \
+		cd edk2 && git submodule update --init --force; \
+	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
 # "tests/uefi-test-tools/Makefile".
Signature · 1 line
2.17.1
Reviewed-by on v1 2/2
#3 lore raw

Re: [Qemu-devel] [PATCH 2/2] roms/Makefile.edk2: don't pull in submodules when building from tarball

Quoted 27 lines · Michael Roth

Currently the make efi target pulls submodules nested under the roms/edk2 submodule as dependencies. However, when we attempt to build from a tarball this fails since we are no longer in a git tree.

A preceding patch will pre-populate these submodules in the tarball, so assume this build dependency is only needed when building from a git tree.

---
 roms/Makefile.edk2 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
roms/Makefile.edk2 +1 -0
index c2f2ff59d5..33a074d3a4 100644
@@ -46,8 +46,13 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # files.
 .INTERMEDIATE: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd)
 
+# Fetch edk2 submodule's submodules. If it is not in a git tree, assume

roms/Makefile.edk2 all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \ · line 58

Quoted 10 lines
roms/Makefile.edk2 +5 -1
+# we're building from a tarball and that they've already been fetched by
+# make-release/tarball scripts.
 submodules:
-	cd edk2 && git submodule update --init --force
+	if test -d edk2/.git; then \
+		cd edk2 && git submodule update --init --force; \
+	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
 # "tests/uefi-test-tools/Makefile".

Thank you,
Laszlo

#4 lore raw

Re: [Qemu-devel] [PATCH 2/2] roms/Makefile.edk2: don't pull in submodules when building from tarball

In reply to Michael Roth
Quoted 22 lines · Michael Roth

Currently the make efi target pulls submodules nested under the roms/edk2 submodule as dependencies. However, when we attempt to build from a tarball this fails since we are no longer in a git tree.

A preceding patch will pre-populate these submodules in the tarball, so assume this build dependency is only needed when building from a git tree.

---
 roms/Makefile.edk2 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
roms/Makefile.edk2 +0 -0
index c2f2ff59d5..33a074d3a4 100644

roms/Makefile.edk2 all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \ · line 51

Quoted 6 lines
roms/Makefile.edk2 +3 -0
@@ -46,8 +46,13 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # files.
 .INTERMEDIATE: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd)
 
+# Fetch edk2 submodule's submodules. If it is not in a git tree, assume
+# we're building from a tarball and that they've already been fetched by
+# make-release/tarball scripts.

Annoying, without using the make-release tool in a fresh clone, I get qemu/roms/edk2/CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf(-1):
error 000E: File/directory not found in workspace

I vaguely remember there was a thread about not using 'git submodule update --init --recursive' in the root directory but can't find it, any idea?

Quoted 9 lines
roms/Makefile.edk2 +3 -1
 submodules:
-	cd edk2 && git submodule update --init --force
+	if test -d edk2/.git; then \
+		cd edk2 && git submodule update --init --force; \
+	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
 # "tests/uefi-test-tools/Makefile".
#5 lore raw

Re: [Qemu-devel] [PATCH 2/2] roms/Makefile.edk2: don't pull in submodules when building from tarball

Quoted 34 lines · Philippe Mathieu-Daudé
Quoted 22 lines · Michael Roth

Currently the make efi target pulls submodules nested under the roms/edk2 submodule as dependencies. However, when we attempt to build from a tarball this fails since we are no longer in a git tree.

A preceding patch will pre-populate these submodules in the tarball, so assume this build dependency is only needed when building from a git tree.

---
 roms/Makefile.edk2 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
roms/Makefile.edk2 +0 -0
index c2f2ff59d5..33a074d3a4 100644

roms/Makefile.edk2 all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \ · line 51

Quoted 6 lines
roms/Makefile.edk2 +3 -0
@@ -46,8 +46,13 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # files.
 .INTERMEDIATE: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd)
 
+# Fetch edk2 submodule's submodules. If it is not in a git tree, assume
+# we're building from a tarball and that they've already been fetched by
+# make-release/tarball scripts.

Annoying, without using the make-release tool in a fresh clone, I get qemu/roms/edk2/CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf(-1):
error 000E: File/directory not found in workspace

Quoted 3 lines

I vaguely remember there was a thread about not using 'git submodule update --init --recursive' in the root directory but can't find it, any idea?

I think that was a point made in the first patch of this series -- the "--recursive" flag is unbounded, and it might pull in several unneeded submodules.

For example, openssl itself appears to have three submodules (boringssl, krb5, pyca-cryptography), and none of those is needed for building openssl the way edk2 consumes it.

I seem to remember that patch#1 in this series stated: we'd be handling the submodules on a case-by-case basis.

Thanks
Laszlo

Quoted 10 lines
Quoted 10 lines
roms/Makefile.edk2 +3 -1
 submodules:
-	cd edk2 && git submodule update --init --force
+	if test -d edk2/.git; then \
+		cd edk2 && git submodule update --init --force; \
+	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
 # "tests/uefi-test-tools/Makefile".
Reviewed-by on the v1 series
#6 lore raw
In reply to Michael Roth
Quoted 4 lines · Michael Roth

Bruce noticed that we cannot build make efi target from the v4.1.0 tarball. This is due to a failure on the part of the make-release script to pull in submodules nested under other submodules, as well as Makefile.edk2's assumptions about being in a git tree.

Hmm I'd expect distributions interested in building EDK2, build it properly tuned, outside of QEMU. I consider what is here as default config useful for testing, not for production use. I might be wrong.

Meanwhile, it is true 'make efi' should not fail.

Quoted 2 lines

Suggestions for more robust solutions are definitely welcome, but for now this series takes what seems to be the most direct approach.

This is fine:

I'll queue these patches in my edk2-next branch since I'm preparing a pull request.

Thanks,

Phil.