edk2 build scripts: eliminate python 2 dependency
v2 cover edk2 build scripts: eliminate python 2 dependency
v1 URL:
http://mid.mail-archive.com/[email protected]
Updates are noted on each patch.
Thanks
Laszlo
Laszlo Ersek (2):
edk2 build scripts: honor external BaseTools flags with
uefi-test-tools
edk2 build scripts: work around TianoCore#1607 without forcing Python
2
roms/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/Makefile | 5 ++++- tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 28 insertions(+), 5 deletions(-)
Signature · 1 line
2.19.1.3.g30247aa5d201
PATCH v2 1/2 edk2 build scripts: honor external BaseTools flags with uefi-test-tools
Unify the recipe for "build-edk2-tools" in "tests/uefi-test-tools/Makefile" with the recipe for "edk2-basetools" in "roms/Makefile".
---
Notes:
v2:
- new patch
tests/uefi-test-tools/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tests/uefi-test-tools/Makefile +3 -1
@@ -99,7 +99,9 @@ Build/bios-tables-test.%.efi: build-edk2-tools +./build.sh $(edk2_dir) BiosTablesTest $* $@ build-edk2-tools: - $(MAKE) -C $(edk2_dir)/BaseTools + $(MAKE) -C $(edk2_dir)/BaseTools \ + EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ + EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)' clean: rm -rf Build Conf log
Signature · 1 line
2.19.1.3.g30247aa5d201
Re: [PATCH v2 1/2] edk2 build scripts: honor external BaseTools flags with uefi-test-tools
Quoted 21 lines · Laszlo Ersek
Unify the recipe for "build-edk2-tools" in "tests/uefi-test-tools/Makefile" with the recipe for "edk2-basetools" in "roms/Makefile".
Cc Philippe Mathieu-Daudé <[email protected]>Signed-off-by Laszlo Ersek <[email protected]>---
Notes:
v2:
- new patchtests/uefi-test-tools/Makefile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)tests/uefi-test-tools/Makefile +0
-0
tests/uefi-test-tools/Makefile Build/bios-tables-test.%.efi: build-edk2-tools · line 107
Quoted 10 lines
tests/uefi-test-tools/Makefile +3
-1@@ -99,7 +99,9 @@ Build/bios-tables-test.%.efi: build-edk2-tools +./build.sh $(edk2_dir) BiosTablesTest $* $@ build-edk2-tools: - $(MAKE) -C $(edk2_dir)/BaseTools + $(MAKE) -C $(edk2_dir)/BaseTools \ + EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ + EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)' clean: rm -rf Build Conf log
PATCH v2 2/2 edk2 build scripts: work around TianoCore#1607 without forcing Python 2
In reply to Laszlo ErsekIt turns out that forcing python2 for running the edk2 "build" utility is neither necessary nor sufficient.
Forcing python2 is not sufficient for two reasons:
- QEMU is moving away from python2, with python2 nearing EOL,
- according to my most recent testing, the lacking dependency information in the makefiles that are generated by edk2's "build" utility can cause parallel build failures even when "build" is executed by python2.
And forcing python2 is not necessary because we can still return to the original idea of filtering out jobserver-related options from MAKEFLAGS. So do that.
While at it, cut short edk2's auto-detection of the python3.* minor version, by setting PYTHON_COMMAND to "python3" (which we expect to be available wherever we intend to build edk2).
With this patch, the guest UEFI binaries that are used as part of the BIOS tables test, and the OVMF and ArmVirtQemu platform firmwares, will be built strictly in a single job, regardless of an outermost "-jN" make option. Alas, there appears to be no reliable way to build edk2 in an (outer make, inner make) environment, with a jobserver enabled.
---
Notes:
v2:
- set PYTHON_COMMAND to python3; update commit message [Phil]
- do not pick up any feedback tags from v1 (v2 is not a trivial update)
v1:
- Tested on RHEL7 (where the outer "make" sets the old-style "--jobserver-fds" flag) and on Fedora 29 (where the outer "make" sets the new-style "--jobserver-auth" flag).
- I've rebuilt all the edk2 binaries with this patch applied. Everything works fine. However, if you test this patch, you might notice that git reports all the build products as modified. That's because when using the python3 code in edk2 BaseTools, the generated makefiles differ greatly from the ones generated when running in python2 mode (e.g. due to different random seeds in python hashes / dictionaries). As a result, parts of the firmware volumes / firmware filesystems could appear in a different order than before. This is harmless, and doesn't necessitate checking in the rebuilt binaries.
roms/Makefile | 1 + tests/uefi-test-tools/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 25 insertions(+), 4 deletions(-)
roms/Makefile +1 -0
@@ -141,6 +141,7 @@ build-efi-roms: build-pxe-roms # edk2-basetools: $(MAKE) -C edk2/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'
tests/uefi-test-tools/Makefile +1 -0
@@ -100,6 +100,7 @@ Build/bios-tables-test.%.efi: build-edk2-tools build-edk2-tools: $(MAKE) -C $(edk2_dir)/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'
roms/edk2-build.sh +2 -2
@@ -27,8 +27,7 @@ shift $num_args cd edk2 -# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>. -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C
@@ -43,6 +42,7 @@ fi # any), for the edk2 "build" utility. source ../edk2-funcs.sh edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target"
roms/edk2-funcs.sh +17 -0
@@ -251,3 +251,20 @@ qemu_edk2_get_thread_count() printf '1\n' fi } + + +# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607> by +# filtering jobserver-related flags out of MAKEFLAGS. Print the result to the +# standard output. +# +# Parameters: +# $1: the value of the MAKEFLAGS variable +qemu_edk2_quirk_tianocore_1607() +{ + local makeflags="$1" + + printf %s "$makeflags" \ + | LC_ALL=C sed --regexp-extended \ + --expression='s/--jobserver-(auth|fds)=[0-9]+,[0-9]+//' \ + --expression='s/-j([0-9]+)?//' +}
tests/uefi-test-tools/build.sh +4 -2
@@ -29,8 +29,7 @@ export PACKAGES_PATH=$(realpath -- "$edk2_dir") export WORKSPACE=$PWD mkdir -p Conf -# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>. -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C
@@ -46,12 +45,15 @@ fi source "$edk2_dir/../edk2-funcs.sh" edk2_arch=$(qemu_edk2_get_arch "$emulation_target") edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") +edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target" # Build the UEFI binary mkdir -p log build \ --arch="$edk2_arch" \ + -n "$edk2_thread_count" \ --buildtarget=DEBUG \ --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ --tagname="$edk2_toolchain" \
Signature · 1 line
2.19.1.3.g30247aa5d201
Re: [PATCH v2 2/2] edk2 build scripts: work around TianoCore#1607 without forcing Python 2
Quoted 11 lines · Laszlo Ersek
It turns out that forcing python2 for running the edk2 "build" utility is neither necessary nor sufficient.
Forcing python2 is not sufficient for two reasons:
- QEMU is moving away from python2, with python2 nearing EOL,
- according to my most recent testing, the lacking dependency information in the makefiles that are generated by edk2's "build" utility can cause parallel build failures even when "build" is executed by python2.
Quoted 7 lines
And forcing python2 is not necessary because we can still return to the original idea of filtering out jobserver-related options from MAKEFLAGS. So do that.
While at it, cut short edk2's auto-detection of the python3.* minor version, by setting
PYTHON_COMMANDto "python3" (which we expect to be available wherever we intend to build edk2).
Yes! This fixes it :)
Quoted 135 lines
With this patch, the guest UEFI binaries that are used as part of the BIOS tables test, and the OVMF and ArmVirtQemu platform firmwares, will be built strictly in a single job, regardless of an outermost "-jN" make option. Alas, there appears to be no reliable way to build edk2 in an (outer make, inner make) environment, with a jobserver enabled.
Cc Eduardo Habkost <[email protected]>Cc John Snow <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Reported-by John Snow <[email protected]>Signed-off-by Laszlo Ersek <[email protected]>---
Notes:
v2:- set
PYTHON_COMMANDto python3; update commit message [Phil]- do not pick up any feedback tags from v1 (v2 is not a trivial update)
v1:
- Tested on RHEL7 (where the outer "make" sets the old-style "--jobserver-fds" flag) and on Fedora 29 (where the outer "make" sets the new-style "--jobserver-auth" flag).
- I've rebuilt all the edk2 binaries with this patch applied. Everything works fine. However, if you test this patch, you might notice that git reports all the build products as modified. That's because when using the python3 code in edk2 BaseTools, the generated makefiles differ greatly from the ones generated when running in python2 mode (e.g. due to different random seeds in python hashes / dictionaries). As a result, parts of the firmware volumes / firmware filesystems could appear in a different order than before. This is harmless, and doesn't necessitate checking in the rebuilt binaries.
roms/Makefile | 1 + tests/uefi-test-tools/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 25 insertions(+), 4 deletions(-)roms/Makefile +1
-0@@ -141,6 +141,7 @@ build-efi-roms: build-pxe-roms # edk2-basetools: $(MAKE) -C edk2/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'tests/uefi-test-tools/Makefile +1
-0@@ -100,6 +100,7 @@ Build/bios-tables-test.%.efi: build-edk2-tools build-edk2-tools: $(MAKE) -C $(edk2_dir)/BaseTools \ + PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'roms/edk2-build.sh +2
-2@@ -27,8 +27,7 @@ shift $num_args cd edk2 -# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>. -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C@@ -43,6 +42,7 @@ fi # any), for the edk2 "build" utility. source ../edk2-funcs.sh edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target"roms/edk2-funcs.sh +17
-0@@ -251,3 +251,20 @@ qemu_edk2_get_thread_count() printf '1\n' fi } + + +# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607> by +# filtering jobserver-related flags out of MAKEFLAGS. Print the result to the +# standard output. +# +# Parameters: +# $1: the value of the MAKEFLAGS variable +qemu_edk2_quirk_tianocore_1607() +{ + local makeflags="$1" + + printf %s "$makeflags" \ + | LC_ALL=C sed --regexp-extended \ + --expression='s/--jobserver-(auth|fds)=[0-9]+,[0-9]+//' \ + --expression='s/-j([0-9]+)?//' +}tests/uefi-test-tools/build.sh +3
-2@@ -29,8 +29,7 @@ export PACKAGES_PATH=$(realpath -- "$edk2_dir") export WORKSPACE=$PWD mkdir -p Conf -# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>. -export PYTHON_COMMAND=python2 +export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3} # Source "edksetup.sh" carefully. set +e +u +C@@ -46,12 +45,15 @@ fi source "$edk2_dir/../edk2-funcs.sh" edk2_arch=$(qemu_edk2_get_arch "$emulation_target") edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") +edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS")
tests/uefi-test-tools/build.sh fi · line 59
Quoted 10 lines
tests/uefi-test-tools/build.sh +1
-0qemu_edk2_set_cross_env "$emulation_target" # Build the UEFI binary mkdir -p log build \ --arch="$edk2_arch" \ + -n "$edk2_thread_count" \ --buildtarget=DEBUG \ --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ --tagname="$edk2_toolchain" \
Quoted 16 lines · Laszlo Ersek
v1 URL:
http://mid.mail-archive.com/[email protected]Updates are noted on each patch.
Cc Eduardo Habkost <[email protected]>Cc John Snow <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Thanks
LaszloLaszlo Ersek (2):
edk2 build scripts: honor external BaseTools flags with
uefi-test-tools
edk2 build scripts: work around TianoCore#1607 without forcing Python
Quoted 9 lines
2roms/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/Makefile | 5 ++++- tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 28 insertions(+), 5 deletions(-)
Thank you again!
I'm sure we'll find all of the rough spots during the rc process, so I am quite content for now.
Quoted 27 lines · John Snow
Quoted 16 lines · Laszlo Ersek
v1 URL:
http://mid.mail-archive.com/[email protected]Updates are noted on each patch.
Cc Eduardo Habkost <[email protected]>Cc John Snow <[email protected]>Cc Philippe Mathieu-Daudé <[email protected]>Thanks
LaszloLaszlo Ersek (2):
edk2 build scripts: honor external BaseTools flags with
uefi-test-tools
edk2 build scripts: work around TianoCore#1607 without forcing PythonQuoted 9 lines
2roms/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/Makefile | 5 ++++- tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 28 insertions(+), 5 deletions(-)
Quoted 7 lines
Thank you again!
I'm sure we'll find all of the rough spots during the rc process, so I am quite content for now.
Reviewed-by John Snow <[email protected]>
Thanks, queued on edk2-next.