Skip to contentSkip to conversation

add the BiosTablesTest UEFI app, build it with the new roms/edk2 submodule

PATCH Superseded Laszlo Ersek 3 participants 19 messages Jan 24, 2019 – Feb 3, 2019 Kbuild

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover add the BiosTablesTest UEFI app, build it with the new roms/edk2 submodule

Previous version (v1):
http://mid.mail-archive.com/[email protected]

Updates in v2 have been noted on each patch in the series.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

Thanks
Laszlo

Laszlo Ersek (5):
roms: add the edk2 project as a git submodule
roms: build the EfiRom utility from the roms/edk2 submodule
tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app tests/uefi-test-tools: add build scripts
tests/data: introduce "uefi-boot-images" with the "bios-tables-test"
ISOs

 .gitmodules                                                              |   3 +
 Makefile                                                                 |   6 +-
 roms/Makefile                                                            |  13 +-
 roms/edk2                                                                |   1 +
 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2           | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2               | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2              | Bin 0 -> 12800 bytes
 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2            | Bin 0 -> 13312 bytes
 tests/uefi-test-tools/.gitignore                                         |   3 +
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 tests/uefi-test-tools/Makefile                                           |  97 +++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 +++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 ++++++++++
 tests/uefi-test-tools/build.sh                                           | 145 ++++++++++++++++++++
 17 files changed, 619 insertions(+), 8 deletions(-)
 create mode 160000 roms/edk2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2
 create mode 100644 tests/uefi-test-tools/.gitignore
 create mode 100644 tests/uefi-test-tools/LICENSE
 create mode 100644 tests/uefi-test-tools/Makefile
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc
 create mode 100755 tests/uefi-test-tools/build.sh
Signature · 1 line
2.19.1.3.g30247aa5d201
#1 lore raw

PATCH v2 1/5 roms: add the edk2 project as a git submodule

The roms/edk2 submodule can help with three goals:
- build the OVMF and ArmVirtQemu virtual UEFI firmware platforms (to be implemented later),
- build the EfiRom tool on the fly, which is used in roms/Makefile, for building the "efirom" target,
- build UEFI test applications (to be run in guests), for qtest support.

Edk2 commit 85588389222a3636baf0f9ed8227f2434af4c3f9 stands for the latest "stable tag", namely "edk2-stable201811".

The edk2 repository tracks some binary files that should not be removed by QEMU's top-level "make clean"; exempt the full pathnames from the "find" command.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- pick up R-b [Gerd]

 Makefile    | 6 +++++-
 .gitmodules | 3 +++
 roms/edk2   | 1 +
 3 files changed, 9 insertions(+), 1 deletion(-)
Makefile +5 -1
index de898eab6234..b0a70b2929ad 100644
@@ -604,7 +604,11 @@ clean:
 	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
 	rm -f qemu-options.def
 	rm -f *.msi
-	find . \( -name '*.so' -o -name '*.dll' -o -name '*.mo' -o -name '*.[oda]' \) -type f -exec rm {} +
+	find . \( -name '*.so' -o -name '*.dll' -o -name '*.mo' -o -name '*.[oda]' \) -type f \
+		! -path ./roms/edk2/ArmPkg/Library/GccLto/liblto-aarch64.a \
+		! -path ./roms/edk2/ArmPkg/Library/GccLto/liblto-arm.a \
+		! -path ./roms/edk2/BaseTools/Source/Python/UPT/Dll/sqlite3.dll \
+		-exec rm {} +
 	rm -f $(filter-out %.tlb,$(TOOLS)) $(HELPERS-y) qemu-ga TAGS cscope.* *.pod *~ */*~
 	rm -f fsdev/*.pod scsi/*.pod
 	rm -f qemu-img-cmds.h
.gitmodules +3 -0
index 6b91176098c8..ceafb0ee29a0 100644
@@ -49,3 +49,6 @@
 [submodule "tests/fp/berkeley-softfloat-3"]
 	path = tests/fp/berkeley-softfloat-3
 	url = https://github.com/cota/berkeley-softfloat-3
+[submodule "roms/edk2"]
+	path = roms/edk2
+	url = https://github.com/tianocore/edk2.git
roms/edk2 new file +1 -0
new file mode 160000
index 000000000000..85588389222a
@@ -0,0 +1 @@
+Subproject commit 85588389222a3636baf0f9ed8227f2434af4c3f9
Signature · 1 line
2.19.1.3.g30247aa5d201
Reviewed-by Tested-by on v2 1/5
#2 lore raw

Re: [Qemu-devel] [PATCH v2 1/5] roms: add the edk2 project as a git submodule

Quoted 14 lines · Laszlo Ersek

The roms/edk2 submodule can help with three goals:
- build the OVMF and ArmVirtQemu virtual UEFI firmware platforms (to be implemented later),
- build the EfiRom tool on the fly, which is used in roms/Makefile, for building the "efirom" target,
- build UEFI test applications (to be run in guests), for qtest support.

Edk2 commit 85588389222a3636baf0f9ed8227f2434af4c3f9 stands for the latest "stable tag", namely "edk2-stable201811".

The edk2 repository tracks some binary files that should not be removed by QEMU's top-level "make clean"; exempt the full pathnames from the "find" command.

Quoted 8 lines
Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>
Quoted 47 lines

---

Notes:
v2:
- pick up R-b [Gerd]

 Makefile    | 6 +++++-
 .gitmodules | 3 +++
 roms/edk2   | 1 +
 3 files changed, 9 insertions(+), 1 deletion(-)
Makefile +5 -1
index de898eab6234..b0a70b2929ad 100644
@@ -604,7 +604,11 @@ clean:
 	rm -f config.mak op-i386.h opc-i386.h gen-op-i386.h op-arm.h opc-arm.h gen-op-arm.h
 	rm -f qemu-options.def
 	rm -f *.msi
-	find . \( -name '*.so' -o -name '*.dll' -o -name '*.mo' -o -name '*.[oda]' \) -type f -exec rm {} +
+	find . \( -name '*.so' -o -name '*.dll' -o -name '*.mo' -o -name '*.[oda]' \) -type f \
+		! -path ./roms/edk2/ArmPkg/Library/GccLto/liblto-aarch64.a \
+		! -path ./roms/edk2/ArmPkg/Library/GccLto/liblto-arm.a \
+		! -path ./roms/edk2/BaseTools/Source/Python/UPT/Dll/sqlite3.dll \
+		-exec rm {} +
 	rm -f $(filter-out %.tlb,$(TOOLS)) $(HELPERS-y) qemu-ga TAGS cscope.* *.pod *~ */*~
 	rm -f fsdev/*.pod scsi/*.pod
 	rm -f qemu-img-cmds.h
.gitmodules +3 -0
index 6b91176098c8..ceafb0ee29a0 100644
@@ -49,3 +49,6 @@
 [submodule "tests/fp/berkeley-softfloat-3"]
 	path = tests/fp/berkeley-softfloat-3
 	url = https://github.com/cota/berkeley-softfloat-3
+[submodule "roms/edk2"]
+	path = roms/edk2
+	url = https://github.com/tianocore/edk2.git
roms/edk2 new file +1 -0
new file mode 160000
index 000000000000..85588389222a
@@ -0,0 +1 @@
+Subproject commit 85588389222a3636baf0f9ed8227f2434af4c3f9
#3 lore raw

PATCH v2 2/5 roms: build the EfiRom utility from the roms/edk2 submodule

In reply to Laszlo Ersek

Building the EfiRom utility from "roms/edk2/BaseTools" should make "roms/Makefile" more self-contained. Otherwise, we'd call the system-wide EfiRom for building the combined iPXE option ROMs, but call the sibling utilities from "roms/edk2/BaseTools" for building "roms/edk2" content.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- pick up R-b / T-b [Gerd, Phil]

 roms/Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
roms/Makefile +6 -7
index a6043eff37e9..78d5dd18c301 100644
@@ -47,10 +47,7 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 # We need that to combine multiple images (legacy bios,
 # efi ia32, efi x64) into a single rom binary.
 #
-# We try to find it in the path.  You can also pass the location on
-# the command line, i.e. "make EFIROM=/path/to/EfiRom efirom"
-#
-EFIROM ?= $(shell which EfiRom 2>/dev/null)
+EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
 
 default:
 	@echo "nothing is build by default"
@@ -59,8 +56,7 @@ default:
 	@echo "  vgabios        -- update vgabios binaries (seabios)"
 	@echo "  sgabios        -- update sgabios binaries"
 	@echo "  pxerom         -- update nic roms (bios only)"
-	@echo "  efirom         -- update nic roms (bios+efi, this needs"
-	@echo "                    the EfiRom utility from edk2 / tianocore)"
+	@echo "  efirom         -- update nic roms (bios+efi)"
 	@echo "  slof           -- update slof.bin"
 	@echo "  skiboot        -- update skiboot.lid"
 	@echo "  u-boot.e500    -- update u-boot.e500"
@@ -106,7 +102,7 @@ pxe-rom-%: build-pxe-roms
 
 efirom: $(patsubst %,efi-rom-%,$(pxerom_variants))
 
-efi-rom-%: build-pxe-roms build-efi-roms
+efi-rom-%: build-pxe-roms build-efi-roms $(EFIROM)
 	$(EFIROM) -f "0x$(VID)" -i "0x$(DID)" -l 0x02 \
 		-b ipxe/src/bin/$(VID)$(DID).rom \
 		-ec ipxe/src/bin-i386-efi/$(VID)$(DID).efidrv \
@@ -124,6 +120,8 @@ build-efi-roms: build-pxe-roms
 		$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \
 		$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))
 
+$(EFIROM):
+	$(MAKE) -C edk2/BaseTools
 
 slof:
 	$(MAKE) -C SLOF CROSS=$(powerpc64_cross_prefix) qemu
@@ -150,6 +148,7 @@ clean:
 	$(MAKE) -C sgabios clean
 	rm -f sgabios/.depend
 	$(MAKE) -C ipxe/src veryclean
+	$(MAKE) -C edk2/BaseTools clean
 	$(MAKE) -C SLOF clean
 	rm -rf u-boot/build.e500
 	$(MAKE) -C u-boot-sam460ex distclean
Signature · 1 line
2.19.1.3.g30247aa5d201
#4 lore raw

PATCH v2 3/5 tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app

In reply to Laszlo Ersek

The "bios-tables-test" program in QEMU's test suite locates the RSD PTR ACPI table in guest RAM, and (chasing pointers to other ACPI tables) performs various sanity checks on the QEMU-generated and firmware-installed tables.

Currently this set of test cases doesn't work with UEFI guests. The ACPI spec defines distinct methods for OSPM to locate the RSD PTR on traditional BIOS vs. UEFI platforms, and the UEFI method is more difficult to implement from the hypervisor side with just raw guest memory access.

Add a UEFI application (to be booted in the UEFI guest) that populates a small, MB-aligned structure in guest RAM. The structure begins with a signature GUID. The hypervisor should loop over all MB-aligned pages in guest RAM until one matches the signature GUID at offset 0, at which point the hypervisor can fetch the RSDP address field(s) from the structure.

QEMU's test logic currently spins on a pre-determined guest address, until that address assumes a magic value. The method described in this patch is conceptually the same ("busy loop until match is found"), except there is no hard-coded address. This plays a lot more nicely with UEFI guest firmware (we'll be able to use the normal page allocation UEFI service). Given the size of EFI_GUID (16 bytes -- 128 bits), mismatches should be astronomically unlikely. In addition, given the typical guest RAM size for such tests (128 MB), there are 128 locations to check in one iteration of the "outer" loop, which shouldn't introduce an intolerable delay after the guest stores the RSDP address(es), and then the GUID.

The GUID that the hypervisor should search for is

AB87A6B1-2034-BDA0-71BD-375007757785

Expressed as a byte array:

{
0xb1, 0xa6, 0x87, 0xab,
0x34, 0x20,
0xa0, 0xbd,
0x71, 0xbd, 0x37, 0x50, 0x07, 0x75, 0x77, 0x85
}

Note that in the patch, we define "gBiosTablesTestGuid" with all bits inverted. This is a simple method to prevent the UEFI binary, which incorporates "gBiosTablesTestGuid", from matching the actual GUID in guest RAM.

The UEFI application is written against the edk2 framework, which was introduced earlier as a git submodule. The next patch will provide build scripts for maintainers.

The source code follows the edk2 coding style, and is licensed under the 2-clause BSDL (in case someone would like to include UefiTestToolsPkg content in a different edk2 platform).

The "UefiTestToolsPkg.dsc" platform description file resolves the used edk2 library classes to instances (= library implementations) such that the UEFI binaries inherit no platform dependencies. They are expected to run on any system that conforms to the UEFI-2.3.1 spec (which was released in 2012). The arch-specific build options are carried over from edk2's ArmVirtPkg and OvmfPkg platforms.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- no change

v1:
- If that's necessary, I'd be glad to be designated as Maintainer or Reviewer in "MAINTAINERS" for "tests/uefi-test-tools/", I just couldn't figure out under what subsystem I should add the magic lines. "MAINTAINERS" needs a Table of Contents! :)

 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 +++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 ++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++++
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 6 files changed, 359 insertions(+)
tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec new file +27 -0
new file mode 100644
index 000000000000..ed3a2fe11084
@@ -0,0 +1,27 @@
+## @file
+# edk2 package declaration for the test helper UEFI applications that run in
+# guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+##
+
+[Defines]
+  DEC_SPECIFICATION = 1.27
+  PACKAGE_NAME      = UefiTestToolsPkg
+  PACKAGE_GUID      = 7b3f1794-0c85-4b27-a536-44dbf0b0669c
+  PACKAGE_VERSION   = 0.1
+
+[Includes]
+  Include
+
+[Guids]
+  gBiosTablesTestGuid = {0x5478594e, 0xdfcb, 0x425f, {0x8e, 0x42, 0xc8, 0xaf, 0xf8, 0x8a, 0x88, 0x7a}}
+
tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc new file +69 -0

69 lines not on this page · raw message

tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf new file +41 -0

41 lines not on this page · raw message

tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h new file +67 -0

67 lines not on this page · raw message

tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c new file +130 -0

130 lines not on this page · raw message

tests/uefi-test-tools/LICENSE new file +25 -0
new file mode 100644
index 000000000000..38b78aecdb0e
@@ -0,0 +1,25 @@
+All the files in this directory and subdirectories are released under the
+2-Clause BSD License (see header in each file).
+
+Copyright (C) 2019, Red Hat, Inc.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Signature · 1 line
2.19.1.3.g30247aa5d201
Reviewed-by on v2 3/5
#5 lore raw

Re: [Qemu-devel] [PATCH v2 3/5] tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app

Quoted 60 lines · Laszlo Ersek

The "bios-tables-test" program in QEMU's test suite locates the RSD PTR ACPI table in guest RAM, and (chasing pointers to other ACPI tables) performs various sanity checks on the QEMU-generated and firmware-installed tables.

Currently this set of test cases doesn't work with UEFI guests. The ACPI spec defines distinct methods for OSPM to locate the RSD PTR on traditional BIOS vs. UEFI platforms, and the UEFI method is more difficult to implement from the hypervisor side with just raw guest memory access.

Add a UEFI application (to be booted in the UEFI guest) that populates a small, MB-aligned structure in guest RAM. The structure begins with a signature GUID. The hypervisor should loop over all MB-aligned pages in guest RAM until one matches the signature GUID at offset 0, at which point the hypervisor can fetch the RSDP address field(s) from the structure.

QEMU's test logic currently spins on a pre-determined guest address, until that address assumes a magic value. The method described in this patch is conceptually the same ("busy loop until match is found"), except there is no hard-coded address. This plays a lot more nicely with UEFI guest firmware (we'll be able to use the normal page allocation UEFI service). Given the size of EFI_GUID (16 bytes -- 128 bits), mismatches should be astronomically unlikely. In addition, given the typical guest RAM size for such tests (128 MB), there are 128 locations to check in one iteration of the "outer" loop, which shouldn't introduce an intolerable delay after the guest stores the RSDP address(es), and then the GUID.

The GUID that the hypervisor should search for is

AB87A6B1-2034-BDA0-71BD-375007757785

Expressed as a byte array:

 {
   0xb1, 0xa6, 0x87, 0xab,
   0x34, 0x20,
   0xa0, 0xbd,
   0x71, 0xbd, 0x37, 0x50, 0x07, 0x75, 0x77, 0x85

}

Note that in the patch, we define "gBiosTablesTestGuid" with all bits inverted. This is a simple method to prevent the UEFI binary, which incorporates "gBiosTablesTestGuid", from matching the actual GUID in guest RAM.

The UEFI application is written against the edk2 framework, which was introduced earlier as a git submodule. The next patch will provide build scripts for maintainers.

The source code follows the edk2 coding style, and is licensed under the 2-clause BSDL (in case someone would like to include UefiTestToolsPkg content in a different edk2 platform).

The "UefiTestToolsPkg.dsc" platform description file resolves the used edk2 library classes to instances (= library implementations) such that the UEFI binaries inherit no platform dependencies. They are expected to run on any system that conforms to the UEFI-2.3.1 spec (which was released in 2012). The arch-specific build options are carried over from edk2's ArmVirtPkg and OvmfPkg platforms.

Quoted 7 lines
Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>
Quoted 416 lines

---

Notes:
v2:
- no change

v1:
- If that's necessary, I'd be glad to be designated as Maintainer or Reviewer in "MAINTAINERS" for "tests/uefi-test-tools/", I just couldn't figure out under what subsystem I should add the magic lines. "MAINTAINERS" needs a Table of Contents! :)

 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 +++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 ++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++++
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 6 files changed, 359 insertions(+)
tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec new file +27 -0
new file mode 100644
index 000000000000..ed3a2fe11084
@@ -0,0 +1,27 @@
+## @file
+# edk2 package declaration for the test helper UEFI applications that run in
+# guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+##
+
+[Defines]
+  DEC_SPECIFICATION = 1.27
+  PACKAGE_NAME      = UefiTestToolsPkg
+  PACKAGE_GUID      = 7b3f1794-0c85-4b27-a536-44dbf0b0669c
+  PACKAGE_VERSION   = 0.1
+
+[Includes]
+  Include
+
+[Guids]
+  gBiosTablesTestGuid = {0x5478594e, 0xdfcb, 0x425f, {0x8e, 0x42, 0xc8, 0xaf, 0xf8, 0x8a, 0x88, 0x7a}}
+
tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc new file +69 -0
new file mode 100644
index 000000000000..c8511cd732bc
@@ -0,0 +1,69 @@
+## @file
+# edk2 platform description for the test helper UEFI applications that run in
+# guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+##
+
+[Defines]
+  DSC_SPECIFICATION       = 1.28
+  PLATFORM_GUID           = 6750ccc1-8365-49f0-8437-948e516a9f55
+  PLATFORM_VERSION        = 0.1
+  PLATFORM_NAME           = UefiTestTools
+  SKUID_IDENTIFIER        = DEFAULT
+  SUPPORTED_ARCHITECTURES = ARM|AARCH64|IA32|X64
+  BUILD_TARGETS           = DEBUG
+
+[BuildOptions.IA32]
+  GCC:*_*_IA32_CC_FLAGS = -mno-mmx -mno-sse
+
+[BuildOptions.X64]
+  GCC:*_*_X64_CC_FLAGS = -mno-mmx -mno-sse
+
+[BuildOptions.ARM.EDKII.UEFI_APPLICATION]
+  GCC:*_*_ARM_DLINK_FLAGS = -z common-page-size=0x1000
+
+[BuildOptions.AARCH64.EDKII.UEFI_APPLICATION]
+  GCC:*_*_AARCH64_DLINK_FLAGS = -z common-page-size=0x1000
+
+[BuildOptions]
+  GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES
+
+[SkuIds]
+  0|DEFAULT
+
+[LibraryClasses]
+  BaseLib|MdePkg/Library/BaseLib/BaseLib.inf
+  DebugLib|MdePkg/Library/UefiDebugLibConOut/UefiDebugLibConOut.inf
+  DebugPrintErrorLevelLib|MdePkg/Library/BaseDebugPrintErrorLevelLib/BaseDebugPrintErrorLevelLib.inf
+  DevicePathLib|MdePkg/Library/UefiDevicePathLibDevicePathProtocol/UefiDevicePathLibDevicePathProtocol.inf
+  MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
+  PcdLib|MdePkg/Library/BasePcdLibNull/BasePcdLibNull.inf
+  PrintLib|MdePkg/Library/BasePrintLib/BasePrintLib.inf
+  UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
+  UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
+  UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
+  UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf
+
+[LibraryClasses.ARM, LibraryClasses.AARCH64]
+  BaseMemoryLib|MdePkg/Library/BaseMemoryLibOptDxe/BaseMemoryLibOptDxe.inf
+  NULL|ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib.inf
+  NULL|MdePkg/Library/BaseStackCheckLib/BaseStackCheckLib.inf
+
+[LibraryClasses.IA32, LibraryClasses.X64]
+  BaseMemoryLib|MdePkg/Library/BaseMemoryLibRepStr/BaseMemoryLibRepStr.inf
+
+[PcdsFixedAtBuild]
+  gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F
+  gEfiMdePkgTokenSpaceGuid.PcdDebugPropertyMask|0x2F
+
+[Components]
+  UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf
tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf new file +41 -0
new file mode 100644
index 000000000000..924d8a80d00b
@@ -0,0 +1,41 @@
+## @file
+# Populate the BIOS_TABLES_TEST structure.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+##
+
+[Defines]
+  INF_VERSION                = 1.27
+  BASE_NAME                  = BiosTablesTest
+  UEFI_SPECIFICATION_VERSION = 2.31
+  FILE_GUID                  = 87f00433-3b7c-45c3-ae78-a56495bd4e62
+  MODULE_TYPE                = UEFI_APPLICATION
+  ENTRY_POINT                = BiosTablesTestMain
+
+[Sources]
+  BiosTablesTest.c
+
+[LibraryClasses]
+  BaseLib
+  BaseMemoryLib
+  MemoryAllocationLib
+  UefiApplicationEntryPoint
+  UefiBootServicesTableLib
+  UefiLib
+
+[Guids]
+  gBiosTablesTestGuid
+  gEfiAcpi10TableGuid
+  gEfiAcpi20TableGuid
+
+[Packages]
+  MdePkg/MdePkg.dec
+  UefiTestToolsPkg/UefiTestToolsPkg.dec
tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h new file +67 -0
new file mode 100644
index 000000000000..0b72c61254af
@@ -0,0 +1,67 @@
+/** @file
+  Expose the address(es) of the ACPI RSD PTR table(s) in a MB-aligned structure
+  to the hypervisor.
+
+  The hypervisor locates the MB-aligned structure based on the signature GUID
+  that is at offset 0 in the structure. Once the RSD PTR address(es) are
+  retrieved, the hypervisor may perform various ACPI checks.
+
+  This feature is a development aid, for supporting ACPI table unit tests in
+  hypervisors. Do not enable in production builds.
+
+  Copyright (C) 2019, Red Hat, Inc.
+
+  This program and the accompanying materials are licensed and made available
+  under the terms and conditions of the BSD License that accompanies this
+  distribution. The full text of the license may be found at
+  <http://opensource.org/licenses/bsd-license.php>.
+
+  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+  WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+**/
+
+#ifndef __BIOS_TABLES_TEST_H__
+#define __BIOS_TABLES_TEST_H__
+
+#include <Uefi/UefiBaseType.h>
+
+#define BIOS_TABLES_TEST_GUID                          \
+  {                                                    \
+    0x5478594e,                                        \
+    0xdfcb,                                            \
+    0x425f,                                            \
+    { 0x8e, 0x42, 0xc8, 0xaf, 0xf8, 0x8a, 0x88, 0x7a } \
+  }
+
+extern EFI_GUID gBiosTablesTestGuid;
+
+//
+// The following structure must be allocated in Boot Services Data type memory,
+// aligned at a 1MB boundary.
+//
+#pragma pack (1)
+typedef struct {
+  //
+  // The signature GUID is written to the MB-aligned structure from
+  // gBiosTablesTestGuid, but with all bits inverted. That's the actual GUID
+  // value that the hypervisor should look for at each MB boundary, looping
+  // over all guest RAM pages with that alignment, until a match is found. The
+  // bit-flipping occurs in order not to store the actual GUID in any UEFI
+  // executable, which might confuse guest memory analysis. Note that EFI_GUID
+  // has little endian representation.
+  //
+  EFI_GUID             InverseSignatureGuid;
+  //
+  // The Rsdp10 and Rsdp20 fields may be read when the signature GUID matches.
+  // Rsdp10 is the guest-physical address of the ACPI 1.0 specification RSD PTR
+  // table, in 8-byte little endian representation. Rsdp20 is the same, for the
+  // ACPI 2.0 or later specification RSD PTR table. Each of these fields may be
+  // zero (independently of the other) if the UEFI System Table does not
+  // provide the corresponding UEFI Configuration Table.
+  //
+  EFI_PHYSICAL_ADDRESS Rsdp10;
+  EFI_PHYSICAL_ADDRESS Rsdp20;
+} BIOS_TABLES_TEST;
+#pragma pack ()
+
+#endif // __BIOS_TABLES_TEST_H__
tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c new file +130 -0
new file mode 100644
index 000000000000..b208e17fb00f
@@ -0,0 +1,130 @@
+/** @file
+  Populate the BIOS_TABLES_TEST structure.
+
+  Copyright (C) 2019, Red Hat, Inc.
+
+  This program and the accompanying materials are licensed and made available
+  under the terms and conditions of the BSD License that accompanies this
+  distribution. The full text of the license may be found at
+  <http://opensource.org/licenses/bsd-license.php>.
+
+  THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+  WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+**/
+
+#include <Guid/Acpi.h>
+#include <Guid/BiosTablesTest.h>
+#include <Library/BaseLib.h>
+#include <Library/BaseMemoryLib.h>
+#include <Library/MemoryAllocationLib.h>
+#include <Library/UefiBootServicesTableLib.h>
+#include <Library/UefiLib.h>
+
+/**
+  Wait for a keypress with a message that the application is about to exit.
+**/
+STATIC
+VOID
+WaitForExitKeyPress (
+  VOID
+  )
+{
+  EFI_STATUS    Status;
+  UINTN         Idx;
+  EFI_INPUT_KEY Key;
+
+  if (gST->ConIn == NULL) {
+    return;
+  }
+  AsciiPrint ("%a: press any key to exit\n", gEfiCallerBaseName);
+  Status = gBS->WaitForEvent (1, &gST->ConIn->WaitForKey, &Idx);
+  if (EFI_ERROR (Status)) {
+    return;
+  }
+  gST->ConIn->ReadKeyStroke (gST->ConIn, &Key);
+}
+
+EFI_STATUS
+EFIAPI
+BiosTablesTestMain (
+  IN EFI_HANDLE       ImageHandle,
+  IN EFI_SYSTEM_TABLE *SystemTable
+  )
+{
+  VOID                          *Pages;
+  volatile BIOS_TABLES_TEST     *BiosTablesTest;
+  CONST VOID                    *Rsdp10;
+  CONST VOID                    *Rsdp20;
+  CONST EFI_CONFIGURATION_TABLE *ConfigTable;
+  CONST EFI_CONFIGURATION_TABLE *ConfigTablesEnd;
+  volatile EFI_GUID             *InverseSignature;
+  UINTN                         Idx;
+
+  Pages = AllocateAlignedPages (EFI_SIZE_TO_PAGES (sizeof *BiosTablesTest),
+            SIZE_1MB);
+  if (Pages == NULL) {
+    AsciiErrorPrint ("%a: AllocateAlignedPages() failed\n",
+      gEfiCallerBaseName);
+    //
+    // Assuming the application was launched by the boot manager as a boot
+    // loader, exiting with error will cause the boot manager to proceed with
+    // the remaining boot options. If there are no other boot options, the boot
+    // manager menu will be pulled up. Give the user a chance to read the error
+    // message.
+    //
+    WaitForExitKeyPress ();
+    return EFI_OUT_OF_RESOURCES;
+  }
+
+  //
+  // Locate both gEfiAcpi10TableGuid and gEfiAcpi20TableGuid config tables in
+  // one go.
+  //
+  Rsdp10 = NULL;
+  Rsdp20 = NULL;
+  ConfigTable = gST->ConfigurationTable;
+  ConfigTablesEnd = gST->ConfigurationTable + gST->NumberOfTableEntries;
+  while ((Rsdp10 == NULL || Rsdp20 == NULL) && ConfigTable < ConfigTablesEnd) {
+    if (CompareGuid (&ConfigTable->VendorGuid, &gEfiAcpi10TableGuid)) {
+      Rsdp10 = ConfigTable->VendorTable;
+    } else if (CompareGuid (&ConfigTable->VendorGuid, &gEfiAcpi20TableGuid)) {
+      Rsdp20 = ConfigTable->VendorTable;
+    }
+    ++ConfigTable;
+  }
+
+  AsciiPrint ("%a: BiosTablesTest=%p Rsdp10=%p Rsdp20=%p\n",
+    gEfiCallerBaseName, Pages, Rsdp10, Rsdp20);
+
+  //
+  // Store the RSD PTR address(es) first, then the signature second.
+  //
+  BiosTablesTest = Pages;
+  BiosTablesTest->Rsdp10 = (UINTN)Rsdp10;
+  BiosTablesTest->Rsdp20 = (UINTN)Rsdp20;
+
+  MemoryFence();
+
+  InverseSignature = &BiosTablesTest->InverseSignatureGuid;
+  InverseSignature->Data1  = gBiosTablesTestGuid.Data1;
+  InverseSignature->Data1 ^= MAX_UINT32;
+  InverseSignature->Data2  = gBiosTablesTestGuid.Data2;
+  InverseSignature->Data2 ^= MAX_UINT16;
+  InverseSignature->Data3  = gBiosTablesTestGuid.Data3;
+  InverseSignature->Data3 ^= MAX_UINT16;
+  for (Idx = 0; Idx < sizeof InverseSignature->Data4; ++Idx) {
+    InverseSignature->Data4[Idx]  = gBiosTablesTestGuid.Data4[Idx];
+    InverseSignature->Data4[Idx] ^= MAX_UINT8;
+  }
+
+  //
+  // The wait below has dual purpose. First, it blocks the application without
+  // wasting VCPU cycles while the hypervisor is scanning guest RAM. Second,
+  // assuming the application was launched by the boot manager as a boot
+  // loader, exiting the app with success causes the boot manager to pull up
+  // the boot manager menu at once (regardless of other boot options); the wait
+  // gives the user a chance to read the info printed above.
+  //
+  WaitForExitKeyPress ();
+  return EFI_SUCCESS;
+}
tests/uefi-test-tools/LICENSE new file +25 -0
new file mode 100644
index 000000000000..38b78aecdb0e
@@ -0,0 +1,25 @@
+All the files in this directory and subdirectories are released under the
+2-Clause BSD License (see header in each file).
+
+Copyright (C) 2019, Red Hat, Inc.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#6 lore raw

PATCH v2 4/5 tests/uefi-test-tools: add build scripts

In reply to Laszlo Ersek

Introduce the following build scripts under "tests/uefi-test-tools":

* "build.sh" builds a single module (a UEFI application) from UefiTestToolsPkg, for a single QEMU emulation target.

"build.sh" relies on cross-compilers when the emulation target and the build host architecture don't match. The cross-compiler prefix is computed according to a fixed, Linux-specific pattern. No attempt is made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" for cross-compiler prefix determination. The reason is that the build host OSes that are officially supported by edk2, and those that are supported by QEMU, intersect only in Linux. (Note that the UNIXGCC toolchain is being removed from edk2,
<https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.)

* "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" application, for arm, aarch64, i386, and x86_64, with the help of "build.sh".

"Makefile" turns each resultant UEFI executable into a UEFI-bootable, qcow2-compressed ISO image. The ISO images are output as "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2".

Each ISO image should be passed to QEMU as follows:

    -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \
    -device virtio-scsi-pci,id=scsi0 \
    -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \

"Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are present.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- add the .NOTPARALLEL target [Phil, help-make, edk2-devel]

 tests/uefi-test-tools/Makefile   |  97 +++++++++++++
 tests/uefi-test-tools/.gitignore |   3 +
 tests/uefi-test-tools/build.sh   | 145 ++++++++++++++++++++
 3 files changed, 245 insertions(+)
tests/uefi-test-tools/Makefile new file +97 -0
new file mode 100644
index 000000000000..61d263861e01
@@ -0,0 +1,97 @@
+# Makefile for the test helper UEFI applications that run in guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+edk2_dir              := ../../roms/edk2
+images_dir            := ../data/uefi-boot-images
+emulation_targets     := arm aarch64 i386 x86_64
+uefi_binaries         := bios-tables-test
+intermediate_suffixes := .efi .fat .iso.raw
+
+images: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(images_dir)/$(binary).$(target).iso.qcow2))
+
+# Preserve all intermediate targets if the build succeeds.
+# - Intermediate targets help with development & debugging.
+# - Preserving intermediate targets also keeps spurious changes out of the
+#   final build products, in case the user re-runs "make" without any changes
+#   to the UEFI source code. Normally, the intermediate files would have been
+#   removed by the last "make" invocation, hence the re-run would rebuild them
+#   from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and
+#   "genisoimage" utilities embed timestamp-based information in their outputs,
+#   which causes git to report differences for the tracked qcow2 ISO images.
+.SECONDARY: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(foreach suffix,$(intermediate_suffixes), \
+				Build/$(binary).$(target)$(suffix))))
+
+# In the pattern rules below, the stem (%, $*) stands for
+# "$(binary).$(target)".
+
+# Convert the raw ISO image to a qcow2 one, enabling compression, and using a
+# small cluster size. This allows for small binary files under git control,
+# hence for small binary patches.
+$(images_dir)/%.iso.qcow2: Build/%.iso.raw
+	mkdir -p -- $(images_dir)
+	$${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \
+		-o cluster_size=512 -- $< $@
+
+# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito
+# boot image.
+Build/%.iso.raw: Build/%.fat
+	genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \
+		-quiet -o $@ -- $<
+
+# Define chained macros in order to map QEMU system emulation targets to
+# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately
+# stripped from, the argument.
+map_arm_to_uefi     = $(subst arm,ARM,$(1))
+map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1)))
+map_i386_to_uefi    = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1)))
+map_x86_64_to_uefi  = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1)))
+map_to_uefi         = $(subst .,,$(call map_x86_64_to_uefi,$(1)))
+
+# Format a "UEFI system partition", using the UEFI binary as the default boot
+# loader. Add 10% size for filesystem metadata, round up to the next KB, and
+# make sure the size is large enough for a FAT filesystem. Name the filesystem
+# after the UEFI binary. (Excess characters are automatically dropped from the
+# filesystem label.)
+Build/%.fat: Build/%.efi
+	rm -f -- $@
+	uefi_bin_b=$$(stat --format=%s -- $<) && \
+		uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \
+		uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \
+		mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT
+	MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \
+		::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI
+
+# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The
+# association between the UEFI binary (such as "bios-tables-test") and the
+# component name from the edk2 platform DSC file (such as "BiosTablesTest") is
+# explicit in each rule.
+
+# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2
+# workspace, at most one "build" instance may be operating at a time. Therefore
+# we must serialize the rebuilding of targets in this Makefile.
+.NOTPARALLEL:
+
+Build/bios-tables-test.%.efi: build-edk2-tools
+	./build.sh $(edk2_dir) BiosTablesTest $* $@
+
+build-edk2-tools:
+	$(MAKE) -C $(edk2_dir)/BaseTools
+
+clean:
+	rm -rf Build Conf log
+	$(MAKE) -C $(edk2_dir)/BaseTools clean
tests/uefi-test-tools/.gitignore new file +3 -0
new file mode 100644
index 000000000000..9f246701dea1
@@ -0,0 +1,3 @@
+Build
+Conf
+log
tests/uefi-test-tools/build.sh new file +145 -0

145 lines not on this page · raw message

Signature · 1 line
2.19.1.3.g30247aa5d201
#7 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Hi Laszlo,

Quoted 133 lines · Laszlo Ersek

Introduce the following build scripts under "tests/uefi-test-tools":

* "build.sh" builds a single module (a UEFI application) from UefiTestToolsPkg, for a single QEMU emulation target.

"build.sh" relies on cross-compilers when the emulation target and the build host architecture don't match. The cross-compiler prefix is computed according to a fixed, Linux-specific pattern. No attempt is made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" for cross-compiler prefix determination. The reason is that the build host OSes that are officially supported by edk2, and those that are supported by QEMU, intersect only in Linux. (Note that the UNIXGCC toolchain is being removed from edk2,
<https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.)

* "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" application, for arm, aarch64, i386, and x86_64, with the help of "build.sh".

"Makefile" turns each resultant UEFI executable into a UEFI-bootable, qcow2-compressed ISO image. The ISO images are output as "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2".

Each ISO image should be passed to QEMU as follows:

    -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \
    -device virtio-scsi-pci,id=scsi0 \
    -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \

"Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are present.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- add the .NOTPARALLEL target [Phil, help-make, edk2-devel]

 tests/uefi-test-tools/Makefile   |  97 +++++++++++++
 tests/uefi-test-tools/.gitignore |   3 +
 tests/uefi-test-tools/build.sh   | 145 ++++++++++++++++++++
 3 files changed, 245 insertions(+)
tests/uefi-test-tools/Makefile new file +77 -0
new file mode 100644
index 000000000000..61d263861e01
@@ -0,0 +1,97 @@
+# Makefile for the test helper UEFI applications that run in guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+edk2_dir              := ../../roms/edk2
+images_dir            := ../data/uefi-boot-images
+emulation_targets     := arm aarch64 i386 x86_64
+uefi_binaries         := bios-tables-test
+intermediate_suffixes := .efi .fat .iso.raw
+
+images: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(images_dir)/$(binary).$(target).iso.qcow2))
+
+# Preserve all intermediate targets if the build succeeds.
+# - Intermediate targets help with development & debugging.
+# - Preserving intermediate targets also keeps spurious changes out of the
+#   final build products, in case the user re-runs "make" without any changes
+#   to the UEFI source code. Normally, the intermediate files would have been
+#   removed by the last "make" invocation, hence the re-run would rebuild them
+#   from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and
+#   "genisoimage" utilities embed timestamp-based information in their outputs,
+#   which causes git to report differences for the tracked qcow2 ISO images.
+.SECONDARY: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(foreach suffix,$(intermediate_suffixes), \
+				Build/$(binary).$(target)$(suffix))))
+
+# In the pattern rules below, the stem (%, $*) stands for
+# "$(binary).$(target)".
+
+# Convert the raw ISO image to a qcow2 one, enabling compression, and using a
+# small cluster size. This allows for small binary files under git control,
+# hence for small binary patches.
+$(images_dir)/%.iso.qcow2: Build/%.iso.raw
+	mkdir -p -- $(images_dir)
+	$${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \
+		-o cluster_size=512 -- $< $@
+
+# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito
+# boot image.
+Build/%.iso.raw: Build/%.fat
+	genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \
+		-quiet -o $@ -- $<
+
+# Define chained macros in order to map QEMU system emulation targets to
+# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately
+# stripped from, the argument.
+map_arm_to_uefi     = $(subst arm,ARM,$(1))
+map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1)))
+map_i386_to_uefi    = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1)))
+map_x86_64_to_uefi  = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1)))
+map_to_uefi         = $(subst .,,$(call map_x86_64_to_uefi,$(1)))
+
+# Format a "UEFI system partition", using the UEFI binary as the default boot
+# loader. Add 10% size for filesystem metadata, round up to the next KB, and
+# make sure the size is large enough for a FAT filesystem. Name the filesystem
+# after the UEFI binary. (Excess characters are automatically dropped from the
+# filesystem label.)
+Build/%.fat: Build/%.efi
+	rm -f -- $@
+	uefi_bin_b=$$(stat --format=%s -- $<) && \
+		uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \
+		uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \
+		mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT
+	MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \
+		::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI

tests/uefi-test-tools/Makefile line 87

Quoted 10 lines
tests/uefi-test-tools/Makefile new file +10 -0
+
+# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The
+# association between the UEFI binary (such as "bios-tables-test") and the
+# component name from the edk2 platform DSC file (such as "BiosTablesTest") is
+# explicit in each rule.
+
+# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2
+# workspace, at most one "build" instance may be operating at a time. Therefore
+# we must serialize the rebuilding of targets in this Makefile.
+.NOTPARALLEL:

Well this doesn't seem to improve my case :|

You can test with:

$ alias make='make -j8 -l7.5'

So you get:

$ make print-MAKEFLAGS
MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4

and this command fails:

$ make -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

Now, using -j1 we have:

$ make print-MAKEFLAGS -j1
MAKEFLAGS=rR -j1 -l7.5

And the builds work:

$ make -j1 -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

Reading 'info make' section "5.7.3 Communicating Options to a Sub-'make'" I figured we can overwrite MAKEFLAGS, and this snippet works like charm:

-- >8 --
@@ -84,8 +84,7 @@ Build/%.fat: Build/%.efi
# "build.sh" invokes the "build" utility of edk2 BaseTools. In any
given edk2
# workspace, at most one "build" instance may be operating at a time.
Therefore

 # we must serialize the rebuilding of targets in this Makefile.
-.NOTPARALLEL:
-
+Build/bios-tables-test.%.efi: MAKEFLAGS=
 Build/bios-tables-test.%.efi: build-edk2-tools
        ./build.sh $(edk2_dir) BiosTablesTest $* $@

---

I tested with:

$ rm -rf tests/uefi-test-tools/Build
$ make -C tests/uefi-test-tools \
  Build/bios-tables-test.{arm,aarch64,i386,x86_64}.efi images -j42

What do you think about using this snippet? I don't know if this is the best way to solve this, but it works...
If you test/agree, Michael could eventually apply the snippet directly.

Quoted 171 lines
tests/uefi-test-tools/Makefile new file +10 -0
+
+Build/bios-tables-test.%.efi: build-edk2-tools
+	./build.sh $(edk2_dir) BiosTablesTest $* $@
+
+build-edk2-tools:
+	$(MAKE) -C $(edk2_dir)/BaseTools
+
+clean:
+	rm -rf Build Conf log
+	$(MAKE) -C $(edk2_dir)/BaseTools clean
tests/uefi-test-tools/.gitignore new file +3 -0
new file mode 100644
index 000000000000..9f246701dea1
@@ -0,0 +1,3 @@
+Build
+Conf
+log
tests/uefi-test-tools/build.sh new file +145 -0
new file mode 100755
index 000000000000..155cb75c4ddb
@@ -0,0 +1,145 @@
+#!/bin/bash
+
+# Build script that determines the edk2 toolchain to use, invokes the edk2
+# "build" utility, and copies the built UEFI binary to the requested location.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+set -e -u -C
+
+# Save the command line arguments. We need to reset $# to 0 before sourcing
+# "edksetup.sh", as it will inherit $@.
+program_name=$(basename -- "$0")
+edk2_dir=$1
+dsc_component=$2
+emulation_target=$3
+uefi_binary=$4
+shift 4
+
+# Set up the environment for edk2 building.
+export PACKAGES_PATH=$(realpath -- "$edk2_dir")
+export WORKSPACE=$PWD
+mkdir -p Conf
+
+# Source "edksetup.sh" carefully.
+set +e +u +C
+source "$PACKAGES_PATH/edksetup.sh"
+ret=$?
+set -e -u -C
+if [ $ret -ne 0 ]; then
+  exit $ret
+fi
+
+# Map the QEMU system emulation target to the following types of architecture
+# identifiers:
+# - edk2,
+# - gcc cross-compilation.
+# Cover only those targets that are supported by the UEFI spec and edk2.
+case "$emulation_target" in
+  (arm)
+    edk2_arch=ARM
+    gcc_arch=arm
+    ;;
+  (aarch64)
+    edk2_arch=AARCH64
+    gcc_arch=aarch64
+    ;;
+  (i386)
+    edk2_arch=IA32
+    gcc_arch=i686
+    ;;
+  (x86_64)
+    edk2_arch=X64
+    gcc_arch=x86_64
+    ;;
+  (*)
+    printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \
+      "$program_name" "$emulation_target" >&2
+    exit 1
+    ;;
+esac
+
+# Check if cross-compilation is needed.
+host_arch=$(uname -m)
+if [ "$gcc_arch" == "$host_arch" ] ||
+   ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
+  cross_prefix=
+else
+  cross_prefix=${gcc_arch}-linux-gnu-
+fi
+
+# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it,
+# determine the suitable edk2 toolchain as well.
+# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers
+#   the gcc-5+ releases.
+# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in
+#   addition to GCC5. Unfortunately, the mapping between the toolchain tags and
+#   the actual gcc releases isn't entirely trivial. Run "git-blame" on
+#   "OvmfPkg/build.sh" in edk2 for more information.
+# And, because the above is too simple, we have to assign cross_prefix to an
+# edk2 build variable that is specific to both the toolchain tag and the target
+# architecture.
+case "$edk2_arch" in
+  (ARM)
+    edk2_toolchain=GCC5
+    export GCC5_ARM_PREFIX=$cross_prefix
+    ;;
+  (AARCH64)
+    edk2_toolchain=GCC5
+    export GCC5_AARCH64_PREFIX=$cross_prefix
+    ;;
+  (IA32|X64)
+    gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}')
+    case "$gcc_version" in
+      ([1-3].*|4.[0-3].*)
+        printf '%s: unsupported gcc version "%s"\n' \
+          "$program_name" "$gcc_version" >&2
+        exit 1
+        ;;
+      (4.4.*)
+        edk2_toolchain=GCC44
+        ;;
+      (4.5.*)
+        edk2_toolchain=GCC45
+        ;;
+      (4.6.*)
+        edk2_toolchain=GCC46
+        ;;
+      (4.7.*)
+        edk2_toolchain=GCC47
+        ;;
+      (4.8.*)
+        edk2_toolchain=GCC48
+        ;;
+      (4.9.*|6.[0-2].*)
+        edk2_toolchain=GCC49
+        ;;
+      (*)
+        edk2_toolchain=GCC5
+        ;;
+    esac
+    eval "export ${edk2_toolchain}_BIN=\$cross_prefix"
+    ;;
+esac
+
+# Build the UEFI binary
+mkdir -p log
+build \
+  --arch="$edk2_arch" \
+  --buildtarget=DEBUG \
+  --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \
+  --tagname="$edk2_toolchain" \
+  --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \
+  --log="log/$dsc_component.$edk2_arch.log" \
+  --report-file="log/$dsc_component.$edk2_arch.report"
+cp -a -- \
+  "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \
+  "$uefi_binary"
#8 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Quoted 149 lines · Philippe Mathieu-Daudé

Hi Laszlo,

Quoted 133 lines · Laszlo Ersek

Introduce the following build scripts under "tests/uefi-test-tools":

* "build.sh" builds a single module (a UEFI application) from UefiTestToolsPkg, for a single QEMU emulation target.

"build.sh" relies on cross-compilers when the emulation target and the build host architecture don't match. The cross-compiler prefix is computed according to a fixed, Linux-specific pattern. No attempt is made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" for cross-compiler prefix determination. The reason is that the build host OSes that are officially supported by edk2, and those that are supported by QEMU, intersect only in Linux. (Note that the UNIXGCC toolchain is being removed from edk2,
<https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.)

* "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" application, for arm, aarch64, i386, and x86_64, with the help of "build.sh".

"Makefile" turns each resultant UEFI executable into a UEFI-bootable, qcow2-compressed ISO image. The ISO images are output as "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2".

Each ISO image should be passed to QEMU as follows:

    -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \
    -device virtio-scsi-pci,id=scsi0 \
    -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \

"Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are present.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- add the .NOTPARALLEL target [Phil, help-make, edk2-devel]

 tests/uefi-test-tools/Makefile   |  97 +++++++++++++
 tests/uefi-test-tools/.gitignore |   3 +
 tests/uefi-test-tools/build.sh   | 145 ++++++++++++++++++++
 3 files changed, 245 insertions(+)
tests/uefi-test-tools/Makefile new file +77 -0
new file mode 100644
index 000000000000..61d263861e01
@@ -0,0 +1,97 @@
+# Makefile for the test helper UEFI applications that run in guests.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+edk2_dir              := ../../roms/edk2
+images_dir            := ../data/uefi-boot-images
+emulation_targets     := arm aarch64 i386 x86_64
+uefi_binaries         := bios-tables-test
+intermediate_suffixes := .efi .fat .iso.raw
+
+images: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(images_dir)/$(binary).$(target).iso.qcow2))
+
+# Preserve all intermediate targets if the build succeeds.
+# - Intermediate targets help with development & debugging.
+# - Preserving intermediate targets also keeps spurious changes out of the
+#   final build products, in case the user re-runs "make" without any changes
+#   to the UEFI source code. Normally, the intermediate files would have been
+#   removed by the last "make" invocation, hence the re-run would rebuild them
+#   from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and
+#   "genisoimage" utilities embed timestamp-based information in their outputs,
+#   which causes git to report differences for the tracked qcow2 ISO images.
+.SECONDARY: $(foreach binary,$(uefi_binaries), \
+		$(foreach target,$(emulation_targets), \
+			$(foreach suffix,$(intermediate_suffixes), \
+				Build/$(binary).$(target)$(suffix))))
+
+# In the pattern rules below, the stem (%, $*) stands for
+# "$(binary).$(target)".
+
+# Convert the raw ISO image to a qcow2 one, enabling compression, and using a
+# small cluster size. This allows for small binary files under git control,
+# hence for small binary patches.
+$(images_dir)/%.iso.qcow2: Build/%.iso.raw
+	mkdir -p -- $(images_dir)
+	$${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \
+		-o cluster_size=512 -- $< $@
+
+# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito
+# boot image.
+Build/%.iso.raw: Build/%.fat
+	genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \
+		-quiet -o $@ -- $<
+
+# Define chained macros in order to map QEMU system emulation targets to
+# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately
+# stripped from, the argument.
+map_arm_to_uefi     = $(subst arm,ARM,$(1))
+map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1)))
+map_i386_to_uefi    = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1)))
+map_x86_64_to_uefi  = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1)))
+map_to_uefi         = $(subst .,,$(call map_x86_64_to_uefi,$(1)))
+
+# Format a "UEFI system partition", using the UEFI binary as the default boot
+# loader. Add 10% size for filesystem metadata, round up to the next KB, and
+# make sure the size is large enough for a FAT filesystem. Name the filesystem
+# after the UEFI binary. (Excess characters are automatically dropped from the
+# filesystem label.)
+Build/%.fat: Build/%.efi
+	rm -f -- $@
+	uefi_bin_b=$$(stat --format=%s -- $<) && \
+		uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \
+		uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \
+		mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI
+	MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT
+	MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \
+		::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI

tests/uefi-test-tools/Makefile line 87

Quoted 10 lines
tests/uefi-test-tools/Makefile new file +10 -0
+
+# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The
+# association between the UEFI binary (such as "bios-tables-test") and the
+# component name from the edk2 platform DSC file (such as "BiosTablesTest") is
+# explicit in each rule.
+
+# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2
+# workspace, at most one "build" instance may be operating at a time. Therefore
+# we must serialize the rebuilding of targets in this Makefile.
+.NOTPARALLEL:

Well this doesn't seem to improve my case :|

Quoted 8 lines

You can test with:

$ alias make='make -j8 -l7.5'

So you get:

$ make print-MAKEFLAGS
MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4

I get something different (with make-3.82-23.el7.x86_64):

MAKEFLAGS=Rrl 7.5 - --jobserver-fds=3,4 -j

Notably, the argument 8 of "-j" is dropped.

Anyway, I think this is a side topic.

Quoted 3 lines

and this command fails:

$ make -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

(1) How *exactly* does it fail for you?

(2) What operating system and "make" are you using?

Because, the command completes perfectly fine for me. (Just for this test, I rebased the v2 branch on top of current master, i.e. commit aefcd2836620, and re-tested there.)

(3) Also, did you update all submodules first? (git submodule update --init --force)

(4) IMPORTANT: did you make sure you didn't have an earlier *mis-built* BaseTools directory in the submodule? For that, run "make clean" in the QEMU project root. Note that "git clean -ffdx" will *not* clean submodules, when issued from the QEMU project root.

Quoted 21 lines

Now, using -j1 we have:

$ make print-MAKEFLAGS -j1
MAKEFLAGS=rR -j1 -l7.5

And the builds work:

$ make -j1 -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

Reading 'info make' section "5.7.3 Communicating Options to a Sub-'make'" I figured we can overwrite MAKEFLAGS, and this snippet works like charm:

-- >8 --

(quoted diff) +0 -0
@@ -84,8 +84,7 @@ Build/%.fat: Build/%.efi
 # "build.sh" invokes the "build" utility of edk2 BaseTools. In any

given edk2
# workspace, at most one "build" instance may be operating at a time.
Therefore

 # we must serialize the rebuilding of targets in this Makefile.
-.NOTPARALLEL:

Review

Quoted 6 lines
(quoted diff) +1 -2
-
+Build/bios-tables-test.%.efi: MAKEFLAGS=
 Build/bios-tables-test.%.efi: build-edk2-tools
        ./build.sh $(edk2_dir) BiosTablesTest $* $@
 
---

No, this is wrong. First, it removes other make flags that are not related to parallelized building. Second, I was advised on the help-make mailing list that if we really want to manipulate make options (as opposed to setting .NOTPARALLEL), then we should explicitly append a "-j1" to the invocation of the inner make. Please see this thread:
<http://lists.gnu.org/archive/html/help-make/2019-01/msg00004.html>.

That would mean modifying edk2 build code (because that's where the inner make is invoked from), and I started researching that, but it wasn't necessary in the end.

Quoted 5 lines

I tested with:

$ rm -rf tests/uefi-test-tools/Build
$ make -C tests/uefi-test-tools \
  Build/bios-tables-test.{arm,aarch64,i386,x86_64}.efi images -j42

This command works for me flawlessly already, with my tree as-is.

Quoted 3 lines

What do you think about using this snippet? I don't know if this is the best way to solve this, but it works...
If you test/agree, Michael could eventually apply the snippet directly.

Again, I disagree with wiping MAKEFLAGS (or messing with it in any way); *if* we need to modify the inner make's command line, that should be done differently. However, my main point is that we need not modify the inner make's invocation.

Can you please answer questions (1) through (4)? If it still fails for you afterwards, I could attempt reproducing the issue in an environment that's similar to yours.

Thanks,
Laszlo

Quoted 172 lines
Quoted 172 lines
tests/uefi-test-tools/Makefile new file +10 -0
+
+Build/bios-tables-test.%.efi: build-edk2-tools
+	./build.sh $(edk2_dir) BiosTablesTest $* $@
+
+build-edk2-tools:
+	$(MAKE) -C $(edk2_dir)/BaseTools
+
+clean:
+	rm -rf Build Conf log
+	$(MAKE) -C $(edk2_dir)/BaseTools clean
tests/uefi-test-tools/.gitignore new file +3 -0
new file mode 100644
index 000000000000..9f246701dea1
@@ -0,0 +1,3 @@
+Build
+Conf
+log
tests/uefi-test-tools/build.sh new file +145 -0
new file mode 100755
index 000000000000..155cb75c4ddb
@@ -0,0 +1,145 @@
+#!/bin/bash
+
+# Build script that determines the edk2 toolchain to use, invokes the edk2
+# "build" utility, and copies the built UEFI binary to the requested location.
+#
+# Copyright (C) 2019, Red Hat, Inc.
+#
+# This program and the accompanying materials are licensed and made available
+# under the terms and conditions of the BSD License that accompanies this
+# distribution. The full text of the license may be found at
+# <http://opensource.org/licenses/bsd-license.php>.
+#
+# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
+# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+
+set -e -u -C
+
+# Save the command line arguments. We need to reset $# to 0 before sourcing
+# "edksetup.sh", as it will inherit $@.
+program_name=$(basename -- "$0")
+edk2_dir=$1
+dsc_component=$2
+emulation_target=$3
+uefi_binary=$4
+shift 4
+
+# Set up the environment for edk2 building.
+export PACKAGES_PATH=$(realpath -- "$edk2_dir")
+export WORKSPACE=$PWD
+mkdir -p Conf
+
+# Source "edksetup.sh" carefully.
+set +e +u +C
+source "$PACKAGES_PATH/edksetup.sh"
+ret=$?
+set -e -u -C
+if [ $ret -ne 0 ]; then
+  exit $ret
+fi
+
+# Map the QEMU system emulation target to the following types of architecture
+# identifiers:
+# - edk2,
+# - gcc cross-compilation.
+# Cover only those targets that are supported by the UEFI spec and edk2.
+case "$emulation_target" in
+  (arm)
+    edk2_arch=ARM
+    gcc_arch=arm
+    ;;
+  (aarch64)
+    edk2_arch=AARCH64
+    gcc_arch=aarch64
+    ;;
+  (i386)
+    edk2_arch=IA32
+    gcc_arch=i686
+    ;;
+  (x86_64)
+    edk2_arch=X64
+    gcc_arch=x86_64
+    ;;
+  (*)
+    printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \
+      "$program_name" "$emulation_target" >&2
+    exit 1
+    ;;
+esac
+
+# Check if cross-compilation is needed.
+host_arch=$(uname -m)
+if [ "$gcc_arch" == "$host_arch" ] ||
+   ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
+  cross_prefix=
+else
+  cross_prefix=${gcc_arch}-linux-gnu-
+fi
+
+# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it,
+# determine the suitable edk2 toolchain as well.
+# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers
+#   the gcc-5+ releases.
+# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in
+#   addition to GCC5. Unfortunately, the mapping between the toolchain tags and
+#   the actual gcc releases isn't entirely trivial. Run "git-blame" on
+#   "OvmfPkg/build.sh" in edk2 for more information.
+# And, because the above is too simple, we have to assign cross_prefix to an
+# edk2 build variable that is specific to both the toolchain tag and the target
+# architecture.
+case "$edk2_arch" in
+  (ARM)
+    edk2_toolchain=GCC5
+    export GCC5_ARM_PREFIX=$cross_prefix
+    ;;
+  (AARCH64)
+    edk2_toolchain=GCC5
+    export GCC5_AARCH64_PREFIX=$cross_prefix
+    ;;
+  (IA32|X64)
+    gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}')
+    case "$gcc_version" in
+      ([1-3].*|4.[0-3].*)
+        printf '%s: unsupported gcc version "%s"\n' \
+          "$program_name" "$gcc_version" >&2
+        exit 1
+        ;;
+      (4.4.*)
+        edk2_toolchain=GCC44
+        ;;
+      (4.5.*)
+        edk2_toolchain=GCC45
+        ;;
+      (4.6.*)
+        edk2_toolchain=GCC46
+        ;;
+      (4.7.*)
+        edk2_toolchain=GCC47
+        ;;
+      (4.8.*)
+        edk2_toolchain=GCC48
+        ;;
+      (4.9.*|6.[0-2].*)
+        edk2_toolchain=GCC49
+        ;;
+      (*)
+        edk2_toolchain=GCC5
+        ;;
+    esac
+    eval "export ${edk2_toolchain}_BIN=\$cross_prefix"
+    ;;
+esac
+
+# Build the UEFI binary
+mkdir -p log
+build \
+  --arch="$edk2_arch" \
+  --buildtarget=DEBUG \
+  --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \
+  --tagname="$edk2_toolchain" \
+  --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \
+  --log="log/$dsc_component.$edk2_arch.log" \
+  --report-file="log/$dsc_component.$edk2_arch.report"
+cp -a -- \
+  "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \
+  "$uefi_binary"
#9 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Quoted 3 lines · Laszlo Ersek

(4) IMPORTANT: did you make sure you didn't have an earlier *mis-built* BaseTools directory in the submodule? For that, run "make clean" in the QEMU project root.

Sorry, thinko above; the command to run is

make -C tests/uefi-test-tools clean

Quoted 2 lines

Note that "git clean -ffdx" will *not* clean submodules, when issued from the QEMU project root.

Thanks
Laszlo

#10 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

In reply to Laszlo Ersek
Quoted 40 lines · Laszlo Ersek
Quoted 9 lines · Philippe Mathieu-Daudé

tests/uefi-test-tools/Makefile line 87

Quoted 4 lines · Laszlo Ersek
tests/uefi-test-tools/Makefile new file +4 -0
+# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2
+# workspace, at most one "build" instance may be operating at a time. Therefore
+# we must serialize the rebuilding of targets in this Makefile.
+.NOTPARALLEL:

Well this doesn't seem to improve my case :|

Quoted 8 lines

You can test with:

$ alias make='make -j8 -l7.5'

So you get:

$ make print-MAKEFLAGS
MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4

I get something different (with make-3.82-23.el7.x86_64):

MAKEFLAGS=Rrl 7.5 - --jobserver-fds=3,4 -j

Notably, the argument 8 of "-j" is dropped.

Anyway, I think this is a side topic.

Quoted 3 lines

and this command fails:

$ make -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

(1) How *exactly* does it fail for you?

(2) What operating system and "make" are you using?

Because, the command completes perfectly fine for me. (Just for this test, I rebased the v2 branch on top of current master, i.e. commit aefcd2836620, and re-tested there.)

Quoted 7 lines

(3) Also, did you update all submodules first? (git submodule update --init --force)

(4) IMPORTANT: did you make sure you didn't have an earlier *mis-built* BaseTools directory in the submodule? For that, run "make clean" in the QEMU project root. Note that "git clean -ffdx" will *not* clean submodules, when issued from the QEMU project root.

I've found the issue myself, through using an up to date Fedora 29 guest for the build.

(a) On RHEL7, I got "make-3.82-23.el7.x86_64". On Fedora 29, it's "make-4.2.1-10.fc29.x86_64". Note in particular "4.2", on Fedora.

(b) In the earlier discussion on help-make
<http://lists.gnu.org/archive/html/help-make/2019-01/msg00004.html>,
Paul Smith responded to me,

Quoted 8 lines · Paul Smith
Quoted 3 lines · Laszlo Ersek

In particular, "--jobserver-fds" is not documented in end-user documentation, apparently, so I get a feeling this option could change at any time, as an implementation detail of distributing jobs.

In fact it DID change, in GNU make 4.2, to be --jobserver-auth. At the same time this new option was published in the documentation and made an official part of the GNU make interface.

(c) That is, the version bump from RHEL7's make to Fedora29's make triggered, in our testing, this change. This explains why

make print-MAKEFLAGS

printed

  MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4

for you, and returned

  MAKEFLAGS=Rrl 7.5 - --jobserver-fds=3,4 -j

for me.

(d) The build *indeed* fails on Fedora29, with the following obscure error:

Quoted 2 lines

make[1]: *** read jobs pipe: Bad file descriptor. Stop. make[1]: *** Waiting for unfinished jobs....

(e) The documentation that Paul referenced, after I found it, provided the critical hint.

https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

Quoted 5 lines

If your tool determines that the --jobserver-auth option is available in MAKEFLAGS but that the file descriptors specified are closed, this means that the calling make process did not think that your tool was a recursive make invocation (e.g., the command line was not prefixed with a + character). [...]

This means that the inner (= edk2's) make process inherits the "--jobserver-auth=3,4" option from the outer (= qemu's) make process, through the MAKEFLAGS variable. However, it does not inherit the file descriptors themselves! And the reason is that the outer make does not believe that the recipe that invokes "build.sh" is in fact an (indirect) recursive make invocation. Therefore the outer make never bothers passing down the file descriptors for the jobserver pipe.

The documentation writes,

Quoted 5 lines

[...] only command lines that make understands to be recursive invocations of make [...] will have access to the jobserver. When writing makefiles you must be sure to mark the command as recursive (most commonly by prefixing the command line with the + indicator [...]

(Also:

Quoted 2 lines

Using the MAKE variable has the same effect as using a '+' character at the beginning of the recipe line.

This is why an explicit "+" is needed only infrequently; most recursive invocations are performed via $(MAKE), and the outer make recognizes that automatically).

(f) So, the solution is to prefix the "./build.sh" recipe with a "+" sign, to mark it as "recursive":

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 93

Quoted 13 lines
tests/uefi-test-tools/Makefile +1 -1
index 61d263861e..449b81d8ba 100644
@@ -87,7 +87,7 @@ Build/%.fat: Build/%.efi
 .NOTPARALLEL:
 
 Build/bios-tables-test.%.efi: build-edk2-tools
-	./build.sh $(edk2_dir) BiosTablesTest $* $@
+	+./build.sh $(edk2_dir) BiosTablesTest $* $@
 
 build-edk2-tools:
 	$(MAKE) -C $(edk2_dir)/BaseTools

This fixes the issue for me on Fedora 29, without breaking the behavior on RHEL7.

I'll submit v3 later. Thank you for catching this error.

Laszlo

Reviewed-by Tested-by
#11 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Quoted 26 lines · Laszlo Ersek
Quoted 25 lines · Laszlo Ersek
Quoted 9 lines · Philippe Mathieu-Daudé

tests/uefi-test-tools/Makefile line 87

Quoted 4 lines · Laszlo Ersek
tests/uefi-test-tools/Makefile new file +4 -0
+# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2
+# workspace, at most one "build" instance may be operating at a time. Therefore
+# we must serialize the rebuilding of targets in this Makefile.
+.NOTPARALLEL:

Well this doesn't seem to improve my case :|

Quoted 8 lines

You can test with:

$ alias make='make -j8 -l7.5'

So you get:

$ make print-MAKEFLAGS
MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4

I get something different (with make-3.82-23.el7.x86_64):

MAKEFLAGS=Rrl 7.5 - --jobserver-fds=3,4 -j

Notably, the argument 8 of "-j" is dropped.

Quoted 8 lines
Quoted 8 lines

Anyway, I think this is a side topic.

Quoted 3 lines

and this command fails:

$ make -C tests/uefi-test-tools Build/bios-tables-test.x86_64.efi

(1) How *exactly* does it fail for you?

In my mailbox the mail appears with Message-ID:
[email protected] in response to
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06393.html but I can't see the mail on the public archives :/

Quoted 44 lines
Quoted 7 lines

(2) What operating system and "make" are you using?

Because, the command completes perfectly fine for me. (Just for this test, I rebased the v2 branch on top of current master, i.e. commit aefcd2836620, and re-tested there.)

Quoted 7 lines

(3) Also, did you update all submodules first? (git submodule update --init --force)

(4) IMPORTANT: did you make sure you didn't have an earlier *mis-built* BaseTools directory in the submodule? For that, run "make clean" in the QEMU project root. Note that "git clean -ffdx" will *not* clean submodules, when issued from the QEMU project root.

I've found the issue myself, through using an up to date Fedora 29 guest for the build.

(a) On RHEL7, I got "make-3.82-23.el7.x86_64". On Fedora 29, it's "make-4.2.1-10.fc29.x86_64". Note in particular "4.2", on Fedora.

(b) In the earlier discussion on help-make
<http://lists.gnu.org/archive/html/help-make/2019-01/msg00004.html>,
Paul Smith responded to me,

Quoted 8 lines · Paul Smith
Quoted 3 lines · Laszlo Ersek

In particular, "--jobserver-fds" is not documented in end-user documentation, apparently, so I get a feeling this option could change at any time, as an implementation detail of distributing jobs.

In fact it DID change, in GNU make 4.2, to be --jobserver-auth. At the same time this new option was published in the documentation and made an official part of the GNU make interface.

(c) That is, the version bump from RHEL7's make to Fedora29's make triggered, in our testing, this change. This explains why

make print-MAKEFLAGS

printed

  MAKEFLAGS=rR -j8 -l7.5 --jobserver-auth=3,4
Quoted 11 lines

for you, and returned

  MAKEFLAGS=Rrl 7.5 - --jobserver-fds=3,4 -j

for me.

(d) The build *indeed* fails on Fedora29, with the following obscure error:

Quoted 2 lines

make[1]: *** read jobs pipe: Bad file descriptor. Stop. make[1]: *** Waiting for unfinished jobs....

Yes, this is the problem.

Quoted 11 lines

(e) The documentation that Paul referenced, after I found it, provided the critical hint.

https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html

Quoted 5 lines

If your tool determines that the --jobserver-auth option is available in MAKEFLAGS but that the file descriptors specified are closed, this means that the calling make process did not think that your tool was a recursive make invocation (e.g., the command line was not prefixed with a + character). [...]

o_O
Cc'ing Paolo and Eric who seems to like that kind of makefile black magic.

Quoted 7 lines

This means that the inner (= edk2's) make process inherits the "--jobserver-auth=3,4" option from the outer (= qemu's) make process, through the MAKEFLAGS variable. However, it does not inherit the file descriptors themselves! And the reason is that the outer make does not believe that the recipe that invokes "build.sh" is in fact an (indirect) recursive make invocation. Therefore the outer make never bothers passing down the file descriptors for the jobserver pipe.

Yes!

Quoted 28 lines

The documentation writes,

Quoted 5 lines

[...] only command lines that make understands to be recursive invocations of make [...] will have access to the jobserver. When writing makefiles you must be sure to mark the command as recursive (most commonly by prefixing the command line with the + indicator [...]

(Also:

Quoted 2 lines

Using the MAKE variable has the same effect as using a '+' character at the beginning of the recipe line.

This is why an explicit "+" is needed only infrequently; most recursive invocations are performed via $(MAKE), and the outer make recognizes that automatically).

(f) So, the solution is to prefix the "./build.sh" recipe with a "+" sign, to mark it as "recursive":

Quoted 7 lines
tests/uefi-test-tools/Makefile +0 -0
index 61d263861e..449b81d8ba 100644
@@ -87,7 +87,7 @@ Build/%.fat: Build/%.efi
 .NOTPARALLEL:
 
Quoted 3 lines

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 90

Quoted 3 lines
tests/uefi-test-tools/Makefile +1 -1
 Build/bios-tables-test.%.efi: build-edk2-tools
-	./build.sh $(edk2_dir) BiosTablesTest $* $@
+	+./build.sh $(edk2_dir) BiosTablesTest $* $@

Yes :) This fixed it!

Quoted 6 lines

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 92

Quoted 3 lines
tests/uefi-test-tools/Makefile +0 -0
 build-edk2-tools:
 	$(MAKE) -C $(edk2_dir)/BaseTools

This fixes the issue for me on Fedora 29, without breaking the behavior on RHEL7.

Excellent! I triggered Travis builds (Ubuntu/Debian).

Quoted 2 lines

I'll submit v3 later. Thank you for catching this error.

Now that we are happy, maybe Michael can do this change when applying, but I guess you'd prefer first to write a line about this '+' in the commit message or the Makefile.

With the +:

Thanks a lot for figuring this out alone and fixing it!

Phil.

#12 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Quoted 9 lines · Philippe Mathieu-Daudé
Quoted 3 lines · Laszlo Ersek
Quoted 1 line · Laszlo Ersek

(1) How *exactly* does it fail for you?

In my mailbox the mail appears with Message-ID:
[email protected] in response to
https://lists.gnu.org/archive/html/qemu-devel/2019-01/msg06393.html but I can't see the mail on the public archives :/

Ouch. I've now searched my mailbox for the Message-ID above, and there's no match. Too bad your email got lost.

Quoted 23 lines
Quoted 10 lines

(f) So, the solution is to prefix the "./build.sh" recipe with a "+" sign, to mark it as "recursive":

Quoted 7 lines
tests/uefi-test-tools/Makefile +0 -0
index 61d263861e..449b81d8ba 100644
@@ -87,7 +87,7 @@ Build/%.fat: Build/%.efi
 .NOTPARALLEL:
 
Quoted 3 lines

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 90

Quoted 3 lines
tests/uefi-test-tools/Makefile +1 -1
 Build/bios-tables-test.%.efi: build-edk2-tools
-	./build.sh $(edk2_dir) BiosTablesTest $* $@
+	+./build.sh $(edk2_dir) BiosTablesTest $* $@

Yes :) This fixed it!

Quoted 6 lines

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 92

Quoted 3 lines
tests/uefi-test-tools/Makefile +0 -0
 build-edk2-tools:
 	$(MAKE) -C $(edk2_dir)/BaseTools

This fixes the issue for me on Fedora 29, without breaking the behavior on RHEL7.

Quoted 8 lines

Excellent! I triggered Travis builds (Ubuntu/Debian).

Quoted 2 lines

I'll submit v3 later. Thank you for catching this error.

Now that we are happy, maybe Michael can do this change when applying, but I guess you'd prefer first to write a line about this '+' in the commit message or the Makefile.

That's right, I'd like to extend the comment that we already have in the Makefile about .NOTPARALLEL, with a note on the "+" indicator.

Quoted 4 lines

With the +:

Awesome, thank you! :)

Quoted 2 lines

Thanks a lot for figuring this out alone and fixing it!

It definitely helped that you mentioned your use of Fedora 29 earlier :)

Also, we're very lucky that GNU Make's documentation is so nice.

Thanks,
Laszlo

#13 lore raw

Re: [Qemu-devel] [PATCH v2 4/5] tests/uefi-test-tools: add build scripts

Quoted 3 lines · Laszlo Ersek
Quoted 2 lines · Philippe Mathieu-Daudé
Quoted 1 line · Laszlo Ersek

On 01/31/19 19:55, Laszlo Ersek wrote:

[...]

Quoted 17 lines
Quoted 17 lines
Quoted 10 lines

(f) So, the solution is to prefix the "./build.sh" recipe with a "+" sign, to mark it as "recursive":

Quoted 7 lines
tests/uefi-test-tools/Makefile +0 -0
index 61d263861e..449b81d8ba 100644
@@ -87,7 +87,7 @@ Build/%.fat: Build/%.efi
 .NOTPARALLEL:
 
Quoted 3 lines

tests/uefi-test-tools/Makefile Build/%.fat: Build/%.efi · line 90

Quoted 3 lines
tests/uefi-test-tools/Makefile +1 -1
 Build/bios-tables-test.%.efi: build-edk2-tools
-	./build.sh $(edk2_dir) BiosTablesTest $* $@
+	+./build.sh $(edk2_dir) BiosTablesTest $* $@

Yes :) This fixed it!

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

build-edk2-tools:
$(MAKE) -C $(edk2_dir)/BaseTools

This fixes the issue for me on Fedora 29, without breaking the behavior on RHEL7.

Excellent! I triggered Travis builds (Ubuntu/Debian).

FYI, everthing build fine there.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I'll submit v3 later. Thank you for catching this error.

[...]

#14 lore raw

PATCH v2 5/5 tests/data: introduce "uefi-boot-images" with the "bios-tables-test" ISOs

In reply to Laszlo Ersek

Add UEFI-bootable qcow2-compressed ISO images built from:

tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest

Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

---

Notes:
v2:
- refresh the images (should only differ in internal timestamps etc)

v1:
- Again, if needed, I'd be happy to be designated as Maintainer for the files being added in this patch; please let me know the right spot in "MAINTAINERS".

 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2 | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2     | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2    | Bin 0 -> 12800 bytes
 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2  | Bin 0 -> 13312 bytes
 4 files changed, 0 insertions(+), 0 deletions(-)
tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..8d7f8e67e85aa2156adb3423322018f6a6ea890c
GIT binary patch
tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..373947a2f3a6b08934009035e7151ad845985ab2
GIT binary patch
tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..7c846195e552b948fd0b50ae3f4a39e9df65f7c6
GIT binary patch
tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2 new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..9d9b391e88145cf0f729ff50d8383096bb3a8b06
GIT binary patch
Signature · 1 line
2.19.1.3.g30247aa5d201
#15 lore raw
In reply to Laszlo Ersek
Quoted 4 lines · Laszlo Ersek

Previous version (v1):
http://mid.mail-archive.com/[email protected]

Updates in v2 have been noted on each patch in the series.

ping

Quoted 52 lines
Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

Thanks
Laszlo

Laszlo Ersek (5):
roms: add the edk2 project as a git submodule
roms: build the EfiRom utility from the roms/edk2 submodule
tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app tests/uefi-test-tools: add build scripts
tests/data: introduce "uefi-boot-images" with the "bios-tables-test"
ISOs

 .gitmodules                                                              |   3 +
 Makefile                                                                 |   6 +-
 roms/Makefile                                                            |  13 +-
 roms/edk2                                                                |   1 +
 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2           | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2               | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2              | Bin 0 -> 12800 bytes
 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2            | Bin 0 -> 13312 bytes
 tests/uefi-test-tools/.gitignore                                         |   3 +
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 tests/uefi-test-tools/Makefile                                           |  97 +++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 +++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 ++++++++++
 tests/uefi-test-tools/build.sh                                           | 145 ++++++++++++++++++++
 17 files changed, 619 insertions(+), 8 deletions(-)
 create mode 160000 roms/edk2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2
 create mode 100644 tests/uefi-test-tools/.gitignore
 create mode 100644 tests/uefi-test-tools/LICENSE
 create mode 100644 tests/uefi-test-tools/Makefile
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc
 create mode 100755 tests/uefi-test-tools/build.sh
#16 lore raw
Quoted 7 lines · Laszlo Ersek
Quoted 4 lines · Laszlo Ersek

Previous version (v1):
http://mid.mail-archive.com/[email protected]

Updates in v2 have been noted on each patch in the series.

ping

I'll queue this, thanks!

Quoted 52 lines
Quoted 52 lines
Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

Thanks
Laszlo

Laszlo Ersek (5):
roms: add the edk2 project as a git submodule
roms: build the EfiRom utility from the roms/edk2 submodule
tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app tests/uefi-test-tools: add build scripts
tests/data: introduce "uefi-boot-images" with the "bios-tables-test"
ISOs

 .gitmodules                                                              |   3 +
 Makefile                                                                 |   6 +-
 roms/Makefile                                                            |  13 +-
 roms/edk2                                                                |   1 +
 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2           | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2               | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2              | Bin 0 -> 12800 bytes
 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2            | Bin 0 -> 13312 bytes
 tests/uefi-test-tools/.gitignore                                         |   3 +
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 tests/uefi-test-tools/Makefile                                           |  97 +++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 +++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 ++++++++++
 tests/uefi-test-tools/build.sh                                           | 145 ++++++++++++++++++++
 17 files changed, 619 insertions(+), 8 deletions(-)
 create mode 160000 roms/edk2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2
 create mode 100644 tests/uefi-test-tools/.gitignore
 create mode 100644 tests/uefi-test-tools/LICENSE
 create mode 100644 tests/uefi-test-tools/Makefile
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc
 create mode 100755 tests/uefi-test-tools/build.sh
#17 lore raw
In reply to Laszlo Ersek
Quoted 7 lines · Laszlo Ersek
Quoted 4 lines · Laszlo Ersek

Previous version (v1):
http://mid.mail-archive.com/[email protected]

Updates in v2 have been noted on each patch in the series.

ping

Oh sorry I noticed Philippe reported some failures with this patchset. Can you guys pls get to the bottom of it? I don't want everyone to suffer.

Quoted 52 lines
Quoted 52 lines
Cc "Michael S. Tsirkin" <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc Gerd Hoffmann <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Philippe Mathieu-Daudé <[email protected]>
Cc Shannon Zhao <[email protected]>

Thanks
Laszlo

Laszlo Ersek (5):
roms: add the edk2 project as a git submodule
roms: build the EfiRom utility from the roms/edk2 submodule
tests: introduce "uefi-test-tools" with the BiosTablesTest UEFI app tests/uefi-test-tools: add build scripts
tests/data: introduce "uefi-boot-images" with the "bios-tables-test"
ISOs

 .gitmodules                                                              |   3 +
 Makefile                                                                 |   6 +-
 roms/Makefile                                                            |  13 +-
 roms/edk2                                                                |   1 +
 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2           | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2               | Bin 0 -> 11776 bytes
 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2              | Bin 0 -> 12800 bytes
 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2            | Bin 0 -> 13312 bytes
 tests/uefi-test-tools/.gitignore                                         |   3 +
 tests/uefi-test-tools/LICENSE                                            |  25 ++++
 tests/uefi-test-tools/Makefile                                           |  97 +++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c   | 130 ++++++++++++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf |  41 ++++++
 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h     |  67 +++++++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec              |  27 ++++
 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc              |  69 ++++++++++
 tests/uefi-test-tools/build.sh                                           | 145 ++++++++++++++++++++
 17 files changed, 619 insertions(+), 8 deletions(-)
 create mode 160000 roms/edk2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.aarch64.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.arm.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.i386.iso.qcow2
 create mode 100644 tests/data/uefi-boot-images/bios-tables-test.x86_64.iso.qcow2
 create mode 100644 tests/uefi-test-tools/.gitignore
 create mode 100644 tests/uefi-test-tools/LICENSE
 create mode 100644 tests/uefi-test-tools/Makefile
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.c
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/Include/Guid/BiosTablesTest.h
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dec
 create mode 100644 tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc
 create mode 100755 tests/uefi-test-tools/build.sh
#18 lore raw
Quoted 11 lines · Michael S. Tsirkin
Quoted 7 lines · Laszlo Ersek
Quoted 4 lines · Laszlo Ersek

Previous version (v1):
http://mid.mail-archive.com/[email protected]

Updates in v2 have been noted on each patch in the series.

ping

Oh sorry I noticed Philippe reported some failures with this patchset. Can you guys pls get to the bottom of it?

Right, we've now got all the info that's needed for me to submit v3. What's still necessary is a good amount of sleep. :)

Quoted 1 line

I don't want everyone to suffer.

Thankfully, this Makefile could only cause a handful of maintainers to suffer, and only if they decided to rebuild the ACPI-test-helper UEFI-bootable ISO images :) The normal QEMU build (including the test suite run) doesn't go near this stuff. The usage pattern is the same as when Gerd manually rebuilds the iPXE oproms.

Thanks,
Laszlo