Skip to contentSkip to conversation

Fix qemu package build

PATCH In review Alexey Neyman 3 participants 9 messages Jul 21, 2021 – Aug 7, 2021 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

cover Fix qemu package build

Patch ping; no response received to patches submitted 5 days ago.

First, there is a problem that is currently reported by buildroot's
autobuilder: the --disable-git-update option is now deprecated and is interpreted as --with-git-submodules=validate, which can only work in a git clone. The correct replacement is --with-git-submodules=ignore.

With that fixed, package/qemu build succeeds. However, on hosts using make 3.82 (e.g CentOS 7), despite reporting build success, it fails to actually build or install any target binaries. The problem is that Qemu's makefile incorrectly interprets the --no-print-directory passed by buildroot as the -n option to make and injects -n (--just-print) into its ninja build.

Alexey Neyman (2):
qemu: do not use deprecated option
qemu: filter out long make options

 package/qemu/qemu.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#2 lore raw

PATCH 1/2 package/qemu: do not use deprecated option

Qemu build fails with:

--disable-git-update deprecated, use --with-git-submodules=validate
ERROR: cannot validate git submodules without .git

Use --with-git-submodules=ignore instead.

---
 package/qemu/qemu.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/qemu/qemu.mk +1 -1
index e0f722cf42..470c285f16 100644
@@ -198,7 +198,7 @@ define QEMU_CONFIGURE_CMDS
 			--disable-vhost-crypto \
 			--disable-libxml2 \
 			--disable-capstone \
-			--disable-git-update \
+			--with-git-submodules=ignore \
 			--disable-opengl \
 			--disable-vhost-user-blk-server \
 			--disable-virtiofsd \
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#3 lore raw

Re: [Buildroot] [PATCH 1/2] package/qemu: do not use deprecated option

Alexey, All,

On 2021-07-21 12:46 -0700, Alexey Neyman spake thusly:

Quoted 8 lines

Qemu build fails with:

--disable-git-update deprecated, use --with-git-submodules=validate
ERROR: cannot validate git submodules without .git

Use --with-git-submodules=ignore instead.

An identical patch by Joseph was already pending in the backlog, so I applied his patch.

Thanks!

Regards,
Yann E. MORIN.

Quoted 24 lines
---
 package/qemu/qemu.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
package/qemu/qemu.mk +1 -1
index e0f722cf42..470c285f16 100644
@@ -198,7 +198,7 @@ define QEMU_CONFIGURE_CMDS
 			--disable-vhost-crypto \
 			--disable-libxml2 \
 			--disable-capstone \
-			--disable-git-update \
+			--with-git-submodules=ignore \
 			--disable-opengl \
 			--disable-vhost-user-blk-server \
 			--disable-virtiofsd \
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#4 lore raw

PATCH 2/2 package/qemu: filter out long make options

In reply to Alexey Neyman

Since qemu started using ninja, they have the following fragment in the Makefile:

MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS))) NINJAFLAGS = ... $(if $(MAKE.n), -n) ...

Buildroot's generated makefile in the O= directory invokes make in the base buildroot with --no-print-directory. However, make's placement of the --no-print-directory in MAKEFLAGS varies between the versions of the host make; make 4.3 places that at the end while make 3.82 places it at the beginning. As a result, if building on a system with an older host make, qemu's makefile invokes ninja -n which does not generate any outputs.

To reproduce, on a CentOS 7 machine or docker image:
mkdir /tmp/br-build && cd /tmp/br-build
make -C ~/buildroot pc_x86_64_bios_defconfig O=pwd make menuconfig # Switch to glibc, enable "QEMU" and "QEMU tools" make all # Build succeeds
find target -name qemu-img # No binary has been built

---
 package/qemu/qemu.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
package/qemu/qemu.mk +3 -2
index 470c285f16..ee3e7c3676 100644
@@ -206,14 +206,15 @@ define QEMU_CONFIGURE_CMDS
 			$(QEMU_OPTS)
 endef
 
+# QEMU Makefile incorrectly interprets long options to make
 define QEMU_BUILD_CMDS
 	unset TARGET_DIR; \
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
+	MAKEFLAGS="$(filter-out --%,$(MAKEFLAGS))" $(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define QEMU_INSTALL_TARGET_CMDS
 	unset TARGET_DIR; \
-	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) $(QEMU_MAKE_ENV) DESTDIR=$(TARGET_DIR) install
+	MAKEFLAGS="$(filter-out --%,$(MAKEFLAGS))" $(TARGET_MAKE_ENV) $(MAKE) -C $(@D) $(QEMU_MAKE_ENV) DESTDIR=$(TARGET_DIR) install
 endef
 
 $(eval $(generic-package))
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#5 lore raw

Re: [Buildroot] [PATCH 2/2] package/qemu: filter out long make options

Hello,

Quoted 14 lines · On Wed, 21 Jul 2021 12:46:05 -0700 Alexey Neyman <[email protected]> wrote:

Since qemu started using ninja, they have the following fragment in the Makefile:

MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS))) NINJAFLAGS = ... $(if $(MAKE.n), -n) ...

Buildroot's generated makefile in the O= directory invokes make in the base buildroot with --no-print-directory. However, make's placement of the --no-print-directory in MAKEFLAGS varies between the versions of the host make; make 4.3 places that at the end while make 3.82 places it at the beginning. As a result, if building on a system with an older host make, qemu's makefile invokes ninja -n which does not generate any outputs.

Quoted 6 lines

To reproduce, on a CentOS 7 machine or docker image:
mkdir /tmp/br-build && cd /tmp/br-build
make -C ~/buildroot pc_x86_64_bios_defconfig O=pwd make menuconfig # Switch to glibc, enable "QEMU" and "QEMU tools" make all # Build succeeds
find target -name qemu-img # No binary has been built

I'm not sure to grasp all the implications of this, but isn't this a bug in Qemu's build machinery, that should be fixed in Qemu's Makefile instead ?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#6 lore raw

Re: [Buildroot] [PATCH 2/2] package/qemu: filter out long make options

Hi,

Quoted 25 lines · Thomas Petazzoni

Hello,

Quoted 14 lines · On Wed, 21 Jul 2021 12:46:05 -0700 Alexey Neyman <[email protected]> wrote:

Since qemu started using ninja, they have the following fragment in the Makefile:

MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS))) NINJAFLAGS = ... $(if $(MAKE.n), -n) ...

Buildroot's generated makefile in the O= directory invokes make in the base buildroot with --no-print-directory. However, make's placement of the --no-print-directory in MAKEFLAGS varies between the versions of the host make; make 4.3 places that at the end while make 3.82 places it at the beginning. As a result, if building on a system with an older host make, qemu's makefile invokes ninja -n which does not generate any outputs.

Quoted 6 lines

To reproduce, on a CentOS 7 machine or docker image:
mkdir /tmp/br-build && cd /tmp/br-build
make -C ~/buildroot pc_x86_64_bios_defconfig O=pwd make menuconfig # Switch to glibc, enable "QEMU" and "QEMU tools" make all # Build succeeds
find target -name qemu-img # No binary has been built

Quoted 3 lines

I'm not sure to grasp all the implications of this, but isn't this a bug in Qemu's build machinery, that should be fixed in Qemu's Makefile instead ?

I was on the fence about it myself. Okay, I'll try to send a fix for qemu/Makefile to Qemu developers and see if they agree to pick it up; they could argue they don't support being invoked with --no-print-directory.

Regards,
Alexey.

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#7 lore raw

Re: [Buildroot] [PATCH 2/2] package/qemu: filter out long make options

Quoted 7 lines · On Wed, 21 Jul 2021 15:18:36 -0700 Alexey Neyman <[email protected]> wrote:
Quoted 3 lines

I'm not sure to grasp all the implications of this, but isn't this a bug in Qemu's build machinery, that should be fixed in Qemu's Makefile instead ?

I was on the fence about it myself. Okay, I'll try to send a fix for qemu/Makefile to Qemu developers and see if they agree to pick it up; they could argue they don't support being invoked with --no-print-directory.

It would be odd for them to consider that as a normal behavior, --no-print-directory is really a standard make option.

Let us know what they say, of course.

Thanks a lot!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#8 lore raw

PATCH 1/1 package/qemu: filter out long make options

Since qemu started using ninja, they have the following fragment in the Makefile:

MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS))) NINJAFLAGS = ... $(if $(MAKE.n), -n) ...

Buildroot's generated makefile in the O= directory invokes make in the base buildroot with --no-print-directory. However, make's placement of the --no-print-directory in MAKEFLAGS varies between the versions of the host make; make 4.3 places that at the end while make 3.82 places it at the beginning. As a result, if building on a system with an older host make, qemu's makefile invokes ninja -n which does not generate any outputs.

To reproduce, on a CentOS 7 machine or docker image:
mkdir /tmp/br-build && cd /tmp/br-build
make -C ~/buildroot pc_x86_64_bios_defconfig O=pwd make menuconfig # Switch to glibc, enable "QEMU" and "QEMU tools" make all # Build succeeds
find target -name qemu-img # No binary has been built

Pick up the fix commited in Qemu upstream.

---
 .../0004-Makefile-ignore-long-options.patch   | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 package/qemu/0004-Makefile-ignore-long-options.patch
package/qemu/0004-Makefile-ignore-long-options.patch new file +41 -0
new file mode 100644
index 0000000000..2d14b70c03
@@ -0,0 +1,41 @@
+From 14833e24dea49303ebc2464813601054b6cdfcac Mon Sep 17 00:00:00 2001
+From: Alexey Neyman <[email protected]>
+Date: Wed, 21 Jul 2021 19:08:46 -0700
+Subject: [PATCH] Makefile: ignore long options
+
+When searching for options like -n in MAKEFLAGS, current code may result
+in a false positive match when make is invoked with long options like
+--no-print-directory. This has been observed with certain versions of
+host make (e.g. 3.82) while building the Qemu package in buildroot.
+
+Filter out such long options before searching for one-character options.
+
+Signed-off-by: Alexey Neyman <[email protected]>
+Message-Id: <[email protected]>
+Signed-off-by: Paolo Bonzini <[email protected]>
+---
+ Makefile | 8 +++++---
+ 1 file changed, 5 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile b/Makefile
+index 6c36330eef..401c623a65 100644
+--- a/Makefile
++++ b/Makefile
+@@ -129,9 +129,11 @@ endif
+ # 4. Rules to bridge to other makefiles
+ 
+ ifneq ($(NINJA),)
+-MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS)))
+-MAKE.k = $(findstring k,$(firstword $(MAKEFLAGS)))
+-MAKE.q = $(findstring q,$(firstword $(MAKEFLAGS)))
++# Filter out long options to avoid flags like --no-print-directory which
++# may result in false positive match for MAKE.n
++MAKE.n = $(findstring n,$(firstword $(filter-out --%,$(MAKEFLAGS))))
++MAKE.k = $(findstring k,$(firstword $(filter-out --%,$(MAKEFLAGS))))
++MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS))))
+ MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq)
+ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
+         $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \
+-- 
+2.27.0
+
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#9 lore raw

Re: [Buildroot] [PATCH] package/qemu: filter out long make options

Alexey, All,

On 2021-08-07 12:15 -0700, Alexey Neyman spake thusly:

Quoted 21 lines

Since qemu started using ninja, they have the following fragment in the Makefile:

MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS))) NINJAFLAGS = ... $(if $(MAKE.n), -n) ...

Buildroot's generated makefile in the O= directory invokes make in the base buildroot with --no-print-directory. However, make's placement of the --no-print-directory in MAKEFLAGS varies between the versions of the host make; make 4.3 places that at the end while make 3.82 places it at the beginning. As a result, if building on a system with an older host make, qemu's makefile invokes ninja -n which does not generate any outputs.

To reproduce, on a CentOS 7 machine or docker image:
mkdir /tmp/br-build && cd /tmp/br-build
make -C ~/buildroot pc_x86_64_bios_defconfig O=pwd make menuconfig # Switch to glibc, enable "QEMU" and "QEMU tools" make all # Build succeeds
find target -name qemu-img # No binary has been built

Quoted 3 lines

Pick up the fix commited in Qemu upstream.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 59 lines
---
 .../0004-Makefile-ignore-long-options.patch   | 41 +++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 package/qemu/0004-Makefile-ignore-long-options.patch
package/qemu/0004-Makefile-ignore-long-options.patch new file +41 -0
new file mode 100644
index 0000000000..2d14b70c03
@@ -0,0 +1,41 @@
+From 14833e24dea49303ebc2464813601054b6cdfcac Mon Sep 17 00:00:00 2001
+From: Alexey Neyman <[email protected]>
+Date: Wed, 21 Jul 2021 19:08:46 -0700
+Subject: [PATCH] Makefile: ignore long options
+
+When searching for options like -n in MAKEFLAGS, current code may result
+in a false positive match when make is invoked with long options like
+--no-print-directory. This has been observed with certain versions of
+host make (e.g. 3.82) while building the Qemu package in buildroot.
+
+Filter out such long options before searching for one-character options.
+
+Signed-off-by: Alexey Neyman <[email protected]>
+Message-Id: <[email protected]>
+Signed-off-by: Paolo Bonzini <[email protected]>
+---
+ Makefile | 8 +++++---
+ 1 file changed, 5 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile b/Makefile
+index 6c36330eef..401c623a65 100644
+--- a/Makefile
++++ b/Makefile
+@@ -129,9 +129,11 @@ endif
+ # 4. Rules to bridge to other makefiles
+ 
+ ifneq ($(NINJA),)
+-MAKE.n = $(findstring n,$(firstword $(MAKEFLAGS)))
+-MAKE.k = $(findstring k,$(firstword $(MAKEFLAGS)))
+-MAKE.q = $(findstring q,$(firstword $(MAKEFLAGS)))
++# Filter out long options to avoid flags like --no-print-directory which
++# may result in false positive match for MAKE.n
++MAKE.n = $(findstring n,$(firstword $(filter-out --%,$(MAKEFLAGS))))
++MAKE.k = $(findstring k,$(firstword $(filter-out --%,$(MAKEFLAGS))))
++MAKE.q = $(findstring q,$(firstword $(filter-out --%,$(MAKEFLAGS))))
+ MAKE.nq = $(if $(word 2, $(MAKE.n) $(MAKE.q)),nq)
+ NINJAFLAGS = $(if $V,-v) $(if $(MAKE.n), -n) $(if $(MAKE.k), -k0) \
+         $(filter-out -j, $(lastword -j1 $(filter -l% -j%, $(MAKEFLAGS)))) \
+-- 
+2.27.0
+
Signature · 6 lines
2.27.0

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot