Skip to contentSkip to conversation

Build fixes

PATCH Superseded Juan Quintela 3 participants 9 messages Nov 5, 2009 – Nov 11, 2009 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Build fixes

Hi

This series fixes the parallel build of qemu. I tried to get the best of Daniel and Stefan ideas with mine.
Once there fixed an obvious fix.

Daniel, can you check that parallel build also works for you? It works for me (TM). Stefan, what do you think?

It avoids the recursive call for serializing dependencies.
We still need to put direct dependencies for qemu-{img,nbd,io}.o of the headers I don't understand why the ones in rules.make are not enough.

Later, Juan.

Juan Quintela (2):
fix parallel build
fix type in CFLAGS name

 Makefile        |    9 ++++++---
 Makefile.target |    6 +++---
 rules.mak       |    4 ++--
 3 files changed, 11 insertions(+), 8 deletions(-)
#1 lore raw

PATCH 1/2 fix parallel build

Based on a ideas of Daniel Jacobowitz + Stefan Weil

---
 Makefile        |    9 ++++++---
 Makefile.target |    6 +++---
 rules.mak       |    2 +-
 3 files changed, 10 insertions(+), 7 deletions(-)
Makefile +6 -3
index c783aa4..ce165b2 100644
@@ -40,8 +40,9 @@ config-all-devices.mak: $(SUBDIR_DEVICES_MAK)
 
 -include config-all-devices.mak
 
-build-all: config-host.h config-all-devices.h $(DOCS) $(TOOLS)
-	$(call quiet-command, $(MAKE) $(SUBDIR_MAKEFLAGS) recurse-all,)
+GENERATED_HEADERS = config-host.h config-all-devices.h
+
+build-all: $(DOCS) $(TOOLS) recurse-all
 
 config-host.h: config-host.h-timestamp
 config-host.h-timestamp: config-host.mak
@@ -67,7 +68,7 @@ romsubdir-%:
 
 ALL_SUBDIRS=$(TARGET_DIRS) $(patsubst %,pc-bios/%, $(ROMS))
 
-recurse-all: $(SUBDIR_RULES) $(ROMSUBDIR_RULES)
+recurse-all: $(GENERATED_HEADERS) $(SUBDIR_RULES) $(ROMSUBDIR_RULES)
 
 #######################################################################
 # block-obj-y is code used by both qemu system emulation and qemu-img
@@ -211,6 +212,8 @@ libqemu_common.a: $(obj-y)
 
 qemu-img.o: qemu-img-cmds.h
 
+qemu-img.o qemu-nbd.o qemu-io.o: $(GENERATED_HEADERS)
+
 qemu-img$(EXESUF): qemu-img.o qemu-tool.o $(block-obj-y)
 
 qemu-nbd$(EXESUF):  qemu-nbd.o qemu-tool.o $(block-obj-y)
Makefile.target +3 -3
index fefd7ac..0673c55 100644
@@ -22,6 +22,7 @@ endif
 endif
 
 PROGS=$(QEMU_PROG)
+GENERATED_HEADERS = config-target.h config-devices.h
 
 LIBS+=-lm
 
@@ -33,8 +34,7 @@ config-target.h-timestamp: config-target.mak
 config-devices.h: config-devices.h-timestamp
 config-devices.h-timestamp: config-devices.mak
 
-all: config-target.h config-devices.h
-	$(call quiet-command, $(MAKE) $(PROGS),)
+all: $(PROGS)
 
 # Dummy command so that make thinks it has done something
 	@true
@@ -308,7 +308,7 @@ endif # CONFIG_SOFTMMU
 
 obj-$(CONFIG_GDBSTUB_XML) += gdbstub-xml.o
 
-$(QEMU_PROG): $(obj-y) $(obj-$(TARGET_BASE_ARCH)-y) $(ARLIBS)
+$(QEMU_PROG): $(GENERATED_HEADERS) $(obj-y) $(obj-$(TARGET_BASE_ARCH)-y) $(ARLIBS)
 	$(call LINK,$(obj-y) $(obj-$(TARGET_BASE_ARCH)-y))
 
 
rules.mak +1 -1
index 5d7e8bb..4eb1f90 100644
@@ -13,7 +13,7 @@ MAKEFLAGS += -rR
 
 QEMU_CFLAGS += -MMD -MP -MT $@
 
-%.o: %.c
+%.o: %.c $(GENERATED_HEADERS)
 	$(call quiet-command,$(CC) $(QEMU_CFLAGS) $(CFLAGS) -c -o $@ $<,"  CC    $(TARGET_DIR)$@")
 
 %.o: %.S
Signature · 1 line
1.6.2.5
#2 lore raw

PATCH 2/2 fix type in CFLAGS name

In reply to Juan Quintela
---
 rules.mak |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
rules.mak +1 -1
index 4eb1f90..f8e5b04 100644
@@ -17,7 +17,7 @@ QEMU_CFLAGS += -MMD -MP -MT $@
 	$(call quiet-command,$(CC) $(QEMU_CFLAGS) $(CFLAGS) -c -o $@ $<,"  CC    $(TARGET_DIR)$@")
 
 %.o: %.S
-	$(call quiet-command,$(CC) $(QEMU_CFLAGS) $(CLAGS) -c -o $@ $<,"  AS    $(TARGET_DIR)$@")
+	$(call quiet-command,$(CC) $(QEMU_CFLAGS) $(CFLAGS) -c -o $@ $<,"  AS    $(TARGET_DIR)$@")
 
 %.o: %.m
 	$(call quiet-command,$(CC) $(QEMU_CFLAGS) $(CFLAGS) -c -o $@ $<,"  OBJC  $(TARGET_DIR)$@")
Signature · 1 line
1.6.2.5
#3 lore raw
In reply to Juan Quintela
Quoted 7 lines · Juan Quintela

Hi

This series fixes the parallel build of qemu. I tried to get the best of Daniel and Stefan ideas with mine.
Once there fixed an obvious fix.

Daniel, can you check that parallel build also works for you? It works for me (TM).

No, it doesn't work.

GEN config-all-devices.mak

  GEN   config-host.h
  GEN   config-all-devices.h
  CC    qemu-tool.o
  CC    cutils.o

In file included from /scratch/dan/qemu/src/qemu-trunk/qemu-tool.c:14:
/scratch/dan/qemu/src/qemu-trunk/qemu-common.h:32:25: error:
config-host.h: No such file or directory
make: *** [qemu-tool.o] Error 1

You have to define GENERATED_HEADERS before you use it in dependencies, I believe. But you've got it way down below the rules.mak invocation.

Also, your changes to $(QEMU_PROG) and recurse-all are not necessary; they're the same sort of ordering assumption that caused the original bug. You only need to depend on generated files if you're going to use them in this rule's commands; for headers, that means invoke a .c to .o compilation. The dependencies should be on exactly the rules that need them.

I didn't bother with Makefile.target since the headers are generated in the top level and that make will remain recursive.

Signature · 2 lines
Daniel Jacobowitz
CodeSourcery
#4 lore raw
Quoted 12 lines · Daniel Jacobowitz
Quoted 7 lines · Juan Quintela

Hi

This series fixes the parallel build of qemu. I tried to get the best of Daniel and Stefan ideas with mine.
Once there fixed an obvious fix.

Daniel, can you check that parallel build also works for you? It works for me (TM).

No, it doesn't work.

GEN config-all-devices.mak

Quoted 8 lines
  GEN   config-host.h
  GEN   config-all-devices.h
  CC    qemu-tool.o
  CC    cutils.o

In file included from /scratch/dan/qemu/src/qemu-trunk/qemu-tool.c:14:
/scratch/dan/qemu/src/qemu-trunk/qemu-common.h:32:25: error:
config-host.h: No such file or directory
make: *** [qemu-tool.o] Error 1

How are you compiling?
It works for me compiling in-tree with make -j3 (only 2 cores)

Quoted 3 lines

You have to define GENERATED_HEADERS before you use it in dependencies, I believe. But you've got it way down below the rules.mak invocation.

Will try it.

Quoted 1 line

Also, your changes to $(QEMU_PROG) and recurse-all are not necessary;

before recurse-all is necesary, because the sub-targets use config-host.h (they don't use config-all-devices.h, but it don't hurt to also generate it).

Quoted 8 lines

they're the same sort of ordering assumption that caused the original bug. You only need to depend on generated files if you're going to use them in this rule's commands; for headers, that means invoke a .c to .o compilation. The dependencies should be on exactly the rules that need them.

I didn't bother with Makefile.target since the headers are generated in the top level and that make will remain recursive.

Not all of them.
configure-target.h and config-device.h is generated there.

</me studies still more the Makefiles>

ok, subdir-% already depends on them.

I am sending a new patch with you suggestions, it works for me (again).

on-tree and out-of-tree compilation with make -j (still on my laptop with 2 cores).

Can you test my new series?

Later, Juan.

#5 lore raw
Quoted 2 lines · Juan Quintela

How are you compiling?
It works for me compiling in-tree with make -j3 (only 2 cores)

I can reliably reproduce it by building all my QEMU configs at once. I think that's a four-core or eight-core machine, -j4 in each build, six separate qemu build trees. I've reproduced it in a single build tree a couple of times, but not reliably.

Quoted 5 lines
Quoted 1 line

Also, your changes to $(QEMU_PROG) and recurse-all are not necessary;

before recurse-all is necesary, because the sub-targets use config-host.h (they don't use config-all-devices.h, but it don't hurt to also generate it).

Oh, you're right.

Signature · 2 lines
Daniel Jacobowitz
CodeSourcery
#6 lore raw
Quoted 8 lines · Daniel Jacobowitz
Quoted 2 lines · Juan Quintela

How are you compiling?
It works for me compiling in-tree with make -j3 (only 2 cores)

I can reliably reproduce it by building all my QEMU configs at once. I think that's a four-core or eight-core machine, -j4 in each build, six separate qemu build trees. I've reproduced it in a single build tree a couple of times, but not reliably.

I have no way to reproduce reliablely :(

If last patch didn't work for you will try in a more powerful machine.

Later, Juan.

#7 lore raw
Quoted 11 lines · Juan Quintela
Quoted 8 lines · Daniel Jacobowitz
Quoted 2 lines · Juan Quintela

How are you compiling?
It works for me compiling in-tree with make -j3 (only 2 cores)

I can reliably reproduce it by building all my QEMU configs at once. I think that's a four-core or eight-core machine, -j4 in each build, six separate qemu build trees. I've reproduced it in a single build tree a couple of times, but not reliably.

I have no way to reproduce reliablely :(

rm -f qemu-tool.[od] config-host.h
make qemu-tool.o

Reproduces reliably.
The following patch should expose any similar failures:

rules.mak +0 -0
@@ -44,5 +44,6 @@ cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc 

/dev/

        @test -f $@ || cp $< $@

%.h-timestamp: %.mak
+ sleep 60

        $(call quiet-command, $(SRC_PATH)/create_config < $< > $@, "  GEN   
$*.h
        @cmp $@ $*.h >/dev/null 2>&1 || cp $@ $*.h

Paul

#8 lore raw
Quoted 11 lines · Paul Brook
Quoted 10 lines · Juan Quintela
Quoted 8 lines · Daniel Jacobowitz
Quoted 2 lines · Juan Quintela

How are you compiling?
It works for me compiling in-tree with make -j3 (only 2 cores)

I can reliably reproduce it by building all my QEMU configs at once. I think that's a four-core or eight-core machine, -j4 in each build, six separate qemu build trees. I've reproduced it in a single build tree a couple of times, but not reliably.

Quoted 7 lines
Quoted 1 line

I have no way to reproduce reliablely :(

rm -f qemu-tool.[od] config-host.h
make qemu-tool.o

Reproduces reliably.
The following patch should expose any similar failures:

My last series of build fixes serie fix this one also. They are already on staging.

Thanks for the test case, Juan.

Quoted 14 lines
rules.mak +0 -0
@@ -44,5 +44,6 @@ cc-option = $(if $(shell $(CC) $1 $2 -S -o /dev/null -xc 

/dev/

        @test -f $@ || cp $< $@

%.h-timestamp: %.mak
+ sleep 60

        $(call quiet-command, $(SRC_PATH)/create_config < $< > $@, "  GEN   
$*.h
        @cmp $@ $*.h >/dev/null 2>&1 || cp $@ $*.h

Paul