Skip to contentSkip to conversation

Parallel make messes up post-build in -next

Buildroot Eric Le Bihan 4 participants 14 messages Feb 20, 2014 Kbuild

lore
Share & tools
lore

Eric Le Bihan

author
#0 lore raw

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Best regards,
ELB

#1 lore raw
Quoted 10 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

I haven't tried it myself yet.
But, the rootfs and post build scripts are done from target-finalize, and the rules to make the rootfs depend on target-finalize. This should be fine (but probably is not and we're overlooking something).

However, aside from the above I have another doubt: I don't see any special handling for the other special targets like target-purgelocales, target-generatelocales, ... These targets should be run after all generic targets, and before creating the rootfs. As far as I can see, they could now run in parallel with the normal targets, right?

Thanks,
Thomas

#2 lore raw
Quoted 18 lines · Thomas De Schampheleire
Quoted 10 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

I haven't tried it myself yet.
But, the rootfs and post build scripts are done from target-finalize, and the rules to make the rootfs depend on target-finalize. This should be fine (but probably is not and we're overlooking something).

Quoted 6 lines

However, aside from the above I have another doubt: I don't see any special handling for the other special targets like target-purgelocales, target-generatelocales, ... These targets should be run after all generic targets, and before creating the rootfs. As far as I can see, they could now run in parallel with the normal targets, right?

Yes, it's a thing to fix for proper parallel bulding support, it's on my todo list.

The targets target-finalize, target-purgelocales, target-generatelocales, toolchain-eclipse-register can be executed in parallel after building the generic targets?

Regards

Signature · 1 line
Fabio Porcedda
#3 lore raw
Quoted 27 lines · Fabio Porcedda
Quoted 18 lines · Thomas De Schampheleire
Quoted 10 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

I haven't tried it myself yet.
But, the rootfs and post build scripts are done from target-finalize, and the rules to make the rootfs depend on target-finalize. This should be fine (but probably is not and we're overlooking something).

Quoted 6 lines

However, aside from the above I have another doubt: I don't see any special handling for the other special targets like target-purgelocales, target-generatelocales, ... These targets should be run after all generic targets, and before creating the rootfs. As far as I can see, they could now run in parallel with the normal targets, right?

Quoted 6 lines

Yes, it's a thing to fix for proper parallel bulding support, it's on my todo list.

The targets target-finalize, target-purgelocales, target-generatelocales, toolchain-eclipse-register can be executed in parallel after building the generic targets?

I think target-finalize should depend on all other target-*. It's
something that you may want to call from the command line in order to get an up-to-date target directory.

toolchain-eclipse-register can be done in parallel with all the rest, it
should just depend on toolchain. There is nothing really that depends on it, so probably 'world' is the best choice.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#4 lore raw
In reply to Fabio Porcedda

Hi Fabio,

On Thu, Feb 20, 2014 at 3:13 PM, Fabio Porcedda
<[email protected]> wrote:
[..]

Quoted 7 lines

Yes, it's a thing to fix for proper parallel bulding support, it's on my todo list.

The targets target-finalize, target-purgelocales, target-generatelocales, toolchain-eclipse-register can be executed in parallel after building the generic targets?

Since you can select both purgelocales and generatelocales, I would expect they need to be sequential: first purge then generate (as done today).
I'd also expect that target-finalize should be finished before starting the locale stuff, as the post-build script or overlay may add additional locale stuff that needs to be purged.

Best regards,
Thomas

#5 lore raw
Quoted 9 lines · Thomas De Schampheleire

Hi Fabio,

On Thu, Feb 20, 2014 at 3:13 PM, Fabio Porcedda
<[email protected]> wrote:
[..]

Quoted 4 lines

Yes, it's a thing to fix for proper parallel bulding support, it's on my todo list.

Quoted 7 lines
Quoted 3 lines

The targets target-finalize, target-purgelocales, target-generatelocales, toolchain-eclipse-register can be executed in parallel after building the generic targets?

Since you can select both purgelocales and generatelocales, I would expect they need to be sequential: first purge then generate (as done today).

True.

Quoted 3 lines

I'd also expect that target-finalize should be finished before starting the locale stuff, as the post-build script or overlay may add additional locale stuff that needs to be purged.

Hm, I would actually expect the reverse: whatever my post-build script
or overlay does should be final, and not be removed again by purgelocales. But that is in fact not what happens at the moment.

Regards,
Arnout

Quoted 4 lines

Best regards,
Thomas

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#6 lore raw
Quoted 20 lines · Arnout Vandecappelle
Quoted 9 lines · Thomas De Schampheleire

Hi Fabio,

On Thu, Feb 20, 2014 at 3:13 PM, Fabio Porcedda
<[email protected]> wrote:
[..]

Quoted 4 lines

Yes, it's a thing to fix for proper parallel bulding support, it's on my todo list.

Quoted 7 lines
Quoted 3 lines

The targets target-finalize, target-purgelocales, target-generatelocales, toolchain-eclipse-register can be executed in parallel after building the generic targets?

Since you can select both purgelocales and generatelocales, I would expect they need to be sequential: first purge then generate (as done today).

True.

Quoted 7 lines
Quoted 3 lines

I'd also expect that target-finalize should be finished before starting the locale stuff, as the post-build script or overlay may add additional locale stuff that needs to be purged.

Hm, I would actually expect the reverse: whatever my post-build script
or overlay does should be final, and not be removed again by purgelocales. But that is in fact not what happens at the moment.

I also had the same doubt, but what I wrote is just to mimic the non-parallel behavior.
The changing of the order of locale stuff and post-build is not related to the parallel build.

Best regards,
Thomas

#7 lore raw
In reply to Eric Le Bihan
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig
$(TARGET_DIR)/usr/share/pkgconfig \
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi

-$(TARGETS_ROOTFS): target-finalize
-
# See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads

 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

Regards

Signature · 1 line
Fabio Porcedda
author
#8 lore raw

Hi!

Quoted 32 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile endif · line 541

Quoted 8 lines
Makefile +0 -2
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
 
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

The patch fixes the issue. Thank you very much!

Best regards,
ELB

#9 lore raw
Quoted 38 lines · Eric Le Bihan

Hi!

Quoted 35 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile +0 -0
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
Quoted 6 lines

Makefile endif · line 541

Quoted 5 lines
Makefile +0 -2
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

The patch fixes the issue. Thank you very much!

That's great, i will send shortly a patch to fix this issue.

Thanks for reporting and testing.

Sorry for the trouble.

Regards

Signature · 1 line
Fabio Porcedda
#10 lore raw
In reply to Fabio Porcedda

Hi Fabio,

Quoted 33 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile endif · line 541

Quoted 8 lines
Makefile +0 -2
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
 
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

Why do you think this would change anything?
As far as I know, the order of the dependency expression does not matter...

#11 lore raw

Hi,

Quoted 39 lines · Thomas De Schampheleire

Hi Fabio,

Quoted 35 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile +0 -0
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
Quoted 10 lines

Makefile endif · line 541

Quoted 6 lines
Makefile +0 -2
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

Why do you think this would change anything?
As far as I know, the order of the dependency expression does not matter...

I spoke too soon.
There was indeed an actual bug in the original patch, adding a new rule in between the target-finalize rule. It was not so obvious because of the many empty lines in target-finalize, though.

Fabio, will you send a patch for this?

Thanks,
Thomas

#12 lore raw
Quoted 54 lines · Thomas De Schampheleire

Hi,

Quoted 39 lines · Thomas De Schampheleire

Hi Fabio,

Quoted 35 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile +0 -0
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
Quoted 10 lines

Makefile endif · line 541

Quoted 6 lines
Makefile +0 -2
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

Why do you think this would change anything?
As far as I know, the order of the dependency expression does not matter...

Quoted 6 lines

I spoke too soon.
There was indeed an actual bug in the original patch, adding a new rule in between the target-finalize rule. It was not so obvious because of the many empty lines in target-finalize, though.

Fabio, will you send a patch for this?

That's for sure!

Regards

Signature · 1 line
Fabio Porcedda
#13 lore raw
In reply to Thomas De Schampheleire
Quoted 39 lines · Thomas De Schampheleire

Hi Fabio,

Quoted 35 lines · Fabio Porcedda
Quoted 11 lines · Eric Le Bihan

Hi!

I am currently testing the -next branch and I've encountered the following
problem: the copy of my overlay and the execution of my post-build script is now performed *after* the generation of my image.

Reverting commit a24877586a566e052897e50e6a0c2f53cfb029f5, which adds support for top-level parallel make, fixes the issue.

Has anyone else experienced this problem?

Can you try this patch?

Makefile +0 -0
index 20aabd6..de568e3 100644
@@ -507,6 +507,8 @@ endif
 STRIP_FIND_CMD += -type f \( -perm /111 -o -name '*.so*' \)
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so*

$(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print

+$(TARGETS_ROOTFS): target-finalize
+
target-finalize: $(TARGETS)

        rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
                $(TARGET_DIR)/usr/lib/pkgconfig

$(TARGET_DIR)/usr/share/pkgconfig \

Makefile +0 -0
@@ -534,8 +536,6 @@ endif
                find $(TARGET_DIR)/lib/modules -type f -name '*.ko' | \
                xargs -r $(KSTRIPCMD); fi
Quoted 10 lines

Makefile endif · line 541

Quoted 6 lines
Makefile +0 -2
-$(TARGETS_ROOTFS): target-finalize
-
 # See http://sourceware.org/gdb/wiki/FAQ, "GDB does not see any threads
 # besides the one in which crash occurred; or SIGTRAP kills my program when
 # I set a breakpoint"

Why do you think this would change anything?
As far as I know, the order of the dependency expression does not matter...

It's my fault for putting the new rule inside another target so i splitted the previous target...
I was confused by the empty line inside the previous rule, it was my fault. It is a bug introduced in my latest revision of the patch set :(

Regards

Signature · 1 line
Fabio Porcedda