btrfs-progs: Add dependencies explicitly to fix a parallel build issue
Subsystems (2)
PATCH 1/1 btrfs-progs: Add dependencies explicitly to fix a parallel build issue
From: Roy Li <[email protected]>
The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.
#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"
---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.
Makefile | 6 ++++++ 1 file changed, 6 insertions(+)
Makefile +6 -0
@@ -81,6 +81,12 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< +btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h +btrfs-debug-tree.o:version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
Signature · 1 line
1.7.10.4
PATCH 1/1 btrfs-progs: Add dependencies explicitly to fix a parallel build issue
Quoted 43 lines · On 9/17/13 8:11 PM, [email protected] wrote:
From: Roy Li <[email protected]>
The dependencies of "all:
version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files requireversion.hfile.#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"Signed-off-by Roy Li <[email protected]>---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.Makefile | 6 ++++++ 1 file changed, 6 insertions(+)Makefile +0
-0@@ -81,6 +81,12 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<
Makefile endif · line 92
Quoted 10 lines
Makefile +6
-0+btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h +btrfs-debug-tree.o:version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
I think this can be done more cleanly, I'll send a patch.
-Eric
Quoted 8 lines · Eric Sandeen
Quoted 3 lines
@echo " [CC] $@"
Quoted 2 lines
$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
I think this can be done more cleanly, I'll send a patch.
-Eric
Thanks, expect your patch
-Roy
Signature · 2 lines
Best Reagrds, Roy | RongQing Li
Quoted 57 lines · Eric Sandeen
Quoted 43 lines · On 9/17/13 8:11 PM, [email protected] wrote:
From: Roy Li <[email protected]>
The dependencies of "all:
version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files requireversion.hfile.#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"Signed-off-by Roy Li <[email protected]>---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.Makefile | 6 ++++++ 1 file changed, 6 insertions(+)Makefile +0
-0@@ -81,6 +81,12 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<Makefile endif · line 92
Quoted 10 lines
Makefile +6
-0+btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h +btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h +btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h +btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h +btrfs-debug-tree.o:version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@I think this can be done more cleanly, I'll send a patch.
Quoted 3 lines
-Eric
If this one is not cleanly, how is the below one
From a56ac083a789605904507b602d9ebf196ae6746d Mon Sep 17 00:00:00 2001
From: Roy Li <[email protected]>
Date: Mon, 23 Sep 2013 09:53:30 +0800
Subject: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel
build issue
The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.
#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"
--- Makefile | 2 ++ 1 file changed, 2 insertions(+)
Makefile +2 -0
@@ -81,6 +81,8 @@ endif @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $< +%.o: version.h + %.static.o: %.c @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
Signature · 8 lines
1.7.10.4 -- Best Reagrds, Roy | RongQing Li
PATCH 1/1 btrfs-progs: Add version.h dependency to fix parallel builds
In reply to [email protected]Change the suffix rule to ensure that version.h is built before we try to create any .o file.
---
I dunno, does this work? My gnu make is rusty.
Makefile +2 -2
@@ -107,13 +107,13 @@ endif %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< -.c.o: +%.o: %.c version.h @$(check_echo) " [SP] $<" $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< -%.static.o: %.c +%.static.o: %.c version.h @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds
Quoted 8 lines · Eric Sandeen
Change the suffix rule to ensure that
version.his built before we try to create any .o file.Reported-by Roy Li <[email protected]>Signed-off-by Eric Sandeen <[email protected]>---
I dunno, does this work? My gnu make is rusty.
I'm actually confused by what the makefile is doing with the dependency generation, maybe my patch isn't right.
this:
%.o.d: %.c
$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<
completely confuses me :)
-Eric
Quoted 25 lines
Makefile +2
-3@@ -107,13 +107,13 @@ endif %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< -.c.o: +%.o: %.c version.h @$(check_echo) " [SP] $<" $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< -%.static.o: %.c +%.static.o: %.c version.h @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ --To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds
Quoted 13 lines · Eric Sandeen
Quoted 8 lines · Eric Sandeen
Change the suffix rule to ensure that
version.his built before we try to create any .o file.Reported-by Roy Li <[email protected]>Signed-off-by Eric Sandeen <[email protected]>---
I dunno, does this work? My gnu make is rusty.
I'm actually confused by what the makefile is doing with the dependency generation, maybe my patch isn't right.
Quoted 8 lines
this:
%.o.d: %.c
$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<completely confuses me :)
-Eric
It lead to the below building error:
n file included from ctree.c:18:0:
| ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory | #include <btrfs/list.h> | ^ | compilation terminated. | In file included from extent-tree.c:22:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from disk-io.c:29:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from radix-tree.c:40:0: | radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory | #include <btrfs/kerncompat.h> | ^ | compilation terminated. | In file included from root-tree.c:19:0: | ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory | #include <btrfs/list.h> | ^ | compilation terminated.
-Roy
Quoted 29 lines
Quoted 26 lines
Makefile +2
-3@@ -107,13 +107,13 @@ endif %.o.d: %.c $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $< -.c.o: +%.o: %.c version.h @$(check_echo) " [SP] $<" $(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $< @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $< -%.static.o: %.c +%.static.o: %.c version.h @echo " [CC] $@" $(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@ --To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Best Reagrds, Roy | RongQing Li
Quoted 25 lines · On Wed, Sep 18, 2013 at 09:11:01AM +0800, [email protected] wrote:
From: Roy Li <[email protected]>
The dependencies of "all:
version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files requireversion.hfile.#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"
Quoted 5 lines
Signed-off-by Roy Li <[email protected]>---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.
I'm running make -j all the time but haven't seen the build fail due to missing version.h for a long time. With this patch or the previous one or with Eric's, all fine. The generated dependency files contain version.h so make has complete information how to order the rules.
The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.
If you think the patch "btrfs-progs: fix parallel build" does not help much I'll drop it.
david
Quoted 32 lines · David Sterba
Quoted 25 lines · On Wed, Sep 18, 2013 at 09:11:01AM +0800, [email protected] wrote:
From: Roy Li <[email protected]>
The dependencies of "all:
version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files requireversion.hfile.#grep '#include "version.h"' ./ -r ./btrfs-corrupt-block.c:#include "version.h" ./btrfs.c:#include "version.h" ./btrfs-image.c:#include "version.h" ./cmds-filesystem.c:#include "version.h" ./btrfs-show-super.c:#include "version.h" ./btrfs-select-super.c:#include "version.h" ./cmds-restore.c:#include "version.h" ./btrfs-find-root.c:#include "version.h" ./mkfs.c:#include "version.h" ./btrfs-zero-log.c:#include "version.h" ./btrfs-defrag.c:#include "version.h" ./cmds-chunk.c:#include "version.h" ./btrfstune.c:#include "version.h" ./btrfs-calc-size.c:#include "version.h" ./btrfs-map-logical.c:#include "version.h" ./cmds-check.c:#include "version.h" ./btrfs-debug-tree.c:#include "version.h"Quoted 5 lines
Signed-off-by Roy Li <[email protected]>---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.
Quoted 5 lines
I'm running make -j all the time but haven't seen the build fail due to missing
version.hfor a long time. With this patch or the previous one or with Eric's, all fine. The generated dependency files containversion.hso make has complete information how to order the rules.
I want to know how many cores your cpu has? I can not reproduce it on my 2 cores cpu, but it always happens when run on a server which is a 16 cores cpu and "make -j20"
Quoted 3 lines
The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.
Do you means the below:
.c.o:
$(Q)$(check) $<
@echo " [CC] $@"
$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<-Roy
Quoted 7 lines
If you think the patch "btrfs-progs: fix parallel build" does not help much I'll drop it.
david
Signature · 2 lines
Best Reagrds, Roy | RongQing Li
Quoted 3 lines · Rongqing Li
I want to know how many cores your cpu has? I can not reproduce it on my 2 cores cpu, but it always happens when run on a server which is a 16 cores cpu and "make -j20"
Depends on what you call a core, I've tested this on a box with 8 cpu cores and 64 logical cpus. I can clearly see that the build is stalled for a few moments at '[SH] version.h' before it proceeds to '[CC] ...'.
Quoted 9 lines
Quoted 2 lines
The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.
Do you means the below:
.c.o:
$(Q)$(check) $< @echo " [CC] $@" $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<
Rather
%.o.d: %.c
$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<
the .o.d files are included at the end of Makefile, I'm not completely sure that they get included before the .c.o rule is processed. That way the .o.d files would be empty and the dependency on version.h missing.
david