Skip to contentSkip to conversation

btrfs-progs: Add dependencies explicitly to fix a parallel build issue

PATCH In review [email protected] 3 participants 10 messages Sep 18, 2013 – Sep 23, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 btrfs-progs: Add dependencies explicitly to fix a parallel build issue

From: Roy Li <[email protected]>

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"

---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.

 Makefile |    6 ++++++
 1 file changed, 6 insertions(+)
Makefile +6 -0
index c43cb68..a7c259c 100644
@@ -81,6 +81,12 @@ endif
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<
 
+btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h
+btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h
+btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h
+btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h
+btrfs-debug-tree.o:version.h
+
 %.static.o: %.c
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
Signature · 1 line
1.7.10.4
#1 lore raw

PATCH 1/1 btrfs-progs: Add dependencies explicitly to fix a parallel build issue

Quoted 43 lines · On 9/17/13 8:11 PM, [email protected] wrote:

From: Roy Li <[email protected]>

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"

---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.

 Makefile |    6 ++++++
 1 file changed, 6 insertions(+)
Makefile +0 -0
index c43cb68..a7c259c 100644
@@ -81,6 +81,12 @@ endif
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<

Makefile endif · line 92

Quoted 10 lines
Makefile +6 -0
 
+btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h
+btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h
+btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h
+btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h
+btrfs-debug-tree.o:version.h
+
 %.static.o: %.c
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@

I think this can be done more cleanly, I'll send a patch.

-Eric

#2 lore raw
Quoted 8 lines · Eric Sandeen
Quoted 3 lines

@echo " [CC] $@"

Quoted 2 lines

$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@

I think this can be done more cleanly, I'll send a patch.

-Eric

Thanks, expect your patch

-Roy

Signature · 2 lines
Best Reagrds,
Roy | RongQing Li
#3 lore raw
In reply to Eric Sandeen
Quoted 57 lines · Eric Sandeen
Quoted 43 lines · On 9/17/13 8:11 PM, [email protected] wrote:

From: Roy Li <[email protected]>

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"

---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.

  Makefile |    6 ++++++
  1 file changed, 6 insertions(+)
Makefile +0 -0
index c43cb68..a7c259c 100644
@@ -81,6 +81,12 @@ endif
  	@echo "    [CC]     $@"
  	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<

Makefile endif · line 92

Quoted 10 lines
Makefile +6 -0
 
+btrfs-corrupt-block.o btrfs.o btrfs-image.o cmds-filesystem.o:version.h
+btrfs-show-super.o btrfs-select-super.o cmds-restore.o:version.h
+btrfs-find-root.o mkfs.o btrfs-zero-log.o btrfs-defrag.o cmds-chunk.o:version.h
+btrfstune.o btrfs-calc-size.o btrfs-map-logical.o cmds-check.o:version.h
+btrfs-debug-tree.o:version.h
+
  %.static.o: %.c
  	@echo "    [CC]     $@"
  	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@

I think this can be done more cleanly, I'll send a patch.

Quoted 3 lines

-Eric

If this one is not cleanly, how is the below one

From a56ac083a789605904507b602d9ebf196ae6746d Mon Sep 17 00:00:00 2001
From: Roy Li <[email protected]>
Date: Mon, 23 Sep 2013 09:53:30 +0800
Subject: [PATCH] btrfs-progs: Add dependencies explicitly to fix a parallel
build issue

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"
---
 Makefile |    2 ++
 1 file changed, 2 insertions(+)
Makefile +2 -0
index c43cb68..901ffa2 100644
@@ -81,6 +81,8 @@ endif
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<
 
+%.o: version.h
+
 %.static.o: %.c
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
Signature · 8 lines
1.7.10.4




-- 
Best Reagrds,
Roy | RongQing Li
#4 lore raw

PATCH 1/1 btrfs-progs: Add version.h dependency to fix parallel builds

In reply to [email protected]

Change the suffix rule to ensure that version.h is built before we try to create any .o file.

---

I dunno, does this work? My gnu make is rusty.

Makefile +2 -2
index 3d715d8..d25054f 100644
@@ -107,13 +107,13 @@ endif
 %.o.d: %.c
 	$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<
 
-.c.o:
+%.o: %.c version.h
 	@$(check_echo) "    [SP]     $<"
 	$(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $<
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $<
 
-%.static.o: %.c
+%.static.o: %.c version.h
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
 
#5 lore raw

Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds

Quoted 8 lines · Eric Sandeen

Change the suffix rule to ensure that version.h is built before we try to create any .o file.

---

I dunno, does this work? My gnu make is rusty.

I'm actually confused by what the makefile is doing with the dependency generation, maybe my patch isn't right.

this:

%.o.d: %.c

        $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<

completely confuses me :)

-Eric

Quoted 25 lines
Makefile +2 -3
index 3d715d8..d25054f 100644
@@ -107,13 +107,13 @@ endif
 %.o.d: %.c
 	$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<
 
-.c.o:
+%.o: %.c version.h
 	@$(check_echo) "    [SP]     $<"
 	$(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $<
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $<
 
-%.static.o: %.c
+%.static.o: %.c version.h
 	@echo "    [CC]     $@"
 	$(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
 
--

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw

Re: [PATCH] btrfs-progs: Add version.h dependency to fix parallel builds

Quoted 13 lines · Eric Sandeen
Quoted 8 lines · Eric Sandeen

Change the suffix rule to ensure that version.h is built before we try to create any .o file.

---

I dunno, does this work? My gnu make is rusty.

I'm actually confused by what the makefile is doing with the dependency generation, maybe my patch isn't right.

Quoted 8 lines

this:

%.o.d: %.c

         $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<

completely confuses me :)

-Eric

It lead to the below building error:

n file included from ctree.c:18:0:

| ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory
|  #include <btrfs/list.h>
|                         ^
| compilation terminated.
| In file included from extent-tree.c:22:0:
| radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory
|  #include <btrfs/kerncompat.h>
|                               ^
| compilation terminated.
| In file included from disk-io.c:29:0:
| radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory
|  #include <btrfs/kerncompat.h>
|                               ^
| compilation terminated.
| In file included from radix-tree.c:40:0:
| radix-tree.h:43:30: fatal error: btrfs/kerncompat.h: No such file or directory
|  #include <btrfs/kerncompat.h>
|                               ^
| compilation terminated.
| In file included from root-tree.c:19:0:
| ctree.h:30:24: fatal error: btrfs/list.h: No such file or directory
|  #include <btrfs/list.h>
|                         ^
| compilation terminated.

-Roy

Quoted 29 lines
Quoted 26 lines
Makefile +2 -3
index 3d715d8..d25054f 100644
@@ -107,13 +107,13 @@ endif
  %.o.d: %.c
  	$(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<
 
-.c.o:
+%.o: %.c version.h
  	@$(check_echo) "    [SP]     $<"
  	$(Q)$(check) $(AM_CFLAGS) $(CFLAGS) $<
  	@echo "    [CC]     $@"
  	$(Q)$(CC) $(AM_CFLAGS) $(CFLAGS) -c $<
 
-%.static.o: %.c
+%.static.o: %.c version.h
  	@echo "    [CC]     $@"
  	$(Q)$(CC) $(AM_CFLAGS) $(STATIC_CFLAGS) -c $< -o $@
 
--

To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Best Reagrds,
Roy | RongQing Li
#7 lore raw
In reply to [email protected]
Quoted 25 lines · On Wed, Sep 18, 2013 at 09:11:01AM +0800, [email protected] wrote:

From: Roy Li <[email protected]>

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"
Quoted 5 lines

---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.

I'm running make -j all the time but haven't seen the build fail due to missing version.h for a long time. With this patch or the previous one or with Eric's, all fine. The generated dependency files contain version.h so make has complete information how to order the rules.

The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.

If you think the patch "btrfs-progs: fix parallel build" does not help much I'll drop it.

david

#8 lore raw
Quoted 32 lines · David Sterba
Quoted 25 lines · On Wed, Sep 18, 2013 at 09:11:01AM +0800, [email protected] wrote:

From: Roy Li <[email protected]>

The dependencies of "all: version.h" or other similar ones can not fix the parallel build failure, only reduce the times; In fact, many *.o files require version.h file.

	#grep '#include "version.h"' ./ -r
	./btrfs-corrupt-block.c:#include "version.h"
	./btrfs.c:#include "version.h"
	./btrfs-image.c:#include "version.h"
	./cmds-filesystem.c:#include "version.h"
	./btrfs-show-super.c:#include "version.h"
	./btrfs-select-super.c:#include "version.h"
	./cmds-restore.c:#include "version.h"
	./btrfs-find-root.c:#include "version.h"
	./mkfs.c:#include "version.h"
	./btrfs-zero-log.c:#include "version.h"
	./btrfs-defrag.c:#include "version.h"
	./cmds-chunk.c:#include "version.h"
	./btrfstune.c:#include "version.h"
	./btrfs-calc-size.c:#include "version.h"
	./btrfs-map-logical.c:#include "version.h"
	./cmds-check.c:#include "version.h"
	./btrfs-debug-tree.c:#include "version.h"
Quoted 5 lines

---
Sorry, The patch [btrfs-progs: fix parallel build] sent by me on Sep 3
can not fix the build failure, when build enough times on a 16 core cpu, the build failure happens again, so I refix it again.

Quoted 5 lines

I'm running make -j all the time but haven't seen the build fail due to missing version.h for a long time. With this patch or the previous one or with Eric's, all fine. The generated dependency files contain version.h so make has complete information how to order the rules.

I want to know how many cores your cpu has? I can not reproduce it on my 2 cores cpu, but it always happens when run on a server which is a 16 cores cpu and "make -j20"

Quoted 3 lines

The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.

Do you means the below:

.c.o:

        $(Q)$(check) $<
        @echo "    [CC]     $@"
        $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<

-Roy

Quoted 7 lines

If you think the patch "btrfs-progs: fix parallel build" does not help much I'll drop it.

david

Signature · 2 lines
Best Reagrds,
Roy | RongQing Li
#9 lore raw
Quoted 3 lines · Rongqing Li

I want to know how many cores your cpu has? I can not reproduce it on my 2 cores cpu, but it always happens when run on a server which is a 16 cores cpu and "make -j20"

Depends on what you call a core, I've tested this on a box with 8 cpu cores and 64 logical cpus. I can clearly see that the build is stalled for a few moments at '[SH] version.h' before it proceeds to '[CC] ...'.

Quoted 9 lines
Quoted 2 lines

The dependency files are generated by an implicit rule .c -> .o.d, so there should be no problem for any of the files listed above.

Do you means the below:

.c.o:

        $(Q)$(check) $<
        @echo "    [CC]     $@"
        $(Q)$(CC) $(DEPFLAGS) $(AM_CFLAGS) $(CFLAGS) -c $<

Rather

%.o.d: %.c

        $(Q)$(CC) -MM -MG -MF $@ -MT $(@:.o.d=.o) -MT $(@:.o.d=.static.o) -MT $@ $(AM_CFLAGS) $(CFLAGS) $<

the .o.d files are included at the end of Makefile, I'm not completely sure that they get included before the .c.o rule is processed. That way the .o.d files would be empty and the dependency on version.h missing.

david