Parallel make
When I try a parallel make, make seems to get it's dependencies out of order.
For example, here's a make oldconfig ; make -j4 output on a fresh 2.5.69
ianw@tartufi:/usr/src/ianw/linux-2.5.69$ make -j4 make -f scripts/Makefile.build obj=scripts gcc -Wp,-MD,scripts/.empty.o.d -D__KERNEL__ -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -g -O2 -fno-strict-aliasing -fno-common -pipe -ffixed-r13 -mfixed-rangeñ0-f15,f32-f127 -falign-functions2 -fomit-frame-pointer -nostdinc -iwithprefix include -mconstant-gp -DKBUILD_BASENAME=empty -DKBUILD_MODNAME=empty -c -o scripts/empty.o scripts/empty.c scripts/mk_elfconfig ia64 < scripts/empty.o > scripts/elfconfig.h gcc -Wp,-MD,scripts/.file2alias.o.d -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -c -o scripts/file2alias.o scripts/file2alias.c gcc -Wp,-MD,scripts/.modpost.o.d -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -c -o scripts/modpost.o scripts/modpost.c gcc -o scripts/modpost scripts/modpost.o scripts/file2alias.o SPLIT include/linux/autoconf.h -> include/config/* gcc -E -Wp,-MD,arch/ia64/.vmlinux.lds.s.d -D__ASSEMBLY__ -D__KERNEL__ -Iinclude -mconstant-gp -nostdinc -iwithprefix include -P -C -Uia64 -o arch/ia64/vmlinux.lds.s arch/ia64/vmlinux.lds.S In file included from arch/ia64/vmlinux.lds.S:4: include/asm/ptrace.h:56:25: asm/offsets.h: No such file or directory make: *** [arch/ia64/vmlinux.lds.s] Error 1 make: *** Waiting for unfinished jobs.... make: *** Waiting for unfinished jobs....
At first I thought it might be because the all target depends on vmlinux, but this isn't it. Currently I'm stumped. 'Single' make works fine, and if you run parallel make a few times eventually offsets.h gets generated and the build continues on fine.
Is it just me?
Quoted 4 lines · Ian Wienand
At first I thought it might be because the all target depends on vmlinux, but this isn't it. Currently I'm stumped. 'Single' make works fine, and if you run parallel make a few times eventually
offsets.hgets generated and the build continues on fine.
sorry to reply to myself but I forgot to say the path of least resistance might be to just remove ptrace.h from vmlinux.lds.S, as it doesn't seem to be needed.
-i
arch/ia64/vmlinux.lds.S 2003-05-14 14:25:32.000000000 +1000 renamed from arch/ia64/vmlinux.lds.S.old 2003-05-14 14:25:17.000000000 +1000 +0 -1
@@ -1,7 +1,6 @@ #include <linux/config.h> #include <asm/cache.h> -#include <asm/ptrace.h> #include <asm/system.h> #define LOAD_OFFSET PAGE_OFFSET
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Ian Wienand
Ian> When I try a parallel make, make seems to get it's dependencies Ian> out of order.
I see the same thing, but it's only for asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)
--david
Quoted 11 lines · David Mosberger
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Ian Wienand
Ian> When I try a parallel make, make seems to get it's dependencies Ian> out of order.
I see the same thing, but it's only for
asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)
Don't wait, do it now:
make prepare
make -j4 compressed
Eric
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Eric Piel
I see the same thing, but it's only for
asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)
Eric> Don't wait, do it now:
Eric> make prepare
Eric> make -j4 compressed
Ah, yes, I think I've seen that once mentioned in a mail.
BTW: have you gotten any feedback regarding the timer bug you reported? I think the same (rounding) bug may have broken the profil() call in libc.
--david
Quoted 4 lines · David Mosberger
BTW: have you gotten any feedback regarding the timer bug you reported? I think the same (rounding) bug may have broken the
profil()call in libc.
Yes, I'm working with George Anzinger on this issue. For now a "workaround" is to revert the functions timespec_to_jiffies()...jiffies_to_timeval() in include/linux/time.h and tstojiffie() in posix-timers.c to the versions of 2.5.67 .
Eric
Quoted 2 lines · Ian Wienand
When I try a parallel make, make seems to get it's dependencies out of order.
Could you try the following patch.
It does:
1) Remove superflous all: target, covered by the main Makefile
2) .PHONY shall only list non-file targets
3) Use make $@, $< where appropriate
4) For the offset file do not duplicate dependencies from main Makefile
Sam
=== arch/ia64/Makefile 1.42 vs edited ==--- 1.42/arch/ia64/Makefile Wed Mar 26 17:30:43 2003
+++ edited/arch/ia64/Makefile Wed May 14 19:46:03 2003
@@ -60,17 +60,15 @@
boot := arch/ia64/boot
tools := arch/ia64/tools
-.PHONY: boot compressed include/asm-ia64/offsets.h -
-all: prepare vmlinux
+.PHONY: boot compressed check
compressed: vmlinux.gz
vmlinux.gz: vmlinux
- $(Q)$(MAKE) $(build)=$(boot) vmlinux.gz
+ $(Q)$(MAKE) $(build)=$(boot) $@
check: vmlinux
- arch/ia64/scripts/unwcheck.sh vmlinux
+ arch/ia64/scripts/unwcheck.sh $<
archclean:
$(Q)$(MAKE) $(clean)=$(boot)
@@ -83,7 +81,7 @@
boot: lib/lib.a vmlinux
$(Q)$(MAKE) $(build)=$(boot) $@
-include/asm-ia64/offsets.h: include/asm include/linux/version.h include/config/MARKER +include/asm-ia64/offsets.h:
$(Q)$(MAKE) $(build)=$(tools) $@
define archhelp
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Sam Ravnborg
Quoted 2 lines · Ian Wienand
Quoted 2 lines
When I try a parallel make, make seems to get it's dependencies out of order.
Sam> Could you try the following patch.
Sam> It does: 1) Remove superflous all: target, covered by the main Sam> Makefile 2) .PHONY shall only list non-file targets 3) Use make Sam> $@, $< where appropriate 4) For the offset file do not Sam> duplicate dependencies from main Makefile
Very nice. Seems to do the trick for me.
--david
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Eric Piel
Quoted 3 lines · Eric> David Mosberger wrote:
Quoted 3 lines
BTW: have you gotten any feedback regarding the timer bug you reported? I think the same (rounding) bug may have broken the
profil()call in libc.
Eric> Yes, I'm working with George Anzinger on this issue. For now a Eric> "workaround" is to revert the functions
Eric> timespec_to_jiffies()...jiffies_to_timeval() in Eric> include/linux/time.h and tstojiffie() in posix-timers.c to the Eric> versions of 2.5.67 .
That works for me. Undoing the time.h changes does make profil() work again.
Thanks,
--david
Quoted 9 lines · David Mosberger
Quoted 5 lines
Quoted 5 lines
I see the same thing, but it's only for
asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)Eric> Don't wait, do it now:
Eric> make prepare
Eric> make -j4 compressed
(Advertorial): Now you've got make -j4, you should try ccache and distcc. They make development on big trees enormously more enjoyable. Andrew Morton and other kernel developers love them.
http://distcc.samba.org/
http://ccache.samba.org/
Signature · 1 line
Martin
Quoted 1 line · Sam Ravnborg
Could you try the following patch.
thanks, it now works perfectly.
-i
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Martin Pool
Quoted 6 lines · David Mosberger
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
I see the same thing, but it's only for
asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)
Eric> Don't wait, do it now:
Eric> make prepare
Eric> make -j4 compressed
Martin> (Advertorial): Now you've got make -j4, you should try ccache and Martin> distcc. They make development on big trees enormously more Martin> enjoyable. Andrew Morton and other kernel developers love them.
Martin> http://distcc.samba.org/
Martin> http://ccache.samba.org/
Actually, Stephane setup distcc here. Unfortunately it doesn't help me all that much, because linking is what takes a large part of my time (because I'm making incremental changes and because I insist on compiling with -g).
I have to say though I was very impressed with how easy to use distcc is. Nice job!
(BTW: still waiting for someone to tell me I need a 64-way SuperDome to get my kernel compiles done... ;-)
--david
Martin,
Quoted 11 lines · Martin Pool
Quoted 9 lines · David Mosberger
Quoted 5 lines
Quoted 5 lines
I see the same thing, but it's only for
asm-ia64/offsets.h. After that, parallel makes work just fine. I tend to be able to live with such "minor" build issues for a loooong time, giving others ample chance to get impatient first... ;-)Eric> Don't wait, do it now:
Eric> make prepare
Eric> make -j4 compressed
Quoted 7 lines
(Advertorial): Now you've got make -j4, you should try ccache and distcc. They make development on big trees enormously more enjoyable. Andrew Morton and other kernel developers love them.
Yes, distcc is really nice. I have been using it for a month or so for my kernel compiles using a cluster of Itanium 2 machines. What I like is that it is really easy to setup. The danger with the approach is that you can easily overload your local (master) host. As for kernel compiles for 2.5, there are a lot of ld steps which are still all done on the master node. Besides that you can really feel the speed difference. I need to take a look at ccache now that you mention it.
Very cool.
Signature · 1 line
-Stephane
Quoted 2 lines · Martin Pool
(Advertorial): Now you've got make -j4, you should try ccache and distcc.
(Endorsement): Stephane and I played around with just distcc for 2.4 ia64 kernels on a smaller cluster of rx2600 (dual CPU). The "master" runs out of CPU around -j16 (8 hosts). But that resulted in 66 second kernel builds.
And distcc really is trivial to setup.
grant
Quoted 4 lines · Stephane Eranian
Yes, distcc is really nice. I have been using it for a month or so for my kernel compiles using a cluster of Itanium 2 machines. What I like is that it is really easy to setup. The danger with the approach is that you can easily overload your local (master) host.
It can work well to use -l as well as -j, to put an upper limit on the load on the master host. If you have 2 local CPUs and 8 altogether then perhaps.
make -l4 -j10
That ought to make concurrency ramp down as it gets into the linking phase.
I ought to add that to the manual/faq.
Quoted 6 lines
As for kernel compiles for 2.5, there are a lot of ld steps which are still all done on the master node. Besides that you can really feel the speed difference. I need to take a look at ccache now that you mention it.
Very cool.
Thanks, I'm glad you like it.
Signature · 1 line
Martin