compat-wireless: ridiculous build speed

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

Quoted 8 lines · Jan Kiszka

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

Quoted 14 lines · On Thursday 11 June 2009 20:48:13 Luis R. Rodriguez wrote:

Quoted 8 lines · Jan Kiszka

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

Quoted 3 lines

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

-j does not help. It will hardly build in parallel then.

There's something
that serializes most of it (implicitly).

Quoted 15 lines · Michael Buesch

Quoted 14 lines · On Thursday 11 June 2009 20:48:13 Luis R. Rodriguez wrote:

Quoted 8 lines · Jan Kiszka

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

Quoted 5 lines

Quoted 3 lines

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

-j does not help. It will hardly build in parallel then.

It works swell for me. The only thing that does not build in parellel is the autoconf stuff but after its done with that step everything builds as expected with -j 2.

Quoted 2 lines

There's something
that serializes most of it (implicitly).

You certain its just not the first part? I had written some patches to make the first autoconf step into another step to then allow make -j foo to work without a warning but that would involve another step. The -j option does work for me though.

Quoted 26 lines · On Thursday 11 June 2009 23:10:52 Luis R. Rodriguez wrote:

Quoted 15 lines · Michael Buesch

Quoted 14 lines · On Thursday 11 June 2009 20:48:13 Luis R. Rodriguez wrote:

Quoted 8 lines · Jan Kiszka

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

Quoted 5 lines

Quoted 3 lines

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

-j does not help. It will hardly build in parallel then.

It works swell for me. The only thing that does not build in parellel is the autoconf stuff but after its done with that step everything builds as expected with -j 2.

Quoted 7 lines

Quoted 2 lines

There's something
that serializes most of it (implicitly).

You certain its just not the first part? I had written some patches to make the first autoconf step into another step to then allow make -j foo to work without a warning but that would involve another step. The -j option does work for me though.

Well, I never measured it, but I don't see any advantage when using -jX. Maybe that is because forever equals forever+1.

What I do see, though, is that the per-cpu load is much higher when building the kernel itself with the same -j parameter.

Quoted 35 lines · On Thursday 11 June 2009 23:10:52 Luis R. Rodriguez wrote:

Quoted 26 lines · Michael Buesch

Quoted 15 lines · On Thursday 11 June 2009 20:48:13 Luis R. Rodriguez wrote:

Quoted 14 lines · Jan Kiszka

Quoted 8 lines

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

Quoted 5 lines

Quoted 3 lines

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

-j does not help. It will hardly build in parallel then.

It works swell for me. The only thing that does not build in parellel is the autoconf stuff but after its done with that step everything builds as expected with -j 2.

Quoted 7 lines

Quoted 2 lines

There's something
that serializes most of it (implicitly).

You certain its just not the first part? I had written some patches to make the first autoconf step into another step to then allow make -j foo to work without a warning but that would involve another step. The -j option does work for me though.

Quoted 5 lines

Well, I never measured it, but I don't see any advantage when using -jX. Maybe that is because forever equals forever+1.

What I do see, though, is that the per-cpu load is much higher when building the kernel itself with the same -j parameter.

Actually, most of the slowness comes from stage 2 of the module build. For me, stage 1 runs through rather fast (and -jX accelerates it quite well), but stage 2 is serialized, and there is a long wait between individual files. This wait is about a second on 2.6.28 for me in vmware, but over 5 seconds (!!!) on 2.6.29. I've tried multiple distros, and the time difference is consistent. (BTW stage 2 is serialized in a full kernel build as well, though there is no reason why it should be so; but no "wait" between steps is observed - in fact, the second stage of a full 2.6.29 kernel build completes faster for me than the second stage of compat-wireless on 2.6.29!)

2009/6/11 Gábor Stefanik <[email protected]>:

Quoted 36 lines · Michael Buesch

Quoted 30 lines · On Thursday 11 June 2009 23:10:52 Luis R. Rodriguez wrote:

Quoted 23 lines · Michael Buesch

Quoted 10 lines · On Thursday 11 June 2009 20:48:13 Luis R. Rodriguez wrote:

Quoted 9 lines · Jan Kiszka

Quoted 8 lines

Hi,

while watching compat-wireless-2009-06-11 doing baby-steps building all its modules, I wonder what slows this down so horribly. No other external module build package I know is that slow (I would say *at least* one order of magnitude slower than normal). Is this really required?

[ Hurray, it's finished! ]

Quoted 8 lines

Quoted 7 lines

I think the many depmod -ae are the culprit. This is in place to account for all the crap of madwifi which may be present or older drivers which have been renamed.

This could be improved. Patches are welcomed.

Ps. Try building with make -j 2, etc.

-j does not help. It will hardly build in parallel then.

It works swell for me. The only thing that does not build in parellel is the autoconf stuff but after its done with that step everything builds as expected with -j 2.

Quoted 6 lines

Quoted 2 lines

There's something
that serializes most of it (implicitly).

You certain its just not the first part? I had written some patches to make the first autoconf step into another step to then allow make -j foo to work without a warning but that would involve another step. The -j option does work for me though.

Quoted 5 lines

Well, I never measured it, but I don't see any advantage when using -jX. Maybe that is because forever equals forever+1.

What I do see, though, is that the per-cpu load is much higher when building the kernel itself with the same -j parameter.

Quoted 10 lines

Actually, most of the slowness comes from stage 2 of the module build. For me, stage 1 runs through rather fast (and -jX accelerates it quite well), but stage 2 is serialized, and there is a long wait between individual files. This wait is about a second on 2.6.28 for me in vmware, but over 5 seconds (!!!) on 2.6.29. I've tried multiple distros, and the time difference is consistent. (BTW stage 2 is serialized in a full kernel build as well, though there is no reason why it should be so; but no "wait" between steps is observed - in fact, the second stage of a full 2.6.29 kernel build completes faster for me than the second stage of compat-wireless on 2.6.29!)

Heh yeah we need to fix this, patches are welcomed, I'll treat a beer sometime.

Including config.mk again when kbuild called into the top level Makefile slowed down the build process awfully. The reason for this were all the dynamic, shell-based variable evaluations that were called over and over again. Fortunately, including config.mk only in the outer Makefile run appears to be enough. So let's remove this warp speed blocker.

Signed-off-by Jan Kiszka <[email protected]>

---
 Makefile |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

Makefile +0 -0

index 0ab7d27..49a786a 100644

Quoted 23 lines · Jan Kiszka

Including config.mk again when kbuild called into the top level Makefile slowed down the build process awfully. The reason for this were all the dynamic, shell-based variable evaluations that were called over and over again. Fortunately, including config.mk only in the outer Makefile run appears to be enough. So let's remove this warp speed blocker.

Signed-off-by Jan Kiszka <[email protected]>

---
 Makefile |    2 --

 1 files changed, 0 insertions(+), 2 deletions(-)

Makefile +0 -2

index 0ab7d27..49a786a 100644
@@ -13,8 +13,6 @@ OLD_IWL=$(shell $(MODPROBE) -l iwl4965)
 
  ifneq ($(KERNELRELEASE),)
 
-include $(M)/$(COMPAT_CONFIG)
-
  NOSTDINC_FLAGS := -I$(M)/include/ -include $(M)/include/net/compat.h $(CFLAGS)
 

Quoted 1 line

 obj-y := net/wireless/ net/mac80211/ net/rfkill/

Hi Jan,

I have tested this patch and it reduced the build time for me from 3:30 to 2:40 minutes on a dual core system. I can not see any changes, expect the reduced build time on my system. The resulting modules are still the same.

Quoted 27 lines · Hauke Mehrtens

Quoted 16 lines · Jan Kiszka

Including config.mk again when kbuild called into the top level Makefile slowed down the build process awfully. The reason for this were all the dynamic, shell-based variable evaluations that were called over and over again. Fortunately, including config.mk only in the outer Makefile run appears to be enough. So let's remove this warp speed blocker.

Signed-off-by Jan Kiszka <[email protected]>

---
 Makefile |    2 --
 1 files changed, 0 insertions(+), 2 deletions(-)

Makefile +0 -0

index 0ab7d27..49a786a 100644

Makefile OLD_IWL=$(shell $(MODPROBE) -l iwl4965) · line 18

Quoted 8 lines

Makefile +0 -2

@@ -13,8 +13,6 @@ OLD_IWL=$(shell $(MODPROBE) -l iwl4965)
 
 ifneq ($(KERNELRELEASE),)
 
-include $(M)/$(COMPAT_CONFIG)
-
 NOSTDINC_FLAGS := -I$(M)/include/ -include $(M)/include/net/compat.h $(CFLAGS)
 
 obj-y := net/wireless/ net/mac80211/ net/rfkill/

Hi Jan,

Quoted 4 lines

I have tested this patch and it reduced the build time for me from 3:30 to 2:40 minutes on a dual core system. I can not see any changes, expect the reduced build time on my system. The resulting modules are still the same.

Reason for including the config.mk is if your current kernel config differs the makefiles won't pick up the new stuff.