Skip to contentSkip to conversation

cef3: use PARALLEL_MAKE to control build parallelism

PATCH In review Andre McCurdy 6 participants 14 messages Nov 11, 2015 – Nov 17, 2015

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 cef3: use PARALLEL_MAKE to control build parallelism

---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +1 -1
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} cefsimple
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
Signature · 1 line
1.9.1
#1 lore raw

PATCH 2/2 chromium: use PARALLEL_MAKE to control build parallelism

---
 recipes-browser/chromium/chromium_40.0.2214.91.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/chromium_40.0.2214.91.bb +1 -1
index a9635cd..f20610c 100644
@@ -196,7 +196,7 @@ do_configure_append() {
 
 do_compile() {
         # build with ninja
-        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} chrome chrome_sandbox
+        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} chrome chrome_sandbox
 }
 
 
Signature · 1 line
1.9.1
#2 lore raw

Re: [meta-browser][PATCH 2/2] chromium: use PARALLEL_MAKE to control build parallelism

this is ok

Quoted 26 lines · Andre McCurdy
---
 recipes-browser/chromium/chromium_40.0.2214.91.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/chromium_40.0.2214.91.bb +1 -2
index a9635cd..f20610c 100644
@@ -196,7 +196,7 @@ do_configure_append() {
 
 do_compile() {
         # build with ninja
-        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} chrome chrome_sandbox
+        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} chrome chrome_sandbox
 }
 
 
--

1.9.1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#3 lore raw

Re: [meta-browser][PATCH 2/2] chromium: use PARALLEL_MAKE to control build parallelism

In reply to Andre McCurdy

Hi Andre,

Quoted 10 lines · Andre McCurdy
---
  recipes-browser/chromium/chromium_40.0.2214.91.bb | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/chromium_40.0.2214.91.bb +0 -0
index a9635cd..f20610c 100644

recipes-browser/chromium/chromium_40.0.2214.91.bb do_configure_append() { · line 200

Quoted 6 lines
recipes-browser/chromium/chromium_40.0.2214.91.bb +1 -1
@@ -196,7 +196,7 @@ do_configure_append() {
 
  do_compile() {
          # build with ninja
-        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} chrome chrome_sandbox
+        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} chrome chrome_sandbox
  }

Can you please share the chromium build times on your build-box with and without this patch? I'm just curious how well this worked for you.

Thanks and regards,
Nikolay

#4 lore raw

Re: [meta-browser][PATCH 2/2] chromium: use PARALLEL_MAKE to control build parallelism

Hi Nikolay,

Quoted 16 lines · Nikolay Dimitrov

Hi Andre,

Quoted 13 lines · Andre McCurdy
---
  recipes-browser/chromium/chromium_40.0.2214.91.bb | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/chromium_40.0.2214.91.bb +0 -0
index a9635cd..f20610c 100644
@@ -196,7 +196,7 @@ do_configure_append() {
 
Quoted 11 lines
Quoted 7 lines
recipes-browser/chromium/chromium_40.0.2214.91.bb +0 -1
  do_compile() {
          # build with ninja
-        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS}

chrome chrome_sandbox
+ ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} chrome
chrome_sandbox
}

Can you please share the chromium build times on your build-box with and without this patch? I'm just curious how well this worked for you.

By default BB_NUMBER_THREADS and PARALLEL_MAKE are both derived from the number of logical CPU cores (as detected by oe.utils.cpu_count()), so the patch should not have any effect on build times.

The patch is about correctness - it's not really right to use BB_NUMBER_THREADS for anything other than setting the maximum number of BitBake threads.

Quoted 2 lines

Thanks and regards,
Nikolay

#5 lore raw

Re: [meta-browser][PATCH 2/2] chromium: use PARALLEL_MAKE to control build parallelism

Hi Andre,

Quoted 31 lines · Andre McCurdy

Hi Nikolay,

Quoted 16 lines · Nikolay Dimitrov

Hi Andre,

Quoted 13 lines · Andre McCurdy
---
   recipes-browser/chromium/chromium_40.0.2214.91.bb | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/chromium_40.0.2214.91.bb +0 -0
index a9635cd..f20610c 100644
@@ -196,7 +196,7 @@ do_configure_append() {
 
Quoted 11 lines
Quoted 7 lines
recipes-browser/chromium/chromium_40.0.2214.91.bb +0 -1
   do_compile() {
           # build with ninja
-        ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS}

chrome chrome_sandbox
+ ninja -C ${S}/out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} chrome
chrome_sandbox
}

Can you please share the chromium build times on your build-box with and without this patch? I'm just curious how well this worked for you.

Quoted 7 lines

By default BB_NUMBER_THREADS and PARALLEL_MAKE are both derived from the number of logical CPU cores (as detected by oe.utils.cpu_count()), so the patch should not have any effect on build times.

The patch is about correctness - it's not really right to use BB_NUMBER_THREADS for anything other than setting the maximum number of BitBake threads.

Ahh, I see. Thanks.

Regards,
Nikolay

#6 lore raw
In reply to Andre McCurdy
Quoted 10 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +0 -0
index d827b8e..6872e5b 100644

recipes-browser/chromium/cef3_280796.bb do_qa_configure() { · line 64

Quoted 6 lines
recipes-browser/chromium/cef3_280796.bb +1 -1
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} cefsimple
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }

this is ok

Quoted 8 lines

--
1.9.1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#7 lore raw
In reply to Andre McCurdy

Hi,

isn't ninja picking up the number of cores separately? I'm not sure why this is needed.

Z

Quoted 26 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +1 -2
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} cefsimple
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
--

1.9.1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Signature · 5 lines
Zoltan Kuscsik

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
#8 lore raw
Quoted 5 lines · Zoltan Kuscsik

Hi,

isn't ninja picking up the number of cores separately? I'm not sure why this is needed.

thats another issue. This one atleast improves the current usage a nit.

Quoted 45 lines

Z

Quoted 20 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +1 -2
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} cefsimple
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
--

1.9.1

Quoted 6 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Zoltan Kuscsik

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
--
_______________________________________________
Openembedded-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#9 lore raw
In reply to Zoltan Kuscsik
Quoted 4 lines · Zoltan Kuscsik

Hi,

isn't ninja picking up the number of cores separately? I'm not sure why this is needed.

Without any -j option, ninja will default to "-j (logical_cores + 2)".

The PARALLEL_MAKE value configured in OE isn't just about maximising parallelism though (although that's it's common usage), it's also useful to limit parallelism and reduce the load on the CPU. If PARALLEL_MAKE isn't passed to ninja builds then they will always fully load the CPU.

Quoted 44 lines

Z

Quoted 20 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +1 -2
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS} cefsimple
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
--

1.9.1

Quoted 6 lines

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Zoltan Kuscsik

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
--
_______________________________________________
Openembedded-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

#10 lore raw

Fair enough.

Z

Quoted 66 lines · Andre McCurdy
Quoted 4 lines · Zoltan Kuscsik

Hi,

isn't ninja picking up the number of cores separately? I'm not sure why this is needed.

Without any -j option, ninja will default to "-j (logical_cores + 2)".

The PARALLEL_MAKE value configured in OE isn't just about maximising parallelism though (although that's it's common usage), it's also useful to limit parallelism and reduce the load on the CPU. If PARALLEL_MAKE isn't passed to ninja builds then they will always fully load the CPU.

Quoted 17 lines

Z

Quoted 13 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +0 -0
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
Quoted 2 lines

recipes-browser/chromium/cef3_280796.bb do_qa_configure() { · line 63

Quoted 2 lines
recipes-browser/chromium/cef3_280796.bb +0 -1
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS}

cefsimple

Quoted 17 lines
Quoted 11 lines
recipes-browser/chromium/cef3_280796.bb +1 -1
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
--

1.9.1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Zoltan Kuscsik

Quoted 8 lines

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
--
_______________________________________________
Openembedded-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Signature · 5 lines
Zoltan Kuscsik

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
#11 lore raw

I've added -j in http://patches.openembedded.org/patch/100133/, because chromium recipe wasn't fair to other recipes. For instance, if bitbake is processing 8 compile tasks (incl. chromium) in parallel on a 24 cores machine, 7 recipes will be compiled with -j${BB_NUMBER_THREADS} whereas chromium will use something like -j24

Just out of my curiosity, what "${PARALLEL_MAKE}" brings compared to "-j${BB_NUMBER_THREADS}" ?

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Zoltan Kuscsik
Sent: jeudi 12 novembre 2015 07:08
To: openembedded-devel <[email protected]>
Subject: Re: [oe] [meta-browser][PATCH 1/2] cef3: use PARALLEL_MAKE to control build parallelism

Fair enough.

Z

Quoted 66 lines · Andre McCurdy
Quoted 4 lines · Zoltan Kuscsik

Hi,

isn't ninja picking up the number of cores separately? I'm not sure why this is needed.

Without any -j option, ninja will default to "-j (logical_cores + 2)".

The PARALLEL_MAKE value configured in OE isn't just about maximising parallelism though (although that's it's common usage), it's also useful to limit parallelism and reduce the load on the CPU. If PARALLEL_MAKE isn't passed to ninja builds then they will always fully load the CPU.

Quoted 17 lines

Z

Quoted 13 lines · Andre McCurdy
---
 recipes-browser/chromium/cef3_280796.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
recipes-browser/chromium/cef3_280796.bb +0 -0
index d827b8e..6872e5b 100644
@@ -60,5 +60,5 @@ do_qa_configure() {
 }
 
Quoted 2 lines

recipes-browser/chromium/cef3_280796.bb do_qa_configure() { · line 63

Quoted 2 lines
recipes-browser/chromium/cef3_280796.bb +0 -1
 do_compile() {
-    ninja -C out/${CHROMIUM_BUILD_TYPE} -j${BB_NUMBER_THREADS}

cefsimple

Quoted 17 lines
Quoted 11 lines
recipes-browser/chromium/cef3_280796.bb +1 -1
+    ninja -C out/${CHROMIUM_BUILD_TYPE} ${PARALLEL_MAKE} cefsimple
 }
--

1.9.1

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
Zoltan Kuscsik

Quoted 8 lines

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
--
_______________________________________________
Openembedded-devel mailing list [email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

--
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Signature · 10 lines
Zoltan Kuscsik

Senior Software Developer, Linaro Digital Home Group
email: [email protected]
IRC: zoli_ on irc.linaro.org
-- 
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel
#12 lore raw
Quoted 5 lines · Julien Brianceau (jbriance)

I've added -j in http://patches.openembedded.org/patch/100133/, because chromium recipe wasn't fair to other recipes. For instance, if bitbake is processing 8 compile tasks (incl. chromium) in parallel on a 24 cores machine, 7 recipes will be compiled with -j${BB_NUMBER_THREADS} whereas chromium will use something like -j24

Just out of my curiosity, what "${PARALLEL_MAKE}" brings compared to "-j${BB_NUMBER_THREADS}" ?

BB_NUMBER_THREADS -> max bitbake tasks run at the same time PARALLEL_MAKE -> make tasks (gcc) at the same time

Andrease

#13 lore raw

That makes sense, thanks for the clarification.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Andreas Müller
Sent: jeudi 12 novembre 2015 10:34
To: [email protected]
Subject: Re: [oe] [meta-browser][PATCH 1/2] cef3: use PARALLEL_MAKE to control build parallelism

Quoted 5 lines · Julien Brianceau (jbriance)

I've added -j in http://patches.openembedded.org/patch/100133/, because chromium recipe wasn't fair to other recipes. For instance, if bitbake is processing 8 compile tasks (incl. chromium) in parallel on a 24 cores machine, 7 recipes will be compiled with -j${BB_NUMBER_THREADS} whereas chromium will use something like -j24

Just out of my curiosity, what "${PARALLEL_MAKE}" brings compared to "-j${BB_NUMBER_THREADS}" ?

BB_NUMBER_THREADS -> max bitbake tasks run at the same time PARALLEL_MAKE -> make tasks (gcc) at the same time

Andrease

Signature · 4 lines
_______________________________________________
Openembedded-devel mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-devel