Skip to contentSkip to conversation

bitbake.conf: set the default pigz thread

PATCH In review Yu, Mingli 2 participants 5 messages Aug 11, 2020

lore
Share & tools
lore

Yu, Mingli

author
#0 lore raw

PATCH 1/1 bitbake.conf: set the default pigz thread

From: Mingli Yu <[email protected]>

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

do_package uses pigz to compress data to speed up the time if there is pigz available on the build server.

But for some big package such as clang, there comes below
error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

So set the default pigz thread and the user also can customize the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3" for the individual package in local.conf to avoid eating so much memory.

---
 meta/conf/bitbake.conf | 3 +++
 1 file changed, 3 insertions(+)
meta/conf/bitbake.conf +3 -0
index 353caacef9..bc87e78d78 100644
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"
+
 ##################################################################
 # Magic Cookie for SANITY CHECK
 ##################################################################
Signature · 1 line
2.26.2
#1 lore raw
Quoted 31 lines · Mingli

From: Mingli Yu <[email protected]>

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

do_package uses pigz to compress data to speed up the time if there is pigz available on the build server.

But for some big package such as clang, there comes below
error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

So set the default pigz thread and the user also can customize the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3" for the individual package in local.conf to avoid eating so much memory.

---
 meta/conf/bitbake.conf | 3 +++
 1 file changed, 3 insertions(+)
meta/conf/bitbake.conf +0 -0
index 353caacef9..bc87e78d78 100644

meta/conf/bitbake.conf XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" · line 808

Quoted 5 lines
meta/conf/bitbake.conf +2 -0
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
 XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
 XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"

Why not just set this in the cases you need it in and leave the defaults alone? You can just export it in the tasks which need it too.

This change actually makes the sstate number of cpus specific which we definitely don't want.

Cheers,

Richard

author
#2 lore raw

Hi RP,

Quoted 35 lines · Richard Purdie
Quoted 34 lines · Mingli

From: Mingli Yu <[email protected]>

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

do_package uses pigz to compress data to speed up the time if there is pigz available on the build server.

But for some big package such as clang, there comes below
error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs-linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cfe8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

So set the default pigz thread and the user also can customize the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3" for the individual package in local.conf to avoid eating so much memory.

---
  meta/conf/bitbake.conf | 3 +++
  1 file changed, 3 insertions(+)
meta/conf/bitbake.conf +0 -0
index 353caacef9..bc87e78d78 100644
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
  XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
  XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
Quoted 5 lines

meta/conf/bitbake.conf XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" · line 808

Quoted 2 lines
meta/conf/bitbake.conf +2 -0
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"

Why not just set this in the cases you need it in and leave the defaults alone? You can just export it in the tasks which need it too.

Actually I just want to use this logic in this patch altogether PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number for clang.

Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take effect.

Quoted 3 lines

This change actually makes the sstate number of cpus specific which we definitely don't want.

From my understanding, the default behavior for pigz is to take all online cores, it should be the same with export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}".

Thanks,

Quoted 6 lines

Cheers,

Richard

#3 lore raw
Quoted 51 lines · Mingli

Hi RP,

Quoted 41 lines · Richard Purdie
Quoted 40 lines · Mingli

From: Mingli Yu <[email protected]>

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

do_package uses pigz to compress data to speed up the time if there is pigz available on the build server.

But for some big package such as clang, there comes below error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs- linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cf e8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

So set the default pigz thread and the user also can customize the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3"
for the individual package in local.conf to avoid eating so much memory.

---
  meta/conf/bitbake.conf | 3 +++
  1 file changed, 3 insertions(+)
meta/conf/bitbake.conf +0 -0
index 353caacef9..bc87e78d78 100644
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least

=2)}"
XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --
threads=${XZ_THREADS}"
XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"

Quoted 6 lines

meta/conf/bitbake.conf XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" · line 808

Quoted 2 lines
meta/conf/bitbake.conf +2 -0
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"

Why not just set this in the cases you need it in and leave the defaults alone? You can just export it in the tasks which need it too.

Quoted 6 lines

Actually I just want to use this logic in this patch altogether PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number for clang.

Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take effect.

You'll have to use something like:

PIGZEXPORT = "0"
PIGZEXPORT_pn-clang = "1"
PIGZ[export] = "${PIGZEXPORT}"

Quoted 7 lines
Quoted 3 lines

This change actually makes the sstate number of cpus specific which we
definitely don't want.

From my understanding, the default behavior for pigz is to take all
online cores, it should be the same with export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}".

I understand why you want to do it and what it does. It breaks sstate signatures for everyone else though so it is not acceptable.

Cheers,

Richard

author
#4 lore raw
Quoted 69 lines · Richard Purdie
Quoted 51 lines · Mingli

Hi RP,

Quoted 41 lines · Richard Purdie
Quoted 40 lines · Mingli

From: Mingli Yu <[email protected]>

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

do_package uses pigz to compress data to speed up the time if there is pigz available on the build server.

But for some big package such as clang, there comes below error during do_package.
gzip: stdout: Cannot allocate memory
tar: TOPDIR/sstate-cache/20/80/sstate:clang:cortexa57-wrs- linux:10.0.1:r0:cortexa57:3:20808f8a746b00ec9470a81057c8595d878cf e8c565cfbcbbaa3ea870528ee34_package.tgz.QF5dss8M: Wrote only 8192 of 10240 bytes
tar: Child returned status 1
tar: Error is not recoverable: exiting now

So set the default pigz thread and the user also can customize the pigz thread such as via PIGZ = "-p 3" or PIGZ_pn-clang = "-p 3"
for the individual package in local.conf to avoid eating so much memory.

---
   meta/conf/bitbake.conf | 3 +++
   1 file changed, 3 insertions(+)
meta/conf/bitbake.conf +0 -0
index 353caacef9..bc87e78d78 100644
@@ -804,6 +804,9 @@ XZ_THREADS ?= "${@oe.utils.cpu_count(at_least

=2)}"
XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --
threads=${XZ_THREADS}"
XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"

Quoted 6 lines

meta/conf/bitbake.conf XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" · line 808

Quoted 2 lines
meta/conf/bitbake.conf +2 -0
+# Default parallelism and resource usage for pigz
+export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}"

Why not just set this in the cases you need it in and leave the defaults alone? You can just export it in the tasks which need it too.

Quoted 6 lines

Actually I just want to use this logic in this patch altogether PIGZ_pn-clang = "-p 3" in local.conf to limit the pigz thread number for clang.

Without this patch, I think PIGZ_pn-clang = "-p 3" doesn't take effect.

You'll have to use something like:

PIGZEXPORT = "0"
PIGZEXPORT_pn-clang = "1"
PIGZ[export] = "${PIGZEXPORT}"

Quoted 4 lines
Quoted 3 lines

This change actually makes the sstate number of cpus specific which we
definitely don't want.

Quoted 6 lines
Quoted 3 lines

From my understanding, the default behavior for pigz is to take all
online cores, it should be the same with export PIGZ = "-p ${@oe.utils.cpu_count(at_least=2)}".

I understand why you want to do it and what it does. It breaks sstate signatures for everyone else though so it is not acceptable.

Got it!

Thanks,

Quoted 9 lines

Cheers,

Richard