lib/oe/utils: allow to set a lower bound on returned cpu_count()
PATCH v2 1/4 lib/oe/utils: allow to set a lower bound on returned cpu_count()
This will be needed for making xz compression more deterministic, as xz archives are created differently in single- vs multi-threaded modes.
This means that due to bitbake's default of using as many threads as there are cores in the system, files compressed with xz will be different if built on a multi-core system compared to single-core systems.
Allowing cpu_count() here to return a lower bound, will allow forcing xz to always use multi-threaded operation.
--- meta/lib/oe/utils.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
meta/lib/oe/utils.py +3 -2
@@ -248,9 +248,10 @@ def trim_version(version, num_parts=2): trimmed = ".".join(parts[:num_parts]) return trimmed -def cpu_count(): +def cpu_count(at_least=1): import multiprocessing - return multiprocessing.cpu_count() + cpus = multiprocessing.cpu_count() + return max(cpus, at_least) def execute_pre_post_process(d, cmds): if cmds is None:
Signature · 1 line
2.23.0.rc1
PATCH v2 2/4 bitbake.conf: more deterministic xz compression (threads)
xz archives can be non-deterministic / non-reproducible:
a) archives are created differently in single- vs multi-threaded modes
b) xz will scale down the compression level so as to be try to work within any memory limit given to it when operating in single-threaded mode
This means that due to bitbake's default of using as many threads as there are cores in the system, files compressed with xz will be different if built on a multi-core system compared to single-core systems. They will also potentially be different if built on single-core systems with different amounts of physical memory, due to bitbake's default of limiting xz's memory consumption.
Force multi-threaded operation by default, even on single-core systems, so as to ensure archives are created in the same way in all cases.
--- meta/conf/bitbake.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
@@ -795,7 +795,7 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}" +XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count(at_least=2)}" ################################################################## # Magic Cookie for SANITY CHECK
Signature · 1 line
2.23.0.rc1
PATCH v2 3/4 bitbake.conf: omit XZ threads and RAM from sstate signatures
In reply to André DraszikThe number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the outcome of the compression if xz operates in multi-threaded mode [1].
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
This is in particular a problem with the opkg package writing backend, as the OPKGBUILDCMD depends on XZ_DEFAULTS. Without the vardepexclude, there is no re-use possible of the package_write_ipk sstate.
Whitelist the maximum number of threads and the memory limit given assumptions outlined in [2] below.
[1] When starting out in multi-threaded mode, the output is always deterministic, as even if xz scales down to single-threaded later, the archives are still split into blocks and size information is still added, thus keeping them compatible with multi-threaded mode. Also, when starting out in multi-threaded mode, xz never scales down the compression level to accomodate memory usage restrictions, it just scales down the number of threads and errors out if it can not accomodate the memory limit.
[2] Assumptions
* We only support multi-threaded mode (threads >= 2), builds should not try to use xz in single-threaded mode
* The thread limit should be set via XZ_THREADS, not via modifying XZ_DEFAULTS or XZ_OPTS, or any other way
* The thread limit should not be set to xz's magic value zero (0), as that will lead to single-threaded mode on single-core systems.
--- meta/conf/bitbake.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +4 -1
@@ -795,7 +795,10 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count(at_least=2)}" +XZ_MEMLIMIT ?= "50%" +XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" +XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" +XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" ################################################################## # Magic Cookie for SANITY CHECK
Signature · 1 line
2.23.0.rc1
PATCH v2 4/4 reproducible: try to ensure reproducible xz archives
In reply to André Draszikxz suffers from a reproducibility problem when not using multi- threaded mode:
a) archives are created differently in single- vs multi-threaded modes
b) xz will scale down the compression level so as to be able to work within any memory limit given to it when being launched in single-threaded mode.
Thus, for reproducible xz archives we need to launch xz with at least two threads.
Add a little sanity test, and error out otherwise, so as to guarantee no difference due this fact.
Assumptions:
* The thread limit should be set via XZ_THREADS, not via modifying XZ_DEFAULTS or XZ_OPTS, or any other way
* The thread limit should not be set to xz's magic value zero (0), as that will lead to single-threaded mode on single-core systems
This patch here doesn't prevent people from shooting themselves into the foot by changing XZ_DEFAULTS to change the number of threads directly, but it's can serve as a hint at least.
--- meta/classes/reproducible_build.bbclass | 5 +++++ 1 file changed, 5 insertions(+)
meta/classes/reproducible_build.bbclass +5 -0
@@ -35,6 +35,7 @@ # SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...) BUILD_REPRODUCIBLE_BINARIES ??= '1' +BUILD_REPRODUCIBLE_XZ_ARCHIVES ??= '1' inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')} SDE_DIR ="${WORKDIR}/source-date-epoch"
@@ -198,4 +199,8 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH" python () { if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1': d.appendVarFlag("do_unpack", "postfuncs", " do_create_source_date_epoch_stamp") + + if d.getVar('BUILD_REPRODUCIBLE_XZ_ARCHIVES') == '1': + if int(d.getVar('XZ_THREADS')) < 2: + bb.fatal("Can not build reproducible XZ archives without threading") }
Signature · 1 line
2.23.0.rc1
Re: [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives
Quoted 21 lines · André Draszik
xz suffers from a reproducibility problem when not using multi- threaded mode:
a) archives are created differently in single- vs multi-threaded modes
b) xz will scale down the compression level so as to be able to work within any memory limit given to it when being launched in single-threaded mode.Thus, for reproducible xz archives we need to launch xz with at least two threads.
Add a little sanity test, and error out otherwise, so as to guarantee no difference due this fact.
Assumptions:
* The thread limit should be set viaXZ_THREADS, not via modifyingXZ_DEFAULTSorXZ_OPTS, or any other way
* The thread limit should not be set to xz's magic value zero (0), as that will lead to single-threaded mode on single-core systems
Quoted 3 lines
This patch here doesn't prevent people from shooting themselves into the foot by changing
XZ_DEFAULTSto change the number of threads directly, but it's can serve as a hint at least.
I don't know if this patch is useful, feel free to drop it.
In an ideal world, it'd parse the output of xz --verbose --verbose, to catch all possible ways people might be adjusting the thread limit, but that's non-trivial.
Cheers,
Andre'
Quoted 27 lines
Signed-off-by André Draszik <[email protected]>--- meta/classes/reproducible_build.bbclass | 5 +++++ 1 file changed, 5 insertions(+)meta/classes/reproducible_build.bbclass +5
-0@@ -35,6 +35,7 @@ # SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...) BUILD_REPRODUCIBLE_BINARIES ??= '1' +BUILD_REPRODUCIBLE_XZ_ARCHIVES ??= '1' inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')} SDE_DIR ="${WORKDIR}/source-date-epoch"@@ -198,4 +199,8 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH" python () { if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1': d.appendVarFlag("do_unpack", "postfuncs", " do_create_source_date_epoch_stamp") + + if d.getVar('BUILD_REPRODUCIBLE_XZ_ARCHIVES') == '1': + if int(d.getVar('XZ_THREADS')) < 2: + bb.fatal("Can not build reproducible XZ archives without threading") }
Re: [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives
Quoted 4 lines · André Draszik
On Tue, 2020-03-03 at 16:05 +0000, André Draszik wrote:
In an ideal world, it'd parse the output of xz --verbose --verbose, to catch all possible ways people might be adjusting the thread limit, but that's non-trivial.
Couldn't we just "enforce" at least two threads? It is quite unlikely we ever use OE on a single core machine (as it'd take few years to finish the build hehe) it seems like a reasonable assumption.
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9 9981-7854 Mobile: +1 (347) 903-9750
Re: [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives
Quoted 11 lines · Otavio Salvador
Quoted 6 lines · André Draszik
On Tue, 2020-03-03 at 16:05 +0000, André Draszik wrote:
In an ideal world, it'd parse the output of xz --verbose --verbose, to catch
all possible ways people might be adjusting the thread limit, but that's
non-trivial.Couldn't we just "enforce" at least two threads? It is quite unlikely we ever use OE on a single core machine (as it'd take few years to finish the build hehe) it seems like a reasonable assumption.
An earlier patch does, unless you actually set XZ_THREADS = "1". If you do that, things are still reproducible in that the output will be consistent, just not with any other value of XZ_THREADS. So I think we should be fine without this patch.
Cheers,
Richard