bitbake.conf: omit XZ threads and RAM from sstate signatures
PATCH v2 1/1 bitbake.conf: omit XZ threads and RAM from sstate signatures
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
This is in particular a problem with the opkg package writing backend, as the OPKGBUILDCMD depends on XZ_DEFAULTS. Without the vardepexclude, there is no re-use possible of the package_write_ipk sstate.
The reason XZ_DEFAULTS as a whole isn't being whitelisted as such here is that some arguments can affect the outcome of the xz operation, so the approach taken is to only whitelist the safe flags memlimit and # of threads.
---
v2: remove Gerrit Change-Id line
--- meta/conf/bitbake.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +4 -1
@@ -794,7 +794,10 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}" PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}" # Default parallelism and resource usage for xz -XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}" +XZ_MEMLIMIT ?= "50%" +XZ_THREADS ?= "${@oe.utils.cpu_count()}" +XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" +XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS" ################################################################## # Magic Cookie for SANITY CHECK
Signature · 1 line
2.23.0.rc1
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
Quoted 7 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...
Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?
cu
Adrian
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I'm actually aware of that, and was contemplating on expanding on that topic ... I guess I should have
Quoted 9 lines
Quoted 8 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...
Quoted 8 lines
Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
I believe none of the above is relevant in the context here, though. The whole point is to avoid running xz again (within
a different
environment) if artefacts are available already.
threads and memlimit in particular are meant to enable compressing on resource constrained systems. Allowing sstate re-use no matter what threading and memory limit was set seems sensible, even if a build- machine has more ram than a deverloper's machine (or vice versa), as the point here is to allow sstate re-use to speed up builds. The point is not to change the compression, or to invalidate upon slight variations due to different amount of RAM.
It's good in particular to re-use the sstate artefacts, rather than creating new ones, just because the build happens to run on a different machine.
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
Quoted 2 lines
I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?
Likely, but a different problem...
Even before this patch, you will get differing results, but this patch doesn't prevent you from being more strict and specifying --no-adjust and it doesn't prevent you from changing the compression level as a whole either.
Cheers,
Andre'
Quoted 5 lines · André Draszik
...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...
meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"
Quoted 2 lines
Cheers,
Andre'
cu
Adrian
Quoted 8 lines · Adrian Bunk
Quoted 5 lines · André Draszik
...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...meta-poky/conf/distro/poky.conf:INHERIT += "
reproducible_build"
I still maintain these are two different problems. I am not trying to fix reproducible builds.
Cheers,
A.
Quoted 8 lines · André Draszik
Quoted 7 lines · Adrian Bunk
Quoted 5 lines · André Draszik
...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...
Quoted 4 lines
Quoted 1 line
meta-poky/conf/distro/poky.conf:INHERIT += "
reproducible_build"I still maintain these are two different problems. I am not trying to fix reproducible builds.
But won't fixing reproducible builds require reverting the patch you suggest for fixing your problem?
Quoted 2 lines
Cheers,
A.
cu
Adrian
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.
I did some tests and:
xz <file>
gave equivalent output to:
xz <file> --threads=1
and
xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50
all give different identical output.
So if we force --threads >=2 we should have determinism?
Quoted 11 lines
Quoted 7 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...Threaded compression can result in slightly worse compression than single-threaded compression.
Quoted 8 lines
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?
I did some tests locally and I could see different output checksums depending on how much memory I gave xz.
Perhaps we should specify a specific high amount like 1GB?
Does anyone know more about the internals and how to have this behave "nicely" for our needs?
FWIW we haven't seen variation on the autobuilder due to this as far as I know.
Cheers,
Richard
Quoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.
I did some tests and:
xz <file>
gave equivalent output to:
xz <file> --threads=1and
Quoted 7 lines
xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50all give different identical output.
So if we force --threads >=2 we should have determinism?
This was also my guess after reading the manpage,
but no definite answer from me.
Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
Quoted 7 lines
Quoted 2 lines
I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?
I did some tests locally and I could see different output checksums depending on how much memory I gave xz.
Perhaps we should specify a specific high amount like 1GB?
xz -9 needs 1.25 GB per thread.
And since xz decompression speed is linear to compressed size, -9 is often wanted since it gives the fastest xz decompression.
Quoted 5 lines
Does anyone know more about the internals and how to have this behave "nicely" for our needs?
FWIW we haven't seen variation on the autobuilder due to this as far as I know.
I assume the autobuilders have plenty of RAM per core?
For any reasonably sizes machine that doesn't OOM on larger C++ projects the memlimit is a nop and can be dropped.
More problematic might be developers with oldish desktops/laptops with many cores and few RAM.
Quoted 3 lines
Cheers,
Richard
cu
Adrian
Quoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.
I did some tests and:
xz <file>
gave equivalent output to:
xz <file> --threads=1and
Quoted 7 lines
xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50all give different identical output.
So if we force --threads >=2 we should have determinism?
How large were your files? Given threaded operation works by operating in 'blocks', you should have a different number of blocks in your output. They're all compressed independently, so I don't see how the result could be identical, unless the block size is large enough for xz to not create as many blocks as you allowed via setting the upper limit on the # of threads.
Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
Quoted 7 lines
Quoted 2 lines
I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?
I did some tests locally and I could see different output checksums depending on how much memory I gave xz.
Perhaps we should specify a specific high amount like 1GB?
If I understand the man page right, at compression level 9 (default in OE), each thread / block will use up to 674MiB of RAM, so a limit of 1GiB effectively reduces parallelism to 1.
Quoted 2 lines
Does anyone know more about the internals and how to have this behave "nicely" for our needs?
What are the needs specifically?
In my case, I have some defined builds that run on their own, so they are allowed to use all memory and a certain (lowish) number of threads.
I also have other builds where several can run in parallel, and xz would just be killed if all of them were allowed to use all RAM, so I have to reduce threads and memory consumption.
This of course means slower builds...
I don't want to penalise myself or anybody else by generally forcing a low number of threads either.
Quoted 2 lines
FWIW we haven't seen variation on the autobuilder due to this as far as I know.
BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.
Files are split into chunks, and chunks are compressed individually in the pbzip2 case.
pigz terminates each chunk with an empty block in the final .gz file.
Cheers,
Andre'
Quoted 27 lines · André Draszik
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I looked into this a bit and its complicated. The threads are used to
compress chunks and their compression should be deterministic whether
done serially or in parallel.I did some tests and:
xz <file>
gave equivalent output to:
xz <file> --threads=1and
xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50
Quoted 5 lines
Quoted 3 lines
all give different identical output.
So if we force --threads >=2 we should have determinism?
How large were your files?
105MB total which should be enough as the largest xz block size is 32MB?
Quoted 6 lines
Given threaded operation works by operating in 'blocks', you should
have a different number of blocks in your output.
They're all compressed independently, so I don't see how the result could be identical, unless the block size is large enough for xz to not create as many blocks as you allowed via setting the upper limit on the # of threads.
I don't follow. The algorithm xz uses appears to be designed to give consistent results regardless of numbers of threads. It gets split into the same number of chunks. Each chunk is compressed independently, but deterministically and then they're assembled in order. It doesn't matter if the chunk is compressed in parallel or serially.
Quoted 37 lines
Quoted 37 lines
Quoted 27 lines
Quoted 12 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count ()}"Quoted 1 line
Files are split into chunks, and chunks are compressed
individually
Quoted 3 lines
in the
pbzip2 case.
pigz terminates each chunk with an empty block in the finalQuoted 8 lines
.gz file.
Quoted 6 lines
Cheers,
Andre'...
Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
Quoted 6 lines
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
I assume 50% wrongly gives the same sstate signature no matter how
much RAM is installed on the local machine?I did some tests locally and I could see different output checksums depending on how much memory I gave xz.
Quoted 6 lines
Quoted 1 line
Perhaps we should specify a specific high amount like 1GB?
If I understand the man page right, at compression level 9 (default in OE),
each thread / block will use up to 674MiB of RAM, so a limit of 1GiB effectively reduces parallelism to 1.
Right, I agree that isn't feasible.
Quoted 4 lines
Quoted 2 lines
Does anyone know more about the internals and how to have this behave "nicely" for our needs?
What are the needs specifically?
The project is aiming to have deterministic builds, our output for given input metadata is binary identical.
Quoted 4 lines
In my case, I have some defined builds that run on their own, so they are allowed to use all memory and a certain (lowish) number of threads.
Quoted 8 lines
I also have other builds where several can run in parallel, and xz would just be killed if all of them were allowed to use all RAM, so I have to reduce threads and memory consumption.
This of course means slower builds...
I don't want to penalise myself or anybody else by generally forcing a low number of threads either.
We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.
Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need. Perhaps we just allow the number of xz threads to be set independently?
Quoted 6 lines
Quoted 3 lines
FWIW we haven't seen variation on the autobuilder due to this as far as
I know.BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.
Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.
Cheers,
Richard
Quoted 5 lines · Richard Purdie
On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
...Quoted 2 lines
I don't want to penalise myself or anybody else by generally forcing a low number of threads either.
Quoted 7 lines
We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.
Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need.
On a Threadripper with 128 cores and 256 GB RAM it would not be a problem to use all cores.
A laptop with 8 cores and 8 GB RAM is problematic.
Quoted 2 lines
Perhaps we just allow the number of xz threads
to be set independently?
dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null xz: Memory usage limit is too low for the given filter setup. xz: 39,972 MiB of memory is required. The limit is 32,051 MiB. $
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
FWIW we haven't seen variation on the autobuilder due to this as far as
I know.BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.
Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.
Good point, dpkg does parallel xz compression and I am not aware of any reproducibility problems this causes.
Quoted 3 lines
Cheers,
Richard
cu
Adrian
Quoted 26 lines · Adrian Bunk
Quoted 5 lines · Richard Purdie
On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
...Quoted 2 lines
I don't want to penalise myself or anybody else by generally forcing a low number of threads either.
Quoted 7 lines
We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.
Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need.
On a Threadripper with 128 cores and 256 GB RAM it would not be a problem to use all cores.
A laptop with 8 cores and 8 GB RAM is problematic.
Quoted 2 lines
Perhaps we just allow the number of xz threads
to be set independently?dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574
Quoted 6 lines
In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null xz: Memory usage limit is too low for the given filter setup. xz: 39,972 MiB of memory is required. The limit is 32,051 MiB. $
The problem with --no-adjust is that it also prevents xz from reducing the number of threads. It will do that in preference to changing compression parameters, so as long as you have more than 2499MiB of memory to support at least 2 threads (with default block size), compression will be identical.
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
FWIW we haven't seen variation on the autobuilder due to this as far as
I know.BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.
Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.
They both actually don't. It's reproducible no matter how many threads. Interesting...
Quoted 3 lines
Good point, dpkg does parallel xz compression and I am not aware of any reproducibility problems this causes.
Debian's dpkg tries to be clever by simply reducing the number of threads until compression uses at most half of physical
ram.
I guess this typically works out as at least two threads in most if not all environments.
Cheers,
Andre'
Quoted 15 lines · André Draszik
On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
...Quoted 7 lines
Quoted 2 lines
Perhaps we just allow the number of xz threads
to be set independently?dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574Quoted 5 lines
In a script it would be possible to use --no-adjust to achieve the same:
$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null xz: Memory usage limit is too low for the given filter setup. xz: 39,972 MiB of memory is required. The limit is 32,051 MiB. $
Quoted 5 lines
The problem with --no-adjust is that it also prevents xz from reducing the number of threads. It will do that in preference to changing compression parameters, so as long as you have more than 2499MiB of memory to support at least 2 threads (with default block size), compression will be identical.
...
5 GB due to the 50%.
But I had something different in mind:
Similar to what dpkg is doing, you could once loop from @oe.utils.cpu_count() down to 2 until "--memlimit=50% --no-adjust -T$(i)" succeeds.
Then run the actual compression without --memlimit.
I would not consider anything with less than 4 GB RAM reasonable for building Yocto (g++ also likes to use more than 2 GB).
Quoted 2 lines
Cheers,
Andre'
cu
Adrian
reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)
In reply to Adrian BunkQuoted 9 lines · Adrian Bunk
Quoted 8 lines · André Draszik
Quoted 7 lines · Adrian Bunk
Quoted 5 lines · André Draszik
...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
meta-poky/conf/distro/poky.conf:INHERIT += "
reproducible_build"I still maintain these are two different problems. I am not trying to fix reproducible builds.
But won't fixing reproducible builds require reverting the patch you suggest for fixing your problem?
No, I don't think so.
To fix reproducible, you need to either patch xz to always behave like in multi-threaded mode, even when --threads=1 was given in the arguments. Additionally, a patch to xz to allow it to scale down the number of threads, but not the compression level (--no-adjust prevents both), could be useful.
Alternatively, you need to decide if you want to support it with --threads=1 or --threads >= 2, and issue at least a warning in reproducible_build.bbclass if somebody deviates.
You should also calculate the memory requirements, reduce the number of threads to fit, and again give at least a warning if the compression can not be done without changing compression level after that.
I still maintain that none of that is related to my original patch.
Cheers,
Andre'
Re: reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)
Quoted 37 lines · André Draszik
Quoted 13 lines · Adrian Bunk
Quoted 12 lines · André Draszik
Quoted 10 lines · Adrian Bunk
Quoted 7 lines · André Draszik
...
Once the artefact has been created, it doesn't matter anymore how it
was created, as long as the high-level options aren't being changed (like
an explicit compression level, checksum method, etc.)
...meta-poky/conf/distro/poky.conf:INHERIT += "
reproducible_build"Quoted 7 lines
Quoted 3 lines
I still maintain these are two different problems. I am not trying to fix
reproducible builds.But won't fixing reproducible builds require reverting the patch you
suggest for fixing your problem?No, I don't think so.
To fix reproducible, you need to either patch xz to always behave like in
multi-threaded mode, even when --threads=1 was given in the arguments.
Additionally, a patch to xz to allow it to scale down the number of threads, but not the compression level (--no-adjust prevents both), could be useful.Alternatively, you need to decide if you want to support it with --threads=1 or --threads >= 2, and issue at least a warning in reproducible_build.bbclass if somebody deviates.
Quoted 6 lines
You should also calculate the memory requirements, reduce the number of threads to fit, and again give at least a warning if the compression
can not be done without changing compression level after that.I still maintain that none of that is related to my original patch.
If I merge your patch, what will happen is that builds on the autobuilder can potentially start failing as it will potentially detect reproduciblity issues which it wouldn't previously see.
To address this, I'd ike to see a new parameter added to oe.utils.cpu_count() which allows us to have a minimum of 2. I'd also like to reparametise the memlimit so we can drop it entirely by default.
This should get us some defaults which won't break things and give reproducibile builds. They may not work for everyone but its then something which we can build upon.
I do want to take a patch as I agree there is a serious issue here but I'd prefer we get it right and try and stop it causing me headaches on the testing infrastructure.
Cheers,
Richard
xz threads / memlimit behaviour (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)
In reply to Richard PurdieQuoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik
The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.
Unfortunately they can affect the result.
I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.
I did some tests and:
xz <file>
gave equivalent output to:
xz <file> --threads=1and
Quoted 7 lines
xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50all give different identical output.
So if we force --threads >=2 we should have determinism?
I did another test...
Even single threaded compression gives the same result as multi-threaded, if single threaded is a result of xz scaling down the memory usage due to --memlimit, ie:
xz -f -c -9 --threads=2 --memlimit=2000M --keep --verbose --verbose xztest > xztest2_down_to_1.xz
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 2 threads.
xz: 2499 MiB of memory is required. The limit is 2000 MiB.
xz: Decompression will need 65 MiB of memory.
xz: Adjusted the number of threads from 2 to 1 to not exceed the memory usage limit of 2000 MiB
is different from
xz -f -c -9 --threads=1 --memlimit=2000M --verbose --verbose xztest > xztest1.xz
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: 674 MiB of memory is required. The limit is 2000 MiB.
xz: Decompression will need 65 MiB of memory.
There even is a comment in the xz sources about this. This simplifies things a bit...
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...
Quoted 8 lines
Quoted 8 lines
Threaded compression can result in slightly worse compression than single-threaded compression.
With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:
When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.
This is not what is actually happening - it appears the documentation only talks about single-threaded mode.
If you look at the code and look at command output, xz only ever tries to reduce the number of threads if multi-threading was enabled. In that case, it will *never* even attempt to change the compression level.
If it can't satisfy the memory limit by reducing the number of threads, it just errors out.
So as long as all builders have at least 2 cores, i.e. --threads >= 2, memlimit doesn't ever affect the outcome (except that it can make the operation fail as a whole).
Cheers,
Andre'