Skip to contentSkip to conversation

bitbake.conf: omit XZ threads and RAM from sstate signatures

PATCH In review André Draszik 3 participants 16 messages Feb 24, 2020 – Feb 26, 2020

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 bitbake.conf: omit XZ threads and RAM from sstate signatures

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).

This is in particular a problem with the opkg package writing backend, as the OPKGBUILDCMD depends on XZ_DEFAULTS. Without the vardepexclude, there is no re-use possible of the package_write_ipk sstate.

The reason XZ_DEFAULTS as a whole isn't being whitelisted as such here is that some arguments can affect the outcome of the xz operation, so the approach taken is to only whitelist the safe flags memlimit and # of threads.

---
v2: remove Gerrit Change-Id line

---
 meta/conf/bitbake.conf | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +4 -1
index 954c06b313..d5a1a2471d 100644
@@ -794,7 +794,10 @@ BB_NUMBER_THREADS ?= "${@oe.utils.cpu_count()}"
 PARALLEL_MAKE ?= "-j ${@oe.utils.cpu_count()}"
 
 # Default parallelism and resource usage for xz
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
+XZ_MEMLIMIT ?= "50%"
+XZ_THREADS ?= "${@oe.utils.cpu_count()}"
+XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
+XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
 
 ##################################################################
 # Magic Cookie for SANITY CHECK
Signature · 1 line
2.23.0.rc1
#1 lore raw
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

Quoted 7 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?

cu
Adrian

#2 lore raw
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I'm actually aware of that, and was contemplating on expanding on that topic ... I guess I should have

Quoted 9 lines
Quoted 8 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Quoted 8 lines

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

I believe none of the above is relevant in the context here, though. The whole point is to avoid running xz again (within
a different
environment) if artefacts are available already.

threads and memlimit in particular are meant to enable compressing on resource constrained systems. Allowing sstate re-use no matter what threading and memory limit was set seems sensible, even if a build- machine has more ram than a deverloper's machine (or vice versa), as the point here is to allow sstate re-use to speed up builds. The point is not to change the compression, or to invalidate upon slight variations due to different amount of RAM.

It's good in particular to re-use the sstate artefacts, rather than creating new ones, just because the build happens to run on a different machine.

Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)

Quoted 2 lines

I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?

Likely, but a different problem...

Even before this patch, you will get differing results, but this patch doesn't prevent you from being more strict and specifying --no-adjust and it doesn't prevent you from changing the compression level as a whole either.

Cheers,
Andre'

#3 lore raw
Quoted 5 lines · André Draszik

...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

Quoted 2 lines

Cheers,
Andre'

cu
Adrian

#4 lore raw
Quoted 8 lines · Adrian Bunk
Quoted 5 lines · André Draszik

...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

I still maintain these are two different problems. I am not trying to fix reproducible builds.

Cheers,
A.

#5 lore raw
Quoted 8 lines · André Draszik
Quoted 7 lines · Adrian Bunk
Quoted 5 lines · André Draszik

...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...

Quoted 4 lines
Quoted 1 line

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

I still maintain these are two different problems. I am not trying to fix reproducible builds.

But won't fixing reproducible builds require reverting the patch you suggest for fixing your problem?

Quoted 2 lines

Cheers,
A.

cu
Adrian

#6 lore raw

reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

Quoted 9 lines · Adrian Bunk
Quoted 8 lines · André Draszik
Quoted 7 lines · Adrian Bunk
Quoted 5 lines · André Draszik

...
Once the artefact has been created, it doesn't matter anymore how it was created, as long as the high-level options aren't being changed (like an explicit compression level, checksum method, etc.)
...

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

I still maintain these are two different problems. I am not trying to fix reproducible builds.

But won't fixing reproducible builds require reverting the patch you suggest for fixing your problem?

No, I don't think so.

To fix reproducible, you need to either patch xz to always behave like in multi-threaded mode, even when --threads=1 was given in the arguments. Additionally, a patch to xz to allow it to scale down the number of threads, but not the compression level (--no-adjust prevents both), could be useful.

Alternatively, you need to decide if you want to support it with --threads=1 or --threads >= 2, and issue at least a warning in reproducible_build.bbclass if somebody deviates.

You should also calculate the memory requirements, reduce the number of threads to fit, and again give at least a warning if the compression can not be done without changing compression level after that.

I still maintain that none of that is related to my original patch.

Cheers,
Andre'

#7 lore raw

Re: reproducible builds involving xz (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

Quoted 37 lines · André Draszik
Quoted 13 lines · Adrian Bunk
Quoted 12 lines · André Draszik
Quoted 10 lines · Adrian Bunk
Quoted 7 lines · André Draszik

...
Once the artefact has been created, it doesn't matter anymore how it
was created, as long as the high-level options aren't being changed (like
an explicit compression level, checksum method, etc.)
...

meta-poky/conf/distro/poky.conf:INHERIT += "reproducible_build"

Quoted 7 lines
Quoted 3 lines

I still maintain these are two different problems. I am not trying to fix
reproducible builds.

But won't fixing reproducible builds require reverting the patch you
suggest for fixing your problem?

No, I don't think so.

To fix reproducible, you need to either patch xz to always behave like in
multi-threaded mode, even when --threads=1 was given in the arguments.
Additionally, a patch to xz to allow it to scale down the number of threads, but not the compression level (--no-adjust prevents both), could be useful.

Alternatively, you need to decide if you want to support it with --threads=1 or --threads >= 2, and issue at least a warning in reproducible_build.bbclass if somebody deviates.

Quoted 6 lines

You should also calculate the memory requirements, reduce the number of threads to fit, and again give at least a warning if the compression
can not be done without changing compression level after that.

I still maintain that none of that is related to my original patch.

If I merge your patch, what will happen is that builds on the autobuilder can potentially start failing as it will potentially detect reproduciblity issues which it wouldn't previously see.

To address this, I'd ike to see a new parameter added to oe.utils.cpu_count() which allows us to have a minimum of 2. I'd also like to reparametise the memlimit so we can drop it entirely by default.

This should get us some defaults which won't break things and give reproducibile builds. They may not work for everyone but its then something which we can build upon.

I do want to take a patch as I agree there is a serious issue here but I'd prefer we get it right and try and stop it causing me headaches on the testing infrastructure.

Cheers,

Richard

#8 lore raw
In reply to Adrian Bunk
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

all give different identical output.

So if we force --threads >=2 we should have determinism?

Quoted 11 lines
Quoted 7 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Threaded compression can result in slightly worse compression than single-threaded compression.

Quoted 8 lines

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?

I did some tests locally and I could see different output checksums depending on how much memory I gave xz.

Perhaps we should specify a specific high amount like 1GB?

Does anyone know more about the internals and how to have this behave "nicely" for our needs?

FWIW we haven't seen variation on the autobuilder due to this as far as I know.

Cheers,

Richard

#9 lore raw
Quoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

Quoted 7 lines

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

all give different identical output.

So if we force --threads >=2 we should have determinism?

This was also my guess after reading the manpage,
but no definite answer from me.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

Quoted 7 lines
Quoted 2 lines

I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?

I did some tests locally and I could see different output checksums depending on how much memory I gave xz.

Perhaps we should specify a specific high amount like 1GB?

xz -9 needs 1.25 GB per thread.

And since xz decompression speed is linear to compressed size, -9 is often wanted since it gives the fastest xz decompression.

Quoted 5 lines

Does anyone know more about the internals and how to have this behave "nicely" for our needs?

FWIW we haven't seen variation on the autobuilder due to this as far as I know.

I assume the autobuilders have plenty of RAM per core?

For any reasonably sizes machine that doesn't OOM on larger C++ projects the memlimit is a nop and can be dropped.

More problematic might be developers with oldish desktops/laptops with many cores and few RAM.

Quoted 3 lines

Cheers,

Richard

cu
Adrian

#10 lore raw
In reply to Richard Purdie
Quoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

Quoted 7 lines

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

all give different identical output.

So if we force --threads >=2 we should have determinism?

How large were your files? Given threaded operation works by operating in 'blocks', you should have a different number of blocks in your output. They're all compressed independently, so I don't see how the result could be identical, unless the block size is large enough for xz to not create as many blocks as you allowed via setting the upper limit on the # of threads.

Quoted 17 lines
Quoted 17 lines
Quoted 7 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

Quoted 7 lines
Quoted 2 lines

I assume 50% wrongly gives the same sstate signature no matter how much RAM is installed on the local machine?

I did some tests locally and I could see different output checksums depending on how much memory I gave xz.

Perhaps we should specify a specific high amount like 1GB?

If I understand the man page right, at compression level 9 (default in OE), each thread / block will use up to 674MiB of RAM, so a limit of 1GiB effectively reduces parallelism to 1.

Quoted 2 lines

Does anyone know more about the internals and how to have this behave "nicely" for our needs?

What are the needs specifically?

In my case, I have some defined builds that run on their own, so they are allowed to use all memory and a certain (lowish) number of threads.

I also have other builds where several can run in parallel, and xz would just be killed if all of them were allowed to use all RAM, so I have to reduce threads and memory consumption.

This of course means slower builds...

I don't want to penalise myself or anybody else by generally forcing a low number of threads either.

Quoted 2 lines

FWIW we haven't seen variation on the autobuilder due to this as far as I know.

BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.

Files are split into chunks, and chunks are compressed individually in the pbzip2 case.
pigz terminates each chunk with an empty block in the final .gz file.

Cheers,
Andre'

#11 lore raw
Quoted 27 lines · André Draszik
Quoted 26 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to
compress chunks and their compression should be deterministic whether
done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

Quoted 5 lines
Quoted 3 lines

all give different identical output.

So if we force --threads >=2 we should have determinism?

How large were your files?

105MB total which should be enough as the largest xz block size is 32MB?

Quoted 6 lines

Given threaded operation works by operating in 'blocks', you should
have a different number of blocks in your output.
They're all compressed independently, so I don't see how the result could be identical, unless the block size is large enough for xz to not create as many blocks as you allowed via setting the upper limit on the # of threads.

I don't follow. The algorithm xz uses appears to be designed to give consistent results regardless of numbers of threads. It gets split into the same number of chunks. Each chunk is compressed independently, but deterministically and then they're assembled in order. It doesn't matter if the chunk is compressed in parallel or serially.

Quoted 37 lines
Quoted 37 lines
Quoted 27 lines
Quoted 12 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count ()}"

Quoted 1 line

Files are split into chunks, and chunks are compressed

individually

Quoted 3 lines

in the
pbzip2 case.
pigz terminates each chunk with an empty block in the final

Quoted 8 lines

.gz file.

Quoted 6 lines

Cheers,
Andre'

...

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

Quoted 6 lines

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

I assume 50% wrongly gives the same sstate signature no matter how
much RAM is installed on the local machine?

I did some tests locally and I could see different output checksums depending on how much memory I gave xz.

Quoted 6 lines
Quoted 1 line

Perhaps we should specify a specific high amount like 1GB?

If I understand the man page right, at compression level 9 (default in OE),
each thread / block will use up to 674MiB of RAM, so a limit of 1GiB effectively reduces parallelism to 1.

Right, I agree that isn't feasible.

Quoted 4 lines
Quoted 2 lines

Does anyone know more about the internals and how to have this behave "nicely" for our needs?

What are the needs specifically?

The project is aiming to have deterministic builds, our output for given input metadata is binary identical.

Quoted 4 lines

In my case, I have some defined builds that run on their own, so they are allowed to use all memory and a certain (lowish) number of threads.

Quoted 8 lines

I also have other builds where several can run in parallel, and xz would just be killed if all of them were allowed to use all RAM, so I have to reduce threads and memory consumption.

This of course means slower builds...

I don't want to penalise myself or anybody else by generally forcing a low number of threads either.

We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.

Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need. Perhaps we just allow the number of xz threads to be set independently?

Quoted 6 lines
Quoted 3 lines

FWIW we haven't seen variation on the autobuilder due to this as far as
I know.

BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.

Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.

Cheers,

Richard

#12 lore raw
Quoted 5 lines · Richard Purdie

On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
...

Quoted 2 lines

I don't want to penalise myself or anybody else by generally forcing a low number of threads either.

Quoted 7 lines

We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.

Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need.

On a Threadripper with 128 cores and 256 GB RAM it would not be a problem to use all cores.

A laptop with 8 cores and 8 GB RAM is problematic.

Quoted 2 lines

Perhaps we just allow the number of xz threads
to be set independently?

dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574

In a script it would be possible to use --no-adjust to achieve the same:

$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

FWIW we haven't seen variation on the autobuilder due to this as far as
I know.

BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.

Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.

Good point, dpkg does parallel xz compression and I am not aware of any reproducibility problems this causes.

Quoted 3 lines

Cheers,

Richard

cu
Adrian

#13 lore raw
Quoted 26 lines · Adrian Bunk
Quoted 5 lines · Richard Purdie

On Mon, 2020-02-24 at 17:14 +0000, André Draszik wrote:
...

Quoted 2 lines

I don't want to penalise myself or anybody else by generally forcing a low number of threads either.

Quoted 7 lines

We didn't have resource issues on the autobuilder with xz but others did report it and its why the memlimit was set. It would be better to remove the memlimit for determinism really as far as I can tell and if necessary throttle the threads count.

Maybe setting an upper limit of say 10 threads and a minimum of 2 might given us what we need.

On a Threadripper with 128 cores and 256 GB RAM it would not be a problem to use all cores.

A laptop with 8 cores and 8 GB RAM is problematic.

Quoted 2 lines

Perhaps we just allow the number of xz threads
to be set independently?

dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574

Quoted 6 lines

In a script it would be possible to use --no-adjust to achieve the same:

$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$

The problem with --no-adjust is that it also prevents xz from reducing the number of threads. It will do that in preference to changing compression parameters, so as long as you have more than 2499MiB of memory to support at least 2 threads (with default block size), compression will be identical.

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

FWIW we haven't seen variation on the autobuilder due to this as far as
I know.

BTW, pigz and pbzip should have a similar thread related problem, according to the man pages, if you read between the lines.

Agreed. We have less of an issue with these as their use is much less frequent. xz is used by deb and ipk for packaging.

They both actually don't. It's reproducible no matter how many threads. Interesting...

Quoted 3 lines

Good point, dpkg does parallel xz compression and I am not aware of any reproducibility problems this causes.

Debian's dpkg tries to be clever by simply reducing the number of threads until compression uses at most half of physical
ram.

I guess this typically works out as at least two threads in most if not all environments.

Cheers,
Andre'

#14 lore raw
Quoted 15 lines · André Draszik

On Tue, 2020-02-25 at 00:00 +0200, Adrian Bunk wrote:
...

Quoted 7 lines
Quoted 2 lines

Perhaps we just allow the number of xz threads
to be set independently?

dpkg manually reduces the number of threads until less than half of the RAM is used:
https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574

Quoted 5 lines

In a script it would be possible to use --no-adjust to achieve the same:

$ xz -9 --memlimit=50% --no-adjust -T32 /dev/null 
xz: Memory usage limit is too low for the given filter setup.
xz: 39,972 MiB of memory is required. The limit is 32,051 MiB.
$
Quoted 5 lines

The problem with --no-adjust is that it also prevents xz from reducing the number of threads. It will do that in preference to changing compression parameters, so as long as you have more than 2499MiB of memory to support at least 2 threads (with default block size), compression will be identical.
...

5 GB due to the 50%.

But I had something different in mind:

Similar to what dpkg is doing, you could once loop from @oe.utils.cpu_count() down to 2 until "--memlimit=50% --no-adjust -T$(i)" succeeds.

Then run the actual compression without --memlimit.

I would not consider anything with less than 4 GB RAM reasonable for building Yocto (g++ also likes to use more than 2 GB).

Quoted 2 lines

Cheers,
Andre'

cu
Adrian

#15 lore raw

xz threads / memlimit behaviour (was: Re: [PATCH v2] bitbake.conf: omit XZ threads and RAM from sstate signatures)

In reply to Richard Purdie
Quoted 20 lines · Richard Purdie
Quoted 6 lines · Adrian Bunk
Quoted 3 lines · André Draszik

The number of threads used, and the amount of memory allowed to be used, should not affect sstate signatures, as they don't affect the result.

Unfortunately they can affect the result.

I looked into this a bit and its complicated. The threads are used to compress chunks and their compression should be deterministic whether done serially or in parallel.

I did some tests and:

xz <file>
gave equivalent output to:
xz <file> --threads=1

and

Quoted 7 lines

xz <file> --threads=2
xz <file> --threads=5
xz <file> --threads=50

all give different identical output.

So if we force --threads >=2 we should have determinism?

I did another test...

Even single threaded compression gives the same result as multi-threaded, if single threaded is a result of xz scaling down the memory usage due to --memlimit, ie:

xz -f -c -9 --threads=2 --memlimit=2000M --keep --verbose --verbose xztest > xztest2_down_to_1.xz
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 2 threads.
xz: 2499 MiB of memory is required. The limit is 2000 MiB.
xz: Decompression will need 65 MiB of memory.
xz: Adjusted the number of threads from 2 to 1 to not exceed the memory usage limit of 2000 MiB

is different from

xz -f -c -9 --threads=1 --memlimit=2000M --verbose --verbose xztest > xztest1.xz
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: 674 MiB of memory is required. The limit is 2000 MiB.
xz: Decompression will need 65 MiB of memory.

There even is a comment in the xz sources about this. This simplifies things a bit...

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Otherwise, it becomes impossible to re-use sstate from automated builders on developer's machines (as the former might execute bitbake with certain constraints different compared to developer's machines).
...
-XZ_DEFAULTS ?= "--memlimit=50% --threads=${@oe.utils.cpu_count()}"
...

Quoted 8 lines
Quoted 8 lines

Threaded compression can result in slightly worse compression than single-threaded compression.

With memlimit the problem is actually the opposite way, and worse than what you were trying to fix:

When a developer hits memlimit during compression, the documented behavour of xz is to scale down the compression level.

This is not what is actually happening - it appears the documentation only talks about single-threaded mode.
If you look at the code and look at command output, xz only ever tries to reduce the number of threads if multi-threading was enabled. In that case, it will *never* even attempt to change the compression level.

If it can't satisfy the memory limit by reducing the number of threads, it just errors out.

So as long as all builders have at least 2 cores, i.e. --threads >= 2, memlimit doesn't ever affect the outcome (except that it can make the operation fail as a whole).

Cheers,
Andre'