How to find out why my do_compile depends on $MACHINE

DISCUSSION Mike Looijmans 4 participants 7 messages Oct 5, 2015 – Oct 6, 2015

Share & tools

Mike Looijmans

author

2015-10-05 09:34 UTC #0 lore raw

I've been puzzling with this for many hours now, but I can't seem to figure this out.

I have a recipe that doesn't mention MACHINE anywhere, but still the sstate-cache won't be re-used for other machines, because OE somehow insists the package does depend on $MACHINE.

The workaround I could find is to put something like this in the recipe:

MACHINE[vardepvalue] = "any"

(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm not interested in forcing this particular recipe, but I'm trying to figure out what makes bitbake think it matters)

Since it builds for an FPGA type, the package has PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.

When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports that MACHINE changed and that is what caused sstate not beign reused. But the recipe does not use anything related to $MACHINE at all.

Is there a way to make it tell me WHY it thinks that MACHINE is important to the recipe?

Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: [email protected]
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Richard Purdie

2015-10-05 10:18 UTC #1 lore raw

Quoted 19 lines · Mike Looijmans

I've been puzzling with this for many hours now, but I can't seem to figure this out.
I have a recipe that doesn't mention MACHINE anywhere, but still the sstate-cache won't be re-used for other machines, because OE somehow insists the package does depend on $MACHINE.
The workaround I could find is to put something like this in the recipe:
MACHINE[vardepvalue] = "any"
(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm not interested in forcing this particular recipe, but I'm trying to figure out what makes bitbake think it matters)
Since it builds for an FPGA type, the package has PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.

Quoted 6 lines

When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports that MACHINE changed and that is what caused sstate not beign reused. But the recipe does not use anything related to $MACHINE at all.
Is there a way to make it tell me WHY it thinks that MACHINE is important to the recipe?

Very briefly (sorry travelling), find the task that you want to know more around and find the sigdata file in the stamps directory for it. Run "bitbake-diffsigs <sigdatafile>" and it will show you how the dependencies are calculated. You can compare two sigdata files using the same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".

Cheers,

Richard

Mike Looijmans

author

2015-10-05 11:50 UTC #2 lore raw

Quoted 24 lines · Richard Purdie

Quoted 23 lines · Mike Looijmans
I've been puzzling with this for many hours now, but I can't seem to figure this out.
I have a recipe that doesn't mention MACHINE anywhere, but still the sstate-cache won't be re-used for other machines, because OE somehow insists the package does depend on $MACHINE.
The workaround I could find is to put something like this in the recipe:
MACHINE[vardepvalue] = "any"
(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm not interested in forcing this particular recipe, but I'm trying to figure out what makes bitbake think it matters)
Since it builds for an FPGA type, the package has PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports that MACHINE changed and that is what caused sstate not beign reused. But the recipe does not use anything related to $MACHINE at all.

Quoted 8 lines

Quoted 2 lines
Is there a way to make it tell me WHY it thinks that MACHINE is important to the recipe?
Very briefly (sorry travelling), find the task that you want to know more around and find the sigdata file in the stamps directory for it. Run "bitbake-diffsigs <sigdatafile>" and it will show you how the dependencies are calculated. You can compare two sigdata files using the same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".

Thanks, that reveiled the issue.

I was smart in local.conf and calculated the FPGA name from the MACHINE name. That yields the same word, but somehow the dependency on the MACHINE variable is kept.

A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the link to MACHINE is lost, but that then results in another rebuild:

List of dependencies for variable FPGA_FAMILY changed from
'set(['MACHINE'])' to 'set([])'
Dependency on Variable MACHINE was removed

The FPGA packages take long to build (4 hours for just one is no exception, on an i7 machine) so preventing them to be rebuilt is really important here.

But having found the issue, I guess after a round of rebuilds, it'll go away now.

For the bigger picture:

If I have a recipe that says:

X = "x"

And I refactor it a bit to read:

Y = "x"
X = "${Y}"

What is the logic behind having to rebuild the package now? Why does the 'source' of the contents matter to the hash? It won't generate different output, regardless whether X is 'calculated' or just 'constant'.

In particular, I've noticed this happening when switching between AUTOREV and a fixed revision in a recipe.

Kind regards,

Mike Looijmans
System Expert

Please consider the environment before printing this e-mail

Martin Jansa

2015-10-05 12:46 UTC #3 lore raw

Quoted 60 lines · Mike Looijmans

Quoted 24 lines · Richard Purdie
Quoted 23 lines · Mike Looijmans
I've been puzzling with this for many hours now, but I can't seem to figure this out.
I have a recipe that doesn't mention MACHINE anywhere, but still the sstate-cache won't be re-used for other machines, because OE somehow insists the package does depend on $MACHINE.
The workaround I could find is to put something like this in the recipe:
MACHINE[vardepvalue] = "any"
(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm not interested in forcing this particular recipe, but I'm trying to figure out what makes bitbake think it matters)
Since it builds for an FPGA type, the package has PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports that MACHINE changed and that is what caused sstate not beign reused. But the recipe does not use anything related to $MACHINE at all.
Quoted 8 lines
Quoted 2 lines
Is there a way to make it tell me WHY it thinks that MACHINE is important to the recipe?
Very briefly (sorry travelling), find the task that you want to know more around and find the sigdata file in the stamps directory for it. Run "bitbake-diffsigs <sigdatafile>" and it will show you how the dependencies are calculated. You can compare two sigdata files using the same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".
Thanks, that reveiled the issue.
I was smart in local.conf and calculated the FPGA name from the MACHINE name. That yields the same word, but somehow the dependency on the MACHINE variable is kept.
A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the link to MACHINE is lost, but that then results in another rebuild:
List of dependencies for variable FPGA_FAMILY changed from
'set(['MACHINE'])' to 'set([])'
Dependency on Variable MACHINE was removed
The FPGA packages take long to build (4 hours for just one is no exception, on an i7 machine) so preventing them to be rebuilt is really important here.
But having found the issue, I guess after a round of rebuilds, it'll go away now.
For the bigger picture:
If I have a recipe that says:
X = "x"

Quoted 8 lines

And I refactor it a bit to read:
Y = "x"
X = "${Y}"
What is the logic behind having to rebuild the package now? Why does the 'source' of the contents matter to the hash? It won't generate different output, regardless whether X is 'calculated' or just 'constant'.

You can set vardepvalue flag like this:
X[vardepvalue] = "${X}"
if all you care is the value of X not the expression used to construct it. This can help with your FGPA_FAMILY and is more readable (at least to me) then immediate assignment.

This cannot be applied by default, because in some cases you have e.g. TOPDIR variable used in value and then it's better to exclude just TOPDIR from the variable dependencies and keep the rest of variables tracked (using just the value would cause different signatures for 2 builds with different TOPDIR).

Quoted 28 lines

In particular, I've noticed this happening when switching between AUTOREV and a fixed revision in a recipe.
Kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: [email protected]
Website: www.topicproducts.com
Please consider the environment before printing this e-mail
Signature · 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 1 line

Martin 'JaMa' Jansa     jabber: [email protected]

Richard Purdie

2015-10-05 13:22 UTC #4 lore raw

In reply to Mike Looijmans

Quoted 40 lines · Mike Looijmans

Quoted 24 lines · Richard Purdie
Quoted 23 lines · Mike Looijmans
I've been puzzling with this for many hours now, but I can't seem to figure this out.
I have a recipe that doesn't mention MACHINE anywhere, but still the sstate-cache won't be re-used for other machines, because OE somehow insists the package does depend on $MACHINE.
The workaround I could find is to put something like this in the recipe:
MACHINE[vardepvalue] = "any"
(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm not interested in forcing this particular recipe, but I'm trying to figure out what makes bitbake think it matters)
Since it builds for an FPGA type, the package has PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports that MACHINE changed and that is what caused sstate not beign reused. But the recipe does not use anything related to $MACHINE at all.
Quoted 8 lines
Quoted 2 lines
Is there a way to make it tell me WHY it thinks that MACHINE is important to the recipe?
Very briefly (sorry travelling), find the task that you want to know more around and find the sigdata file in the stamps directory for it. Run "bitbake-diffsigs <sigdatafile>" and it will show you how the dependencies are calculated. You can compare two sigdata files using the same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".
Thanks, that reveiled the issue.
I was smart in local.conf and calculated the FPGA name from the MACHINE name. That yields the same word, but somehow the dependency on the MACHINE variable is kept.

Quoted 6 lines

A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the link to MACHINE is lost, but that then results in another rebuild:
List of dependencies for variable FPGA_FAMILY changed from
'set(['MACHINE'])' to 'set([])'
Dependency on Variable MACHINE was removed

FWIW you can use vardepvalue, e.g.:

FEATURE_INSTALL[vardepvalue] = "${FEATURE_INSTALL}"

which will make it depend on the value of the variable rather than the dependencies. Yes, a rebuild will result after this but it should be a one off.

Quoted 16 lines

The FPGA packages take long to build (4 hours for just one is no exception, on an i7 machine) so preventing them to be rebuilt is really important here.
But having found the issue, I guess after a round of rebuilds, it'll go away now.
For the bigger picture:
If I have a recipe that says:
X = "x"
And I refactor it a bit to read:
Y = "x"
X = "${Y}"

Quoted 6 lines

What is the logic behind having to rebuild the package now? Why does the 'source' of the contents matter to the hash? It won't generate different output, regardless whether X is 'calculated' or just 'constant'.
In particular, I've noticed this happening when switching between AUTOREV and a fixed revision in a recipe.

This is simply just the way the system is designed. It checksums the intermediate steps as well as the endpoints. I guess in the above example the system might do "export Y" for example.

There are alternative ways to do things but its simply not the way things were implemented. If you do need to collapse the dependency chain, you can use vardepvalue which was added for that reason. There are only a small number of places you really need to do that.

Cheers,

Richard

Mike Looijmans

author

2015-10-06 12:55 UTC #5 lore raw

Quoted 3 lines · Richard Purdie

Quoted 2 lines · Mike Looijmans
Quoted 1 line · Richard Purdie
On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:

...

Quoted 15 lines

Quoted 15 lines
For the bigger picture:
If I have a recipe that says:
X = "x"
And I refactor it a bit to read:
Y = "x"
X = "${Y}"
What is the logic behind having to rebuild the package now? Why does the 'source' of the contents matter to the hash? It won't generate different output, regardless whether X is 'calculated' or just 'constant'.

Quoted 6 lines

Quoted 2 lines
In particular, I've noticed this happening when switching between AUTOREV and a fixed revision in a recipe.
This is simply just the way the system is designed. It checksums the intermediate steps as well as the endpoints. I guess in the above example the system might do "export Y" for example.

In that case, "Y" would end up in the package's sstate hash, and hence it would rebuild when Y changes.

I have a workaround now, multiple workarounds actually, but I'm trying to understand why the system would be designed this way. Currently I'm tempted to splatter X[vardepvalue]="${X}" all over the place just to get rid of this annoying behaviour.

Quoted 4 lines

There are alternative ways to do things but its simply not the way things were implemented. If you do need to collapse the dependency chain, you can use vardepvalue which was added for that reason. There are only a small number of places you really need to do that.

Lets make things more concrete. My distro.conf now reads:

FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"

Say I want to build packages with varying optimization levels. I consider myself pretty smart, so in this distro.conf file, I change that line to:

OPT_LEVEL ?= "-O2"
FULL_OPTIMIZATION = "${OPT_LEVEL} -pipe ${DEBUG_FLAGS}"

So now I can replace the optimization level of some packages by simply changing the OPT_LEVEL variable.

Nice, but this will trigger a rebuild of ALL packages, event though the build flags did not change at all!

I can't think of any reason why this would be desirable. Why is it designed this way?

Kind regards,

Mike Looijmans
System Expert

Please consider the environment before printing this e-mail

Christopher Larson

2015-10-06 15:40 UTC #6 lore raw

Quoted 59 lines · Mike Looijmans

Quoted 23 lines · Richard Purdie
Quoted 5 lines · Mike Looijmans
Quoted 2 lines · Richard Purdie
On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
...
For the bigger picture:
Quoted 14 lines
If I have a recipe that says:
X = "x"
And I refactor it a bit to read:
Y = "x"
X = "${Y}"
What is the logic behind having to rebuild the package now? Why does the 'source' of the contents matter to the hash? It won't generate different output, regardless whether X is 'calculated' or just 'constant'.
Quoted 9 lines
Quoted 4 lines
In particular, I've noticed this happening when switching between AUTOREV and
a fixed revision in a recipe.
This is simply just the way the system is designed. It checksums the intermediate steps as well as the endpoints. I guess in the above example the system might do "export Y" for example.
In that case, "Y" would end up in the package's sstate hash, and hence it would rebuild when Y changes.
I have a workaround now, multiple workarounds actually, but I'm trying to understand why the system would be designed this way. Currently I'm tempted to splatter X[vardepvalue]="${X}" all over the place just to get rid of this annoying behaviour.
There are alternative ways to do things but its simply not the way
Quoted 4 lines
things were implemented. If you do need to collapse the dependency chain, you can use vardepvalue which was added for that reason. There are only a small number of places you really need to do that.
Lets make things more concrete. My distro.conf now reads:
FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
Say I want to build packages with varying optimization levels. I consider myself pretty smart, so in this distro.conf file, I change that line to:
OPT_LEVEL ?= "-O2"
FULL_OPTIMIZATION = "${OPT_LEVEL} -pipe ${DEBUG_FLAGS}"

Quoted 8 lines

So now I can replace the optimization level of some packages by simply changing the OPT_LEVEL variable.
Nice, but this will trigger a rebuild of ALL packages, event though the build flags did not change at all!
I can't think of any reason why this would be desirable. Why is it designed this way?

This behavior has bugged me for quite a while, I'd love to see this change in the long term, assuming Richard isn't opposed to it. We should care about the results, not how we got there, generally.

Signature · 5 lines

Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics