Skip to contentSkip to conversation

sstate info

DISCUSSION Gary Thomas 5 participants 24 messages Jan 12, 2012 – Jan 15, 2012

lore
Share & tools
lore
#0 lore raw

I'm trying to understand why sstate sometimes [most times in my case] is not reusable. Comparing some of the .siginfo files gives me some info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426

How do I figure out what went into the hash(es) that changed?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 9 lines · Gary Thomas

I'm trying to understand why sstate sometimes [most times in my case] is not reusable. Comparing some of the .siginfo files gives me some info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo 
p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426

How do I figure out what went into the hash(es) that changed?

Try to look in stamps directory

something like:
tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

Cheers,

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#2 lore raw
Quoted 9 lines · Martin Jansa
Quoted 8 lines · Gary Thomas

I'm trying to understand why sstate sometimes [most times in my case] is not reusable. Comparing some of the .siginfo files gives me some info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
Quoted 6 lines
Quoted 1 line

How do I figure out what went into the hash(es) that changed?

Try to look in stamps directory

something like:
tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

How can I interpret the contents of that file?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
Quoted 10 lines · Gary Thomas
Quoted 9 lines · Martin Jansa
Quoted 8 lines · Gary Thomas

I'm trying to understand why sstate sometimes [most times in my case] is not reusable. Comparing some of the .siginfo files gives me some info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

How do I figure out what went into the hash(es) that changed?

Try to look in stamps directory

something like:
tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

How can I interpret the contents of that file?

find file for virtual:nativepseudo_1.2.bb.do_install with checksum 9edd41e331b29f08941225709c325a0d and with checksum 9865df7c4aa84fdb288c06faecf63426 and then compare them with bitbake-diffsigs again

or use bitbake-diffsigs for one sigdata file

Cheers,

Signature · 1 line
Martin 'JaMa' Jansa     jabber: [email protected]
#4 lore raw
Quoted 18 lines · Martin Jansa
Quoted 17 lines · Gary Thomas
Quoted 15 lines · Martin Jansa
Quoted 9 lines · Gary Thomas

I'm trying to understand why sstate sometimes [most times in my case] is not reusable. Comparing some of the .siginfo files gives me some info, e.g.

$ bitbake-diffsigs p60_poky/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-66ad8e10f260d506d065831a738c04a3_populate-sysroot.tgz.siginfo
p60_test_pass1/sstate-cache/sstate-pseudo-native-i686-linux-1.2-r4-i686-2-c7e01d52b467bbe3af0556a43806521a_populate-sysroot.tgz.siginfo
Hash for dependent task virtual:nativepseudo_1.2.bb.do_install changed from 9edd41e331b29f08941225709c325a0d to 9865df7c4aa84fdb288c06faecf63426

How do I figure out what went into the hash(es) that changed?

Try to look in stamps directory

something like:
tmp-eglibc/stamps/x86_64-linux/pseudo-native-1.2-r4.do_install.sigdata.504f9aac443e1f135aa2d1cbcf84c614

Quoted 8 lines
Quoted 1 line

How can I interpret the contents of that file?

find file for virtual:nativepseudo_1.2.bb.do_install with checksum 9edd41e331b29f08941225709c325a0d and with checksum 9865df7c4aa84fdb288c06faecf63426 and then compare them with bitbake-diffsigs again

or use bitbake-diffsigs for one sigdata file

I worked this down to the do_patch in pseudo-native:

$ bitbake-diffsigs p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
basehash changed from f5994d735ee2b909b72ee811364befc1 to 564be8db4146afa5fc3579dc4c1618d3
Variable DATE value changed from 20120110 to 20120111
Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021

Why is DATE as a variable important?
Why is it not ignored by this statement in ./meta/classes/patch.bbclass?
patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"

As is, it seems that the sstate cache is only good on the day it was made :-(

Here's what I tried yesterday:
* Create a new build, e.g.
% . /local/poky/oe-init-build-env /local/test_pass1 -- adjust local.conf to match desired target
% bitbake core-image-minimal
* Create a new build, using the previous one's sstate-cache % . /local/poky/oe-init-build-env /local/test_pass2 -- adjust local.conf to match desired target
-- enable sstate cache via these lines in local.conf:
SSTATE_MIRRORS ?= "\
file://.* file:///local/test_pass1/sstate-cache/"
% bitbake core-image-minimal

This all went to plan - the sstate cache was reused for every package except one. For some reason, dbus-1 had to be rebuilt.

Today, I tried it again, just like the second step. Low and behold, virtually none of the sstate packages were reused :-( This seems totally broken to me and has been filed as bug #1896

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw

First off, what version of Yocto are you running? I've made a lot of fixes post 1.1 release (others have made some as well ;)), and I believe lots of these fixes will be in the 1.1.1 release. They should all be in master, although I've not tested master sstate-cache in a while now.

Quoted 10 lines · Gary Thomas

I worked this down to the do_patch in pseudo-native:

$ bitbake-diffsigs
p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c
p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
basehash changed from f5994d735ee2b909b72ee811364befc1 to
564be8db4146afa5fc3579dc4c1618d3
Variable DATE value changed from 20120110 to 20120111
Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed
from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021

This is telling you that you need to look at the quilt-native task now. Nothing is wrong about the signature for psuedo-natve.

Quoted 3 lines

Why is DATE as a variable important?
Why is it not ignored by this statement in ./meta/classes/patch.bbclass? patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

Quoted 2 lines

As is, it seems that the sstate cache is only good on the day it was made :-(

This is possible, but you need to check the upstream task that changed. Since psuedo-native depends on quilt-native it's getting a new signature since something that changed in quilt-native might effect psuedo-native.

Quoted 13 lines

Here's what I tried yesterday:
 * Create a new build, e.g.
   % . /local/poky/oe-init-build-env /local/test_pass1 -- adjust local.conf to match desired target
   % bitbake core-image-minimal
 * Create a new build, using the previous one's sstate-cache % . /local/poky/oe-init-build-env /local/test_pass2 -- adjust local.conf to match desired target
   -- enable sstate cache via these lines in local.conf:
        SSTATE_MIRRORS ?= "\
        file://.* file:///local/test_pass1/sstate-cache/" % bitbake core-image-minimal

Quoted 2 lines

This all went to plan - the sstate cache was reused for every package except one.  For some reason, dbus-1 had to be rebuilt.

This should work. I'm building on one machine and reusing on others including native cache. I'm doing thing on something close to what 1.1.1 will look like.

Quoted 4 lines

Today, I tried it again, just like the second step.  Low and behold, virtually
none of the sstate packages were reused :-(  This seems totally broken to me and has been filed as bug #1896

-M

#6 lore raw
Quoted 21 lines · On 2012-01-12 09:53, McClintock Matthew-B29882 wrote:

First off, what version of Yocto are you running? I've made a lot of fixes post 1.1 release (others have made some as well ;)), and I believe lots of these fixes will be in the 1.1.1 release. They should all be in master, although I've not tested master sstate-cache in a while now.

Quoted 10 lines · Gary Thomas

I worked this down to the do_patch in pseudo-native:

$ bitbake-diffsigs
p60_poky/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.6e97a41d825568db298fdc9dbaed563c
p60_test_pass1/tmp/stamps/i686-linux/pseudo-native-1.2-r4.do_patch.sigdata.a0d4e99625c211f4f1d793213014ac07
basehash changed from f5994d735ee2b909b72ee811364befc1 to
564be8db4146afa5fc3579dc4c1618d3
Variable DATE value changed from 20120110 to 20120111
Hash for dependent task quilt-native_0.50.bb.do_populate_sysroot changed
from 9307106f3c5dfd49d953e206cb1b6da5 to 24151a370083d625b84a2afb77955021

This is telling you that you need to look at the quilt-native task now. Nothing is wrong about the signature for psuedo-natve.

Quoted 6 lines
Quoted 3 lines

Why is DATE as a variable important?
Why is it not ignored by this statement in ./meta/classes/patch.bbclass?
patch_do_patch[vardepsexclude] = "DATE SRCDATE PATCHRESOLVE"

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

It seems that DATE needs to be ignored other places as well. Digging further, I find:

$ bitbake-diffsigs p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
basehash changed from 4863b21531f8236184a7c3b16da3ea13 to 7ee430d1251148b2509ef2aefb6e5443
Variable DATE value changed from 20120110 to 20120111

which is why pseudo-native changed which is why almost every other recipe signature changed :-(

Quoted 22 lines
Quoted 3 lines

As is, it seems that the sstate cache is only good on the day it was made :-(

This is possible, but you need to check the upstream task that changed. Since psuedo-native depends on quilt-native it's getting a new signature since something that changed in quilt-native might effect psuedo-native.

Quoted 13 lines

Here's what I tried yesterday:
* Create a new build, e.g.
% . /local/poky/oe-init-build-env /local/test_pass1 -- adjust local.conf to match desired target
% bitbake core-image-minimal
* Create a new build, using the previous one's sstate-cache % . /local/poky/oe-init-build-env /local/test_pass2 -- adjust local.conf to match desired target
-- enable sstate cache via these lines in local.conf:
SSTATE_MIRRORS ?= "\
file://.* file:///local/test_pass1/sstate-cache/"
% bitbake core-image-minimal

Quoted 6 lines
Quoted 2 lines

This all went to plan - the sstate cache was reused for every package except one. For some reason, dbus-1 had to be rebuilt.

This should work. I'm building on one machine and reusing on others including native cache. I'm doing thing on something close to what 1.1.1 will look like.

It did work *YESTERDAY*, but the exact same steps today do not work. There were no changes to my poky source tree and no new packages were downloaded.

Quoted 7 lines
Quoted 5 lines

Today, I tried it again, just like the second step. Low and behold, virtually
none of the sstate packages were reused :-( This seems totally broken to me and has been filed as bug #1896

-M

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#7 lore raw
Quoted 4 lines · Gary Thomas
Quoted 2 lines

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

Quoted 8 lines

It seems that DATE needs to be ignored other places as well.  Digging further, I find:

$ bitbake-diffsigs
p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
7ee430d1251148b2509ef2aefb6e5443

You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf which should ignore DATE in all recipes. I thought this was there, you can check the history to see...

-M

#8 lore raw
Quoted 14 lines · On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
Quoted 4 lines · Gary Thomas
Quoted 2 lines

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

Quoted 8 lines

It seems that DATE needs to be ignored other places as well. Digging further, I find:

$ bitbake-diffsigs
p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca
p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0
basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
7ee430d1251148b2509ef2aefb6e5443
Quoted 3 lines

You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf which should ignore DATE in all recipes. I thought this was there, you can check the history to see...

I'm trying this now. Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#9 lore raw
Quoted 21 lines · Gary Thomas
Quoted 20 lines · On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
Quoted 16 lines · Gary Thomas
Quoted 3 lines

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

It seems that DATE needs to be ignored other places as well.  Digging further, I find:

$ bitbake-diffsigs

p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca

p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0 basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
7ee430d1251148b2509ef2aefb6e5443

Quoted 7 lines
Quoted 3 lines

You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf which should ignore DATE in all recipes. I thought this was there, you can check the history to see...

I'm trying this now.  Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Signature · 1 line
Christopher Larson
#10 lore raw
Quoted 5 lines · Chris Larson
Quoted 2 lines

I'm trying this now.  Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

This would be a good way to test it out...

-M

#11 lore raw
In reply to Chris Larson
Quoted 27 lines · Chris Larson
Quoted 26 lines · Gary Thomas
Quoted 23 lines · On 2012-01-12 10:23, McClintock Matthew-B29882 wrote:
Quoted 9 lines · Gary Thomas
Quoted 3 lines

DATE is not important, that is why we are explicitly telling the signature to ignore this variable.

It seems that DATE needs to be ignored other places as well. Digging further, I find:

Quoted 7 lines
$ bitbake-diffsigs

p60_poky/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.216968eaca38198b88a7520be02e3fca

p60_test_pass1/tmp/stamps/i686-linux/quilt-native-0.50-r0.do_patch.sigdata.0ce7898e700de80ff8afcb0850ec4ed0 basehash changed from 4863b21531f8236184a7c3b16da3ea13 to
7ee430d1251148b2509ef2aefb6e5443

You might just try adding to BB_HASHBASE_WHITELIST in bitbake.conf which should ignore DATE in all recipes. I thought this was there, you can check the history to see...

Quoted 5 lines
Quoted 2 lines

I'm trying this now. Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#12 lore raw
Quoted 5 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Quoted 7 lines

Actually, I didn't think of that.  I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like.  The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Are the hashes the same too?

-M

#13 lore raw
Quoted 9 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 8 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

Quoted 6 lines
Quoted 4 lines

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cheers,

Richard

#14 lore raw
Quoted 15 lines · Richard Purdie
Quoted 14 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 12 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Quoted 4 lines
Quoted 1 line

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#15 lore raw
Quoted 16 lines · Gary Thomas
Quoted 15 lines · Richard Purdie
Quoted 14 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 12 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

I won't know until later as this change itself invalidates the sstate cache, so I have to start a full rebuild to test it properly.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#16 lore raw
Quoted 19 lines · Gary Thomas
Quoted 18 lines · Gary Thomas
Quoted 17 lines · Richard Purdie
Quoted 15 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 5 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Quoted 7 lines

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Are the hashes the same too?

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

I won't know until later as this change itself invalidates the sstate cache, so I have to start a full rebuild to test it properly.

Richard's change works just as well as adding DATE to BB_HASH_WHITELIST.

In my minimal test, there are three packages being rebuilt: dbus-1, base-files and shadow-sysroot. It's still not clear to me why this happened. I've put the tmp/stamps trees at http://www.mlbassoc.com/misc/poky_stamps.tgz
This has two directories in it
p60_test_pass1 - the original tree, build without any sstate p60_test_pass2 - the dependent tree, build using p60_test_pass1 as sstate
Perhaps someone can give me a clue why these three packages were
rebuilt. Note: there were no changes to the source repository
between the two builds and I always build with BB_NO_NETWORK="1"

Thanks

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#17 lore raw
Quoted 1 line · Gary Thomas

Richard's change works just as well as adding DATE to BB_HASH_WHITELIST.

Great!

Quoted 9 lines

In my minimal test, there are three packages being rebuilt: dbus-1, base-files and shadow-sysroot. It's still not clear to me why this happened. I've put the tmp/stamps trees at http://www.mlbassoc.com/misc/poky_stamps.tgz
This has two directories in it
p60_test_pass1 - the original tree, build without any sstate p60_test_pass2 - the dependent tree, build using p60_test_pass1 as sstate
Perhaps someone can give me a clue why these three packages were
rebuilt. Note: there were no changes to the source repository
between the two builds and I always build with BB_NO_NETWORK="1"

I suspect its useradd related and there is already an open bug:

http://bugzilla.yoctoproject.org/show_bug.cgi?id=1721

Cheers,

Richard

#18 lore raw
In reply to Gary Thomas
Quoted 16 lines · Gary Thomas
Quoted 15 lines · Richard Purdie
Quoted 14 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 12 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

I've also noticed a problem with gcc being rebuilt. It seems to be from do_headerfix() and:

meta/recipes-devtools/gcc/gcc-configure-common.inc +3 -0
index d014980..3a82720 100644
@@ -77,6 +77,9 @@ do_headerfix () {
 
 addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
 do_configure_prepend () {
        # teach gcc to find correct target includedir when checking libc ssp support
        mkdir -p ${B}/gcc

"fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:

$ ls *headerfix*
gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
$ bitbake-diffsigs *headerfix*
Dependency on task gcc-cross_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed
#19 lore raw
Quoted 42 lines · Richard Purdie
Quoted 16 lines · Gary Thomas
Quoted 15 lines · Richard Purdie
Quoted 14 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 12 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

I've also noticed a problem with gcc being rebuilt. It seems to be from do_headerfix() and:

meta/recipes-devtools/gcc/gcc-configure-common.inc +3 -0
index d014980..3a82720 100644
@@ -77,6 +77,9 @@ do_headerfix () {
 
  addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
  do_configure_prepend () {
         # teach gcc to find correct target includedir when checking libc ssp support
         mkdir -p ${B}/gcc
Quoted 8 lines

"fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:

$ ls *headerfix*
gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
$ bitbake-diffsigs *headerfix*
Dependency on task gcc-cross_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed

While you're looking at gcc, I noticed something else. I have multiple MACHINE configurations which are all the same basic hardware (OMAP3530), the only difference being kernel settings and other minor target variations. As far as things like the compiler and common ARM tools, they are identical.

I tried sharing the sstate between two of these and for the most part, all was well - only the truly machine dependent recipes were rebuilt. Except for gcc-cross-4.6.2 which seems to have a dependency on the MACHINE variable. This cascaded into also building a few other packages (glib, libtool, shadow? and udev) [note: I was building a board with SDK tools which may have brought some of these in]. Does this seem reasonable to you? I would hope that these packages, which are placed in the architecture (armv7a-vfp-neon-amltd-linux-gnueabi) tree should *not* depend on MACHINE.

Thanks for your help with this.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#20 lore raw
Quoted 17 lines · Gary Thomas

While you're looking at gcc, I noticed something else.  I have multiple MACHINE configurations which are all the same basic hardware (OMAP3530), the only difference being kernel settings and other minor target variations.  As far as things like the compiler and common ARM tools, they are identical.

I tried sharing the sstate between two of these and for the most part, all was well - only the truly machine dependent recipes were rebuilt.  Except for gcc-cross-4.6.2 which seems to have a dependency on the MACHINE variable.  This cascaded into also building a few other packages (glib, libtool, shadow? and udev)  [note: I was building a board with SDK tools which may have brought some of these in].  Does this seem reasonable to you?  I would hope that these packages, which are placed in the architecture (armv7a-vfp-neon-amltd-linux-gnueabi) tree should *not* depend on MACHINE.

Quoted 1 line

Thanks for your help with this.

More than likely the MACHINE variable should not be a dependency. Which variable is depending on MACHINE? You should do some more investigation.

-M

#21 lore raw
In reply to Gary Thomas
Quoted 16 lines · Gary Thomas
Quoted 15 lines · Richard Purdie
Quoted 14 lines · On Fri, 2012-01-13 at 15:40 +0000, McClintock Matthew-B29882 wrote:
Quoted 12 lines · Gary Thomas
Quoted 3 lines

You can't just override DATE in local.conf? I don't see the point in holding off, unless your builds will take till tomorrow to finish?

Actually, I didn't think of that. I also didn't see this message until Friday (the next day) due to some network issues, so it's moot.

That notwithstanding, I tried it today (new DATE) and it behaved as I would like. The only problem was that dbus-1 is still being rebuilt. I compared the siginfo files between the two builds and they are identical, so I don't know why.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Are the hashes the same too?

I've found the root cause of this problem and have posted a fix on the OE-Core mailing list.

Cool, I'll start testing this now :-)

I've also noticed a problem with gcc being rebuilt. It seems to be from do_headerfix() and:

meta/recipes-devtools/gcc/gcc-configure-common.inc +3 -0
index d014980..3a82720 100644
@@ -77,6 +77,9 @@ do_headerfix () {
 
 addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
 do_configure_prepend () {
        # teach gcc to find correct target includedir when checking libc ssp support
        mkdir -p ${B}/gcc

"fixes" it. I merged the other fix but this one needs a little bit more thought. bitbake-diffsigs can also mislead:

$ ls *headerfix*
gcc-cross-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.84c0ca9d0fc07438f453910901a222b6
$ bitbake-diffsigs *headerfix*
Dependency on task gcc-cross_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was removed

Note that the hashes are the same and these are in fact identical. The dependency hashes *do* match and its bitbake-diffsigs that is wrong here as the file changed names.

Cheers,

Richard

#22 lore raw
In reply to Gary Thomas
Quoted 2 lines · Gary Thomas

I'm trying this now.  Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

Yes, it's hard (long) to test this. Then you will want to wait another day to know for sure the DATE dependency is gone to know it's work for sure. I'd recommend purchasing a large machine to do this testing on.... it's well worth the investment ;)

-M

#23 lore raw
Quoted 8 lines · On 2012-01-12 14:13, McClintock Matthew-B29882 wrote:
Quoted 2 lines · Gary Thomas

I'm trying this now. Of course, touching that variable invalidates the sstate cache, so I won't know until tomorrow if it works.

Yes, it's hard (long) to test this. Then you will want to wait another day to know for sure the DATE dependency is gone to know it's work for sure. I'd recommend purchasing a large machine to do this testing on.... it's well worth the investment ;)

I have many such large machines at my disposal, but sadly, no time machine :-)

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------