RFC/RFT Checksums as part of stamp files
We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.
What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.
It also means if you change something in the core packaging, everything should repackage and so on.
The final key to enabling this is this commit:
This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.
I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.
For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.
This would solve one of Koen's major complaints about PR bumps in general once and for all but there is some effort needed to implement something that could provide this service.
Also, on the plus side, we could also now make image generation a "stamp" task with a stamp that will only change when the image inputs change. That would change a users expectations in some regard so again needs discussion.
Cheers,
Richard
Quoted 3 lines · Richard Purdie
We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.
... snip
Quoted 9 lines
My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.
For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.
How would this play with non-core packages/recipes, e.g. the ones that are only in my local trees?
Also, how could this work in a no-network-available environment?
Signature · 4 lines
------------------------------------------------------------ Gary Thomas | Consulting for the MLB Associates | Embedded world ------------------------------------------------------------
Quoted 11 lines · Gary Thomas
Quoted 3 lines · Richard Purdie
We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.
... snipQuoted 6 lines
My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.
Quoted 8 lines
Quoted 3 lines
For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.
How would this play with non-core packages/recipes, e.g. the ones that are only in my local trees?
Also, how could this work in a no-network-available environment?
Keep in mind this server solution is only needed where you are feeding the output of multiple build machines into one package feed and need to synchronise them with respect to versions. This is something you'd have to explicitly setup (just like the package feed itself).
If you are feeding to a package feed its assumed you have some kind of connectivity to it and that it can presumably host the synchronisation solution.
Local recipes would work just the same as core ones, they all behave the same way.
Yocto itself has no plans to host actual package feeds, we are likely to end up with some kind of public sstate cache but that doesn't need any synchronisation.
Cheers,
Richard
Quoted 19 lines
From: Richard Purdie
Sent: Friday, January 07, 2011 6:56 AMWe've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.
What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.
It also means if you change something in the core packaging, everything should repackage and so on.
The final key to enabling this is this commit:
http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259
b901cae11b0e5715a92a580c9432af52cc
Quoted 6 lines
This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.
I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.
I can help test it out too.
Quoted 5 lines
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...
Quoted 11 lines
My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.
For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.
Quoted 9 lines
This would solve one of Koen's major complaints about PR bumps in general once and for all but there is some effort needed to implement something that could provide this service.
Also, on the plus side, we could also now make image generation a "stamp" task with a stamp that will only change when the image inputs change. That would change a users expectations in some regard so again needs discussion.
Thanks
Kevin
Op 6 jan 2011, om 23:56 heeft Richard Purdie het volgende geschreven:
Quoted 18 lines
We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.
What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.
It also means if you change something in the core packaging, everything should repackage and so on.
The final key to enabling this is this commit:
This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.
Quoted 8 lines
I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
You saved me from writing a lengthy email about that :)
Quoted 9 lines
My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.
For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.
As a simpler short term solution, can the build "master" write the info to disk (e.g. sqlite) so we can put it in a layer and build "slaves" would just read that back? That we we only need a variable to either read (slaves) or write (master) to that file. It also involves as little as possible network access, since it sits in the metadata git.
regards,
Koen
Quoted 9 lines · Kevin
Quoted 4 lines
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...
They are potentially heavier, yes.
With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:
BB_SIGNATURE_HANDLER = "basichash"
to local.conf to enable this functionality (the default is "basic").
At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.
For reference, my test builds completed successfully.
Cheers,
Richard
Quoted 20 lines
From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 07, 2011 7:09 PMQuoted 9 lines · Kevin
Quoted 4 lines
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...
They are potentially heavier, yes.
With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:
Quoted 9 lines
BB_SIGNATURE_HANDLER= "basichash"to local.conf to enable this functionality (the default is "basic").
At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.
For reference, my test builds completed successfully.
my build finishes successfully too. I will manually bump PR to test its chain rebuild effect for some gut-feeling. :-)
Thanks
Kevin
Quoted 27 lines
From: Tian, Kevin
Sent: Friday, January 07, 2011 7:23 PMQuoted 24 lines
From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 07, 2011 7:09 PMQuoted 9 lines · Kevin
Quoted 4 lines
I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.
worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...
They are potentially heavier, yes.
With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:
BB_SIGNATURE_HANDLER= "basichash"to local.conf to enable this functionality (the default is "basic").
Quoted 9 lines
Quoted 5 lines
At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.
For reference, my test builds completed successfully.
my build finishes successfully too. I will manually bump PR to test its chain rebuild effect for some gut-feeling. :-)
I tried to bump PR for m4-native, and it's effectively rebuilt from task 743 of 1625 with lots of native recipes rebuilt accordingly, as expected.
Thanks
Kevin