Skip to contentSkip to conversation

RFC/RFT Checksums as part of stamp files

DISCUSSION Richard Purdie 4 participants 8 messages Jan 6, 2011 – Jan 7, 2011

lore
Share & tools
lore
#0 lore raw

We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.

What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.

It also means if you change something in the core packaging, everything should repackage and so on.

The final key to enabling this is this commit:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259b901cae11b0e5715a92a580c9432af52cc

This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.

I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.

For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.

This would solve one of Koen's major complaints about PR bumps in general once and for all but there is some effort needed to implement something that could provide this service.

Also, on the plus side, we could also now make image generation a "stamp" task with a stamp that will only change when the image inputs change. That would change a users expectations in some regard so again needs discussion.

Cheers,

Richard

#1 lore raw
Quoted 3 lines · Richard Purdie

We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.

... snip

Quoted 9 lines

My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.

For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.

How would this play with non-core packages/recipes, e.g. the ones that are only in my local trees?

Also, how could this work in a no-network-available environment?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#2 lore raw
Quoted 11 lines · Gary Thomas
Quoted 3 lines · Richard Purdie

We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.

    ... snip
Quoted 6 lines

My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.

Quoted 8 lines
Quoted 3 lines

For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.

How would this play with non-core packages/recipes, e.g. the ones that are only in my local trees?

Also, how could this work in a no-network-available environment?

Keep in mind this server solution is only needed where you are feeding the output of multiple build machines into one package feed and need to synchronise them with respect to versions. This is something you'd have to explicitly setup (just like the package feed itself).

If you are feeding to a package feed its assumed you have some kind of connectivity to it and that it can presumably host the synchronisation solution.

Local recipes would work just the same as core ones, they all behave the same way.

Yocto itself has no plans to host actual package feeds, we are likely to end up with some kind of public sstate cache but that doesn't need any synchronisation.

Cheers,

Richard

#3 lore raw
In reply to Richard Purdie
Quoted 19 lines

From: Richard Purdie
Sent: Friday, January 07, 2011 6:56 AM

We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.

What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.

It also means if you change something in the core packaging, everything should repackage and so on.

The final key to enabling this is this commit:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259
b901cae11b0e5715a92a580c9432af52cc

Quoted 6 lines

This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.

I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.

I can help test it out too.

Quoted 5 lines

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...

Quoted 11 lines

My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.

For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.

Quoted 9 lines

This would solve one of Koen's major complaints about PR bumps in general once and for all but there is some effort needed to implement something that could provide this service.

Also, on the plus side, we could also now make image generation a "stamp" task with a stamp that will only change when the image inputs change. That would change a users expectations in some regard so again needs discussion.

Thanks
Kevin

#7 lore raw
In reply to Richard Purdie

Op 6 jan 2011, om 23:56 heeft Richard Purdie het volgende geschreven:

Quoted 18 lines

We've been working on checksums and sstate for a while, one of the pieces we've not put in place so far is using the checksums in the stamp files.

What this would mean in English is the end of PR bumps since when you change the recipe, the checksums of the appropriate tasks change and everything automatically rebuilds that changed.

It also means if you change something in the core packaging, everything should repackage and so on.

The final key to enabling this is this commit:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/bitbake-wip&id=b82259b901cae11b0e5715a92a580c9432af52cc

This hasn't been merged to master yet but we should discuss when this change could/should be merged and under what criteria.

Quoted 8 lines

I've only just started testing full builds with the change myself so keep that in mind. I'd appreciate testing but the change is still very new and likely has bugs to iron out.

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

You saved me from writing a lengthy email about that :)

Quoted 9 lines

My proposal for dealing with this will be to start having the packaging backend take ownership of PE and PR fields. These should be auto incremented when the build changes. That in itself is easy enough if you have one build area, the tricky bit is where you have multiple build sources feeding into a common packages repo like Angstrom.

For this we're going to need some kind of network service that takes a version and provides the PE/PR values back, incrementing them each time it sees a new version/checksum.

As a simpler short term solution, can the build "master" write the info to disk (e.g. sqlite) so we can put it in a layer and build "slaves" would just read that back? That we we only need a variable to either read (slaves) or write (master) to that file. It also involves as little as possible network access, since it sits in the metadata git.

regards,

Koen

#4 lore raw
In reply to Tian, Kevin
Quoted 9 lines · Kevin
Quoted 4 lines

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...

They are potentially heavier, yes.

With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:

BB_SIGNATURE_HANDLER = "basichash"

to local.conf to enable this functionality (the default is "basic").

At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.

For reference, my test builds completed successfully.

Cheers,

Richard

#5 lore raw
Quoted 20 lines

From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 07, 2011 7:09 PM

Quoted 9 lines · Kevin
Quoted 4 lines

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...

They are potentially heavier, yes.

With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:

Quoted 9 lines

BB_SIGNATURE_HANDLER = "basichash"

to local.conf to enable this functionality (the default is "basic").

At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.

For reference, my test builds completed successfully.

my build finishes successfully too. I will manually bump PR to test its chain rebuild effect for some gut-feeling. :-)

Thanks
Kevin

#6 lore raw
Quoted 27 lines

From: Tian, Kevin
Sent: Friday, January 07, 2011 7:23 PM

Quoted 24 lines

From: Richard Purdie [mailto:[email protected]]
Sent: Friday, January 07, 2011 7:09 PM

Quoted 9 lines · Kevin
Quoted 4 lines

I'm also aware that this idea isn't going to be universally liked, particularly I can guess that Koen is going to have concerns about this and Angstrom since the lack of PR bumps means package feeds could start to suffer.

worthy of a config option? We may need see from real use whether this will trigger many unnecessary rebuilds and how much impact to people's daily work. Basically the incremental builds will be much more heavier than before, though this in concept is more correct...

They are potentially heavier, yes.

With this and other factors in mind, I've merged a new signature generator class into poky with this as the only added feature. Now with poky's master branch you can add:

BB_SIGNATURE_HANDLER = "basichash"

to local.conf to enable this functionality (the default is "basic").

Quoted 9 lines
Quoted 5 lines

At some point we are likely to decide to change the default in bitbake.conf but this makes it easier for people to test.

For reference, my test builds completed successfully.

my build finishes successfully too. I will manually bump PR to test its chain rebuild effect for some gut-feeling. :-)

I tried to bump PR for m4-native, and it's effectively rebuilt from task 743 of 1625 with lots of native recipes rebuilt accordingly, as expected.

Thanks
Kevin