devtool: Upgrade feature

This is a new devtool's feature, which upgrades a recipe to a particular version number. Code was taken from [1] where some modifications were done (remove all email, buildhistory and statistics features, use devtool logger instead of AUH logger) to adapt the devtool framework. Once the upgrade is over, the new recipe will be located under the devtool's workspace. This is a first approach to this feature; pending tasks include:

1. The AUH [1] is used to rename and update the recipe. AUH is not using the lib/oe/recipeutils library to do this work. AUH ported code should use this library which is the one being used by the rest of the devtool features.

2. Currently, when 'update-recipe' is executed, the recipe under workspace is updated with latest commits. The only task missing is to replace the new recipe with the old one, commonly located under the meta layer.

3. When patches fail to apply, follow the PATCHRESOLVE behavior instead of just failing.

4. Patches most of the time do not apply correctly on the new recipe version, so include a command line option to indicate the system not to apply these, so it can be applied manually later on.

Hi Leo,

Quoted 17 lines · On Tuesday 11 August 2015 14:04:14 [email protected]…

This is a new devtool's feature, which upgrades a recipe to a particular version number. Code was taken from [1] where some modifications were done (remove all email, buildhistory and statistics features, use devtool logger instead of AUH logger) to adapt the devtool framework. Once the upgrade is over, the new recipe will be located under the devtool's workspace. This is a first approach to this feature; pending tasks include:

1. The AUH [1] is used to rename and update the recipe. AUH is not using the lib/oe/recipeutils library to do this work. AUH ported code should use this library which is the one being used by the rest of the devtool features.

2. Currently, when 'update-recipe' is executed, the recipe under workspace is updated with latest commits. The only task missing is to replace the new recipe with the old one, commonly located under the meta layer.

Quoted 6 lines

3. When patches fail to apply, follow the PATCHRESOLVE behavior instead of just failing.

4. Patches most of the time do not apply correctly on the new recipe version, so include a command line option to indicate the system not to apply these, so it can be applied manually later on.

So, this is a good first implementation and gives an idea of how the command will work - and I'm quite keen for us to have this, I've had several people asking me recently about when we'll be adding it, so it's definitely something we want.

However, I'm concerned about the volume of code we're adding here, some of which duplicates what we already have in devtool or elsewhere in lib/oe. I know this is the easiest approach and you've noted it's todo item #1 above to work on - I'll be a lot happier if you can improve that before the final version.

In particular, I see the upgrade() function is duplicating a bunch of code from modify() - we should split the common parts out to separate function(s) that can be called from both places instead.

Some other comments:

* I can see it's actually making changes to the original recipe - I found this because I'd run a devtool upgrade on dropbear and specified the wrong version to --version Ctrl+C'd out during the fetch part, and the original dropbear.inc was modified instead of the one in the workspace. We don't want to touch the original recipe, not until you run update-recipe at least.

* As far as the source tree is concerned I think that this is doing things slightly backwards - it's using the AUH code to do the upgrade, and then it's extracting the code. What I'd like to see happen is it extract the code for the old version (including applying patches), use the AUH code to figure out how to fetch the new version, fetch it into a different branch (assuming it's not already there since it was fetched from git), then rebase the patches onto the new version - the user can then use git to fix things up if patches don't apply.

* I'm not sure if using an existing source tree is a reasonable default for this command - I'm not even sure it's something people will want to do at all in the context of an upgrade. I'm willing to be convinced otherwise though - any opinions (from anyone)?

* We need oe-selftest tests for this that test as much of the functionality as possible. This may require adding some simple "synthetic" recipes to meta- selftest so that we have something that's always there to upgrade. Based upon my recent experience in devtool (with my own code), the sooner we add these the better.

* I'd like to see the implementation for this in its own file rather than in standard.py.

* Probably just a result of this being WIP, but I found that if I don't specify a version with --version then it says "NOTE: Upgrading to None" and then fails with "ERROR: cannot concatenate 'str' and 'NoneType' objects".

Cheers,
Paul

Quoted 42 lines · Paul Eggleton

Quoted 20 lines · On Tuesday 11 August 2015 14:04:14 [email protected]>…

This is a new devtool's feature, which upgrades a recipe to a particular version number. Code was taken from [1] where some modifications
were done (remove all email, buildhistory and statistics features, use devtool logger instead of AUH logger) to adapt the devtool framework. Once the upgrade is over, the new recipe will be located under the devtool's workspace. This is a first approach to this feature; pending tasks include:

1. The AUH [1] is used to rename and update the recipe. AUH is not using the lib/oe/recipeutils library to do this work. AUH ported code should use this library which is the one being used by the rest of the devtool features.

2. Currently, when 'update-recipe' is executed, the recipe under workspace
is updated with latest commits. The only task missing is to replace the new
recipe with the old one, commonly located under the meta layer.

Quoted 7 lines

3. When patches fail to apply, follow the PATCHRESOLVE behavior instead of
just failing.

4. Patches most of the time do not apply correctly on the new recipe version, so include a command line option to indicate the system not to apply these, so it can be applied manually later on.

So, this is a good first implementation and gives an idea of how the command will work - and I'm quite keen for us to have this, I've had several people asking me recently about when we'll be adding it, so it's definitely something we want.

However, I'm concerned about the volume of code we're adding here, some of which duplicates what we already have in devtool or elsewhere in lib/oe. I know this is the easiest approach and you've noted it's todo item #1 above to work on - I'll be a lot happier if you can improve that before the final version.

Quoted 3 lines

In particular, I see the upgrade() function is duplicating a bunch of code from modify() - we should split the common parts out to separate function(s) that can be called from both places instead.

As you noticed, there is code duplication specially on the standard.py. Code on this file needs to be refactor somehow so the function upgrade can call these new functions.

Quoted 8 lines

Some other comments:

* I can see it's actually making changes to the original recipe - I found this because I'd run a devtool upgrade on dropbear and specified the wrong version to --version Ctrl+C'd out during the fetch part, and the original dropbear.inc was modified instead of the one in the workspace. We don't want to touch the original recipe, not until you run update-recipe at least.

Ok. This is definitely a bug. All work must be done in the workspace. This is contrary to the AUH code, which works on poky default layers, so this bug may be something I did not change on the ported code.

Quoted 8 lines

* As far as the source tree is concerned I think that this is doing things slightly backwards - it's using the AUH code to do the upgrade, and then it's extracting the code. What I'd like to see happen is it extract the code for the old version (including applying patches), use the AUH code to figure out how to fetch the new version, fetch it into a different branch (assuming it's not already there since it was fetched from git), then rebase the patches onto the new version - the user can then use git to fix things up if patches don't apply.

got it. Looks good to me. I also though using the recipetool script (instead of AUH), but this one creates a recipe from scratch so it is not really useful at this moment.

Quoted 10 lines

* I'm not sure if using an existing source tree is a reasonable default for this command - I'm not even sure it's something people will want to do at all in the context of an upgrade. I'm willing to be convinced otherwise though - any opinions (from anyone)?

* We need oe-selftest tests for this that test as much of the functionality as possible. This may require adding some simple "synthetic" recipes to meta- selftest so that we have something that's always there to upgrade. Based upon my recent experience in devtool (with my own code), the sooner we add these the better.

good point. I will add these on v2.

Quoted 2 lines

* I'd like to see the implementation for this in its own file rather than in standard.py.

Good. What about the other features (add, update-recipe, modify, etc.)? will these remain in the same place?

Quoted 4 lines

* Probably just a result of this being WIP, but I found that if I don't specify a version with --version then it says "NOTE: Upgrading to None" and
then fails with "ERROR: cannot concatenate 'str' and 'NoneType' objects".

Bug. I will also add it on V2.

Quoted 86 lines · On Thursday 13 August 2015 12:45:08 Leonardo Sandoval wrote:

Quoted 61 lines · Paul Eggleton

Quoted 34 lines · On Tuesday 11 August 2015 14:04:14 [email protected]>…

This is a new devtool's feature, which upgrades a recipe to a particular version number. Code was taken from [1] where some modifications
were done (remove all email, buildhistory and statistics features,
use devtool logger instead of AUH logger) to adapt the devtool framework.
Once the upgrade is over, the new recipe will be located under the
devtool's workspace. This is a first approach to this feature; pending
tasks include:

1. The AUH [1] is used to rename and update the recipe. AUH is not
using the lib/oe/recipeutils library to do this work. AUH ported code
should use this library which is the one being used by the rest of the
devtool features.

2. Currently, when 'update-recipe' is executed, the recipe under
workspace
is updated with latest commits. The only task missing is to replace the
new
recipe with the old one, commonly located under the meta layer.

3. When patches fail to apply, follow the PATCHRESOLVE behavior instead
of
just failing.

Quoted 5 lines

4. Patches most of the time do not apply correctly on the new recipe
version, so include a command line option to indicate the system not to
apply these, so it can be applied manually later on.

So, this is a good first implementation and gives an idea of how the
command will work - and I'm quite keen for us to have this, I've had
several people asking me recently about when we'll be adding it, so it's
definitely something we want.

However, I'm concerned about the volume of code we're adding here, some of
which duplicates what we already have in devtool or elsewhere in lib/oe. I
know this is the easiest approach and you've noted it's todo item #1 above
to work on - I'll be a lot happier if you can improve that before the
final version.

Quoted 4 lines

In particular, I see the upgrade() function is duplicating a bunch of code
from modify() - we should split the common parts out to separate function(s) that can be called from both places instead.

As you noticed, there is code duplication specially on the standard.py.
Code on this file needs to be refactor somehow so the function upgrade
can call these new functions.

Quoted 12 lines

Some other comments:

* I can see it's actually making changes to the original recipe - I found
this because I'd run a devtool upgrade on dropbear and specified the
wrong version to --version Ctrl+C'd out during the fetch part, and the
original dropbear.inc was modified instead of the one in the workspace.
We don't want to touch the original recipe, not until you run update-recipe at least.

Quoted 5 lines

Ok. This is definitely a bug. All work must be done in the workspace.
This is contrary to the AUH code, which works on poky default layers, so
this bug may be something I did not change on the ported code.

That's what I'd assume as well yes.

Quoted 16 lines

Quoted 15 lines

* As far as the source tree is concerned I think that this is doing things
slightly backwards - it's using the AUH code to do the upgrade, and then
it's extracting the code. What I'd like to see happen is it extract the
code for the old version (including applying patches), use the AUH code
to figure out how to fetch the new version, fetch it into a different
branch (assuming it's not already there since it was fetched from git),
then rebase the patches onto the new version - the user can then use git
to fix things up if patches don't apply.

Quoted 4 lines

got it. Looks good to me. I also though using the recipetool script (instead of AUH), but this one creates a recipe from scratch so it is
not really useful at this moment.

For pointers, look at how "devtool extract" and "devtool modify" work (for the
extraction part, they use the same _extract_source() function).

Quoted 8 lines

Quoted 8 lines

* I'm not sure if using an existing source tree is a reasonable default
for this command - I'm not even sure it's something people will want to do
at all in the context of an upgrade. I'm willing to be convinced otherwise
though - any opinions (from anyone)?

Quoted 14 lines

Quoted 7 lines

* We need oe-selftest tests for this that test as much of the functionality as possible. This may require adding some simple "synthetic" recipes to meta- selftest so that we have something that's
always there to upgrade. Based upon my recent experience in devtool (with
my own code), the sooner we add these the better.

good point. I will add these on v2.

Quoted 3 lines

* I'd like to see the implementation for this in its own file rather than
in standard.py.

Quoted 3 lines

Good. What about the other features (add, update-recipe, modify, etc.)?
will these remain in the same place?

At some point we might refactor these out, but I don't know if I'd make it a
priority at the moment.

Quoted 9 lines

Quoted 7 lines

* Probably just a result of this being WIP, but I found that if I don't
specify a version with --version then it says "NOTE: Upgrading to None"
and
then fails with "ERROR: cannot concatenate 'str' and 'NoneType' objects".

Bug. I will also add it on V2.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre