Allow oe-pkgdata-util package-info to display more info
cover Allow oe-pkgdata-util package-info to display more info
After a build in our autobuilder, we use oe-pkgdata-util package-info -f package.manifest to store a file with information about each installed package. This is typically used to compare builds later on. Since not every difference to a package is explained by its version, we have found it beneficial to also include the SRC_URI in the generated file.
This patch set adds SRC_URI to the pkgdata that is stored for each package, and adds a new option to oe-pkgdata-util package-info, -e <var>, that can be used to display extra variables from the pkgdata.
//Peter
The following changes since commit 0fcbb4d0ec26ef24db78190dcdc5568af3beaa14:
GNU_MIRROR: switch from ftp to https (2017-05-11 16:56:00 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib pkj/oe-pkgdata-util_extra
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/oe-pkgdata-util_extra
Peter Kjellerstedt (2):
package.bbclass: Add SRC_URI to pkgdata
oe-pkgdata-util: package-info: Allow extra variables to be displayed
meta/classes/package.bbclass | 1 + scripts/oe-pkgdata-util | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
Signature · 1 line
2.12.0
PATCH 1/2 package.bbclass: Add SRC_URI to pkgdata
Emit SRC_URI variable in pkgdata. Having this available is beneficial when comparing the information generated for two different builds and looking for changes not directly related to the package versions.
--- meta/classes/package.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes/package.bbclass +1 -0
@@ -1370,6 +1370,7 @@ python emit_pkgdata() { write_if_exists(sf, pkg, 'LICENSE') write_if_exists(sf, pkg, 'DESCRIPTION') write_if_exists(sf, pkg, 'SUMMARY') + write_if_exists(sf, pkg, 'SRC_URI') write_if_exists(sf, pkg, 'RDEPENDS') rprov = write_if_exists(sf, pkg, 'RPROVIDES') write_if_exists(sf, pkg, 'RRECOMMENDS')
Signature · 1 line
2.12.0
PATCH 2/2 oe-pkgdata-util: package-info: Allow extra variables to be displayed
In reply to Peter KjellerstedtBy specifying the -e <var> option one or multiple times, extra variables available in the pkgdata can be displayed, e.,g, oe-pkgdata-util package-info -e SUMMARY -e SRC_URI .... The extra variables displayed are quoted since some of them may contain whitespace.
--- scripts/oe-pkgdata-util | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
scripts/oe-pkgdata-util +9 -2
@@ -325,8 +325,14 @@ def package_info(args): recipe_version = recipe_version + "-" + mappings[pkg]['PR'] pkg_size = mappings[pkg]['PKGSIZE'] - items.append("%s %s %s %s %s" % - (pkg, pkg_version, recipe, recipe_version, pkg_size)) + line = "%s %s %s %s %s" % (pkg, pkg_version, recipe, recipe_version, pkg_size) + + for val in args.extra: + src_uri = mappings[pkg][val].strip() + src_uri = re.sub(r'\s+', ' ', src_uri) + line += ' "%s"' % src_uri + + items.append(line) print('\n'.join(items)) def get_recipe_pkgs(pkgdata_dir, recipe, unpackaged):
@@ -530,6 +536,7 @@ def main(): description='Looks up the specified runtime package(s) and display information') parser_package_info.add_argument('pkg', nargs='*', help='Runtime package name to look up') parser_package_info.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)') + parser_package_info.add_argument('-e', '--extra', help='Extra variables to display, e.g., SRC_URI (can be specified multiple times)', action='append') parser_package_info.set_defaults(func=package_info) parser_find_path = subparsers.add_parser('find-path',
Signature · 1 line
2.12.0
Quoted 8 lines · Peter Kjellerstedt
After a build in our autobuilder, we use `oe-pkgdata-util package- info
-f package.manifest` to store a file with information about each installed package. This is typically used to compare builds later on. Since not every difference to a package is explained by its version, we have found it beneficial to also include theSRC_URIin the generated file.
Quoted 4 lines
This patch set adds
SRC_URIto the pkgdata that is stored for each package, and adds a new option to oe-pkgdata-util package-info, -e <var>, that can be used to display extra variables from the pkgdata.
I'm going to say no to this.
The reason is that pkgdata is not really about collecting up all build information. If you want to know how two different builds differ, you'd use the sigdata files. If I take this patch, more will follow where you find some new difference you want to track and there are other mechanisms I'd suggest (buildhistory and siginfo for starters). I don't want to turn the pkgdata files into something they're not.
Cheers,
Richard
Quoted 31 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 16 maj 2017 15:15
To: Peter Kjellerstedt <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info to display more infoQuoted 8 lines · Peter Kjellerstedt
After a build in our autobuilder, we use `oe-pkgdata-util package- info
-f package.manifest` to store a file with information about each installed package. This is typically used to compare builds later on. Since not every difference to a package is explained by its version, we have found it beneficial to also include theSRC_URIin the generated file.Quoted 4 lines
This patch set adds
SRC_URIto the pkgdata that is stored for each package, and adds a new option to oe-pkgdata-util package-info, -e <var>, that can be used to display extra variables from the pkgdata.I'm going to say no to this.
The reason is that pkgdata is not really about collecting up all build information. If you want to know how two different builds differ, you'd use the sigdata files. If I take this patch, more will follow where you find some new difference you want to track and there are other mechanisms I'd suggest (buildhistory and siginfo for starters). I don't want to turn the pkgdata files into something they're not.
Quoted 3 lines
Cheers,
Richard
As you may have expected, that was not the response I had hoped for. However, I will see if we can come to a working solution that you can accept.
Since I hope that the change to oe-pkgdata-util is acceptable to you, I will focus on the change in package.bbclass.
Regarding siginfo: I assume it has the information in there somewhere. However, I have been working with OE for five years now, and it is still basically a black hole to me. I have no idea how to do anything useful with it.
Regarding buildhistory: AFAICT there is nothing in there about neither recipe names nor SRC_URI. I guess that can be added, though, if really needed.
On the other hand, we have a tool, oe-pkgdata-util, that provides a simple interface to access the package information, and can produce a simple information file (oe-pkgdata-util package-info) with one line per package. This file is simple enough that I can give it to our maintenance team and they can look at it to see if there are any differences they need to know about. An additional benefit here is that we can run oe-pkgdata-util from within a bitbake task to generate the file together with the other artefacts we produce for a release.
That said, I can understand that you do not want to add information to the pkgdata that is not really needed to build. However, would you accept a way to add to this data, e.g., by specifying a BitBake variable such as EXTRA_PKGDATA_VARS with the names of the extra variables that should be stored in pkgdata? That way it would not affect anyone unless they actually need this extra data, and I would not have the burden of carrying a backported version of package.bbclass forever in our layers with all the extra maintenance that incurs.
//Peter
Quoted 79 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Peter Kjellerstedt
Sent: den 16 maj 2017 18:30
To: Richard Purdie <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info to display more infoQuoted 31 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 16 maj 2017 15:15
To: Peter Kjellerstedt <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package-info to display more infoQuoted 11 lines · Peter Kjellerstedt
After a build in our autobuilder, we use
oe-pkgdata-util package- info -f package.manifestto store a file with information about each installed package. This is typically used to compare builds later on. Since not every difference to a package is explained by its version, we have found it beneficial to also include theSRC_URIin the generated file.This patch set adds
SRC_URIto the pkgdata that is stored for each package, and adds a new option to oe-pkgdata-util package-info, -e <var>, that can be used to display extra variables from the pkgdata.I'm going to say no to this.
The reason is that pkgdata is not really about collecting up all build information. If you want to know how two different builds differ, you'd use the sigdata files. If I take this patch, more will follow where you find some new difference you want to track and there are other mechanisms I'd suggest (buildhistory and siginfo for starters). I don't want to turn the pkgdata files into something they're not.
Quoted 3 lines
Cheers,
Richard
As you may have expected, that was not the response I had hoped for. However, I will see if we can come to a working solution that you can accept.
Since I hope that the change to oe-pkgdata-util is acceptable to you, I will focus on the change in package.bbclass.
Regarding siginfo: I assume it has the information in there somewhere. However, I have been working with OE for five years now, and it is still basically a black hole to me. I have no idea how to do anything useful with it.
Regarding buildhistory: AFAICT there is nothing in there about neither recipe names nor
SRC_URI. I guess that can be added, though, if really needed.On the other hand, we have a tool, oe-pkgdata-util, that provides a simple interface to access the package information, and can produce a simple information file (oe-pkgdata-util package-info) with one line per package. This file is simple enough that I can give it to our maintenance team and they can look at it to see if there are any differences they need to know about. An additional benefit here is that we can run oe-pkgdata-util from within a bitbake task to generate the file together with the other artefacts we produce for a release.
That said, I can understand that you do not want to add information to the pkgdata that is not really needed to build. However, would you accept a way to add to this data, e.g., by specifying a BitBake variable such as
EXTRA_PKGDATA_VARSwith the names of the extra variables that should be stored in pkgdata? That way it would not affect anyone unless they actually need this extra data, and I would not have the burden of carrying a backported version of package.bbclass forever in our layers with all the extra maintenance that incurs.
Quoted 1 line
//Peter
It has been a week now. Any response would be appreciated.
//Peter
Quoted 40 lines · Peter Kjellerstedt
Quoted 29 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 16 maj 2017 15:15
To: Peter Kjellerstedt <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH 0/2] Allow oe-pkgdata-util package- info
to display more infoQuoted 11 lines · Peter Kjellerstedt
After a build in our autobuilder, we use `oe-pkgdata-util package-
info
-f package.manifest` to store a file with information about each installed package. This is typically used to compare builds later on. Since not every difference to a package is explained by its version, we have found it beneficial to also include theSRC_URIin
the generated file.Quoted 5 lines
This patch set adds
SRC_URIto the pkgdata that is stored for each
package, and adds a new option to oe-pkgdata-util package-info, -e <var>, that can be used to display extra variables from the pkgdata.I'm going to say no to this.
Quoted 11 lines
The reason is that pkgdata is not really about collecting up all build
information. If you want to know how two different builds differ, you'd
use the sigdata files. If I take this patch, more will follow where you
find some new difference you want to track and there are other mechanisms I'd suggest (buildhistory and siginfo for starters). I don't
want to turn the pkgdata files into something they're not.
Quoted 6 lines
As you may have expected, that was not the response I had hoped for. However, I will see if we can come to a working solution that you can accept.
Since I hope that the change to oe-pkgdata-util is acceptable to you, I will focus on the change in package.bbclass.
That change is ok in principle but I don't like the usage of SRC_URI in the help text or src_uri as a variable name in the code. I could read that as you're not even making an effort to think bout the bigger picture or generic patches with a variable name like that :(.
Quoted 4 lines
Regarding siginfo: I assume it has the information in there somewhere. However, I have been working with OE for five years now, and it is still basically a black hole to me. I have no idea how to do anything useful with it.
The siginfo files are pickled python data. You can look at the code in siggen.py (called from bitbake-dumpsig) so an example of how to view it. They contain all the information about what went into a given task.
If two siginfo files differ, they'll have different checksums. You can then see from bitbake-diffsigs exactly what those differences were. On a per variable basis. They therefore offer the ultimate in build different analysis.
Quoted 3 lines
Regarding buildhistory: AFAICT there is nothing in there about neither recipe names nor
SRC_URI. I guess that can be added, though, if really needed.
buildhistory is about output comparison so it probably doesn't make sense there. I do believe it would offer more generic mechanisms than we can afford with pkgdata though. Your best bet for generic build comparison is still the siginfo files.
Quoted 9 lines
On the other hand, we have a tool, oe-pkgdata-util, that provides a simple interface to access the package information, and can produce a simple information file (oe-pkgdata-util package-info) with one line per package. This file is simple enough that I can give it to our maintenance team and they can look at it to see if there are any differences they need to know about. An additional benefit here is that we can run oe-pkgdata-util from within a bitbake task to generate the file together with the other artefacts we produce for a release.
SRC_URI is not a piece of metadata associated with a package, its associated with a recipe. Yes, there is recipe metadata that feeds into packages but we really do need to have data in the right places. We could list every possible recipe variable in that list, for the reasons you argue. For example, perhaps S should be listed (and B) since we might want to locate the source or the built artefacts (and so on). I believe pkgdata should be limited to package variables though. I appreciate that isn't what you want to hear.
Quoted 9 lines
That said, I can understand that you do not want to add information to the pkgdata that is not really needed to build. However, would you accept a way to add to this data, e.g., by specifying a BitBake variable such as
EXTRA_PKGDATA_VARSwith the names of the extra variables that should be stored in pkgdata? That way it would not affect anyone unless they actually need this extra data, and I would not have the burden of carrying a backported version of package.bbclass forever in our layers with all the extra maintenance that incurs.
The data from pkgdata is really meant to be used by the package_write_* tasks as a mechanism to transfer data from the do_package task. oe- pkgdata-util was created to solve problems of visibility into the same *generated* data. We are planning to add a visualiser of that data too.
What I don't like is adding data that is:
a) not package specific
b) not generated
c) available elsewhere
There are only a very specific set of variables which contain appropriate data, so no, I don't think we do want to support usecases which "abuse" this data.
I'm travelling, distracted and tired but you're pushing me for a reply so there you have it, no, we're not doing this, sorry.
Cheers,
Richard