core/show-info: export extra information about the package build (branch yem/show-info-extras)
cover core/show-info: export extra information about the package build (branch yem/show-info-extras)
Hello All!
People (and scripts) who want to report on the build progress will need ato peek inside the internal details of the package build steps, so we expose this extra information in show-info.
Regards,
Yann E. MORIN.
The following changes since commit 91238c2605babd1a3d262bacdf04e14d39e59810
package/uftp: bump to version 4.10.2 (2020-04-11 10:07:02 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to 0493cfafbdc9dd03d461d8707de2358156f9ad5d
core/show-info: report image name of filesystems (2020-04-11 10:10:18 +0200)
----------------------------------------------------------------
Yann E. MORIN (8):
core/show-info: do not show install types for host packages infra/pkg-generic: don't set INSTALL_{TARGET,STAGING,IMAGES} for host core/show-info: report install types for virtual packages too core/show-info: report the package build directory core/show-info: report whether a package is overriden core/show-info: report package stamp files core/show-info: report the ordered list of build steps core/show-info: report image name of filesystems
package/pkg-generic.mk | 7 +++---- package/pkg-utils.mk | 52 +++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 52 insertions(+), 7 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 1/8 core/show-info: do not show install types for host packages
By their very nature, host packages have no target, staging, or images install steps; they have a single install step, that is always performed.
However, we currently report those three types of install, which is misleading at best, and really incorrect.
If we were to report any install type for host package, that would be a single one, and it would always be true.
So, do not report any install type for host packages, as it does not make sense to report anything that is always true.
--- package/pkg-utils.mk | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
package/pkg-utils.mk +5 -3
@@ -112,9 +112,11 @@ define _json-info-pkg-details "version": "$($(1)_DL_VERSION)", "licenses": "$($(1)_LICENSE)", "dl_dir": "$($(1)_DL_SUBDIR)", - "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET)), - "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING)), - "install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES)), + $(if $(filter target,$($(1)_TYPE)), \ + "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \ + "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \ + "install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \ + ) "downloads": [ $(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)), {
Signature · 1 line
2.20.1
Thomas Petazzoni
AppliedPATCH 1/8 core/show-info: do not show install types for host packages
Quoted 13 lines · On Sat, 11 Apr 2020 10:12:26 +0200 "Yann E. MORIN" <[email protected]>…
By their very nature, host packages have no target, staging, or images install steps; they have a single install step, that is always performed.
However, we currently report those three types of install, which is misleading at best, and really incorrect.
If we were to report any install type for host package, that would be a single one, and it would always be true.
So, do not report any install type for host packages, as it does not make sense to report anything that is always true.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 2/8 infra/pkg-generic: don't set INSTALL_{TARGET, STAGING, IMAGES} for host
In reply to Yann E. MORINBy their very nature, host packages have no target, staging, or images install steps; they have a single install step, that is always performed.
As sucj, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES} variables does not make sense for host packages.
However, people (and scripts) may get confused when they process the output of printvars, e.g.:
$ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET
HOST_LIBTOOL_INSTALL_TARGET=YESOnly set those variables for target packages. There is no corresponding variable for host packages, as they are always installed (and only once).
--- package/pkg-generic.mk | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
package/pkg-generic.mk +3 -4
@@ -706,10 +706,6 @@ $(2)_FINAL_RECURSIVE_RDEPENDENCIES = $$(sort \ ) \ $$($(2)_FINAL_RECURSIVE_RDEPENDENCIES__X)) -$(2)_INSTALL_STAGING ?= NO -$(2)_INSTALL_IMAGES ?= NO -$(2)_INSTALL_TARGET ?= YES - # define sub-target stamps $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed
@@ -770,6 +766,9 @@ $(1): $(1)-install ifeq ($$($(2)_TYPE),host) $(1)-install: $(1)-install-host else +$(2)_INSTALL_STAGING ?= NO +$(2)_INSTALL_IMAGES ?= NO +$(2)_INSTALL_TARGET ?= YES $(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images endif
Signature · 1 line
2.20.1
Thomas Petazzoni
AppliedPATCH 2/8 infra/pkg-generic: don't set INSTALL_{TARGET, STAGING, IMAGES} for host
Quoted 17 lines · On Sat, 11 Apr 2020 10:12:27 +0200 "Yann E. MORIN" <[email protected]>…
By their very nature, host packages have no target, staging, or images install steps; they have a single install step, that is always performed.
As sucj, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES} variables does not make sense for host packages.
However, people (and scripts) may get confused when they process the output of printvars, e.g.:
$ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET HOST_LIBTOOL_INSTALL_TARGET=YESOnly set those variables for target packages. There is no corresponding variable for host packages, as they are always installed (and only once).
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-generic.mk | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/8 core/show-info: report install types for virtual packages too
In reply to Yann E. MORINAlthough virtual packages do not usually install anything, they do participate in the dependency chain, and they also transition smoothly from 'not handled yet' all the way down to 'all set'. And in fact, it happens that some virtual packages do install something; for example, the virtual package 'toolchain' does tweak the musl headers during a post staging hook.
As such, people may want to know when those packages have been handled. To that goal, they will need to know what stampfile to look at (the target, staging, or images ones) to identify what step the virtual package is at.
So, report the type of install steps for virtual packages too, which means for all packages.
--- package/pkg-utils.mk | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
package/pkg-utils.mk +5 -5
@@ -100,6 +100,11 @@ define _json-info-pkg "virtual": false$(comma) $(call _json-info-pkg-details,$(1)) \ ) + $(if $(filter target,$($(1)_TYPE)), \ + "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \ + "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \ + "install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \ + ) "dependencies": [ $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES))) ],
@@ -112,11 +117,6 @@ define _json-info-pkg-details "version": "$($(1)_DL_VERSION)", "licenses": "$($(1)_LICENSE)", "dl_dir": "$($(1)_DL_SUBDIR)", - $(if $(filter target,$($(1)_TYPE)), \ - "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \ - "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \ - "install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \ - ) "downloads": [ $(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)), {
Signature · 1 line
2.20.1
PATCH 3/8 core/show-info: report install types for virtual packages too
Quoted 3 lines · On Sat, 11 Apr 2020 10:12:28 +0200 "Yann E. MORIN" <[email protected]>…
Although virtual packages do not usually install anything, they do participate in the dependency chain, and they also transition smoothly from 'not handled yet' all the way down to 'all set'.
I'm having a hard time to understand what you mean by:
they also transition smoothly from 'not handled yet' all the way down to 'all set'.
Could you clarify ?
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/8 core/show-info: report install types for virtual packages too
Thomas, All,
On 2020-04-11 10:41 +0200, Thomas Petazzoni spake thusly:
Quoted 10 lines
Quoted 3 lines · On Sat, 11 Apr 2020 10:12:28 +0200 "Yann E. MORIN" <[email protected]>…
Although virtual packages do not usually install anything, they do participate in the dependency chain, and they also transition smoothly from 'not handled yet' all the way down to 'all set'.
I'm having a hard time to understand what you mean by:
they also transition smoothly from 'not handled yet' all the way down to 'all set'.
Could you clarify ?
I mean that virtual package, like all other packages, have a sequence of a few steps that transition the virtual package from "not handled yet" to "installed and dependencies can start", i.e. virtual packages are:
1. configured
2. built
3. installed in staging (optional)
4. installed in target (optional)
5. isntalled in images (optional)
Virtual packages are just packages in this respect. It's just that their deoendencies are treated specially.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Thomas Petazzoni
AppliedPATCH 3/8 core/show-info: report install types for virtual packages too
In reply to Yann E. MORINQuoted 15 lines · On Sat, 11 Apr 2020 10:12:28 +0200 "Yann E. MORIN" <[email protected]>…
Although virtual packages do not usually install anything, they do participate in the dependency chain, and they also transition smoothly from 'not handled yet' all the way down to 'all set'. And in fact, it happens that some virtual packages do install something; for example, the virtual package 'toolchain' does tweak the musl headers during a post staging hook.
As such, people may want to know when those packages have been handled. To that goal, they will need to know what stampfile to look at (the target, staging, or images ones) to identify what step the virtual package is at.
So, report the type of install steps for virtual packages too, which means for all packages.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
I've applied, but with a significantly reworked commit log to clarify the intention. Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 4/8 core/show-info: report the package build directory
In reply to Yann E. MORINSome people want to be able to inspect the current state of the build, and report information about it, like:
- report the number of packages already built out of the total,
- list the packages being actually built (e.g. for TLPB)
- etc...
However, the location where a package is built is inherently an internal detail, so expose that to the user in the output of show-info. We only expose the location relative to the base directory (basically, either output/ or the user-suppiled $(O)), so that show-info does not contain local information (the output of show-info can be shared).
Interested parties will be able to poke in there to identify the stamp files and deduce the package's state.
--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-utils.mk +1 -0
@@ -100,6 +100,7 @@ define _json-info-pkg "virtual": false$(comma) $(call _json-info-pkg-details,$(1)) \ ) + "build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))", $(if $(filter target,$($(1)_TYPE)), \ "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \ "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
Signature · 1 line
2.20.1
Thomas Petazzoni
AppliedPATCH 4/8 core/show-info: report the package build directory
Quoted 15 lines · On Sat, 11 Apr 2020 10:12:29 +0200 "Yann E. MORIN" <[email protected]>…
Some people want to be able to inspect the current state of the build, and report information about it, like:
- report the number of packages already built out of the total,
- list the packages being actually built (e.g. for TLPB)
- etc...However, the location where a package is built is inherently an internal detail, so expose that to the user in the output of show-info. We only expose the location relative to the base directory (basically, either output/ or the user-suppiled $(O)), so that show-info does not contain local information (the output of show-info can be shared).
Interested parties will be able to poke in there to identify the stamp files and deduce the package's state.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/8 core/show-info: report whether a package is overriden
In reply to Yann E. MORINWhen a package is overridden, the download, extract, and patch stampfiles are replaced by a single rsync stampfile.
People (and scripts) who want to report on the current status of the build, will want to know what stamp file to look at.
Expose in show-info whether the package is overridden or not.
We do not expose the override location, because this is a purely-local information, and show info should never contain such local information (the output of show-info can be shared). Besides, that information is already known by the user, as they passed it to Buildroot via local.mk to begin with.
--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-utils.mk +1 -0
@@ -100,6 +100,7 @@ define _json-info-pkg "virtual": false$(comma) $(call _json-info-pkg-details,$(1)) \ ) + "overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false), "build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))", $(if $(filter target,$($(1)_TYPE)), \ "install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
Signature · 1 line
2.20.1
PATCH 5/8 core/show-info: report whether a package is overriden
Quoted 28 lines · On Sat, 11 Apr 2020 10:12:30 +0200 "Yann E. MORIN" <[email protected]>…
When a package is overridden, the download, extract, and patch stampfiles are replaced by a single rsync stampfile.
People (and scripts) who want to report on the current status of the build, will want to know what stamp file to look at.
Expose in show-info whether the package is overridden or not.
We do not expose the override location, because this is a purely-local information, and show info should never contain such local information (the output of show-info can be shared). Besides, that information is already known by the user, as they passed it to Buildroot via local.mk to begin with.
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)package/pkg-utils.mk +0
-0
package/pkg-utils.mk define _json-info-pkg · line 103
Quoted 4 lines
package/pkg-utils.mk +1
-0@@ -100,6 +100,7 @@ define _json-info-pkg "virtual": false$(comma) $(call _json-info-pkg-details,$(1)) \ ) + "overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
I don't have a good suggestion, but I'm not sure "overriden" is the most appropriate term. Indeed, the download/extract/patch steps are also replaced by a rsync step for packages that use _SITE_METHOD = local, and such packages are not "overriden".
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/8 core/show-info: report whether a package is overriden
Thomas, All,
On 2020-04-11 10:36 +0200, Thomas Petazzoni spake thusly:
Quoted 26 lines
Quoted 23 lines · On Sat, 11 Apr 2020 10:12:30 +0200 "Yann E. MORIN" <[email protected]>…
When a package is overridden, the download, extract, and patch stampfiles are replaced by a single rsync stampfile.
People (and scripts) who want to report on the current status of the build, will want to know what stamp file to look at.
Expose in show-info whether the package is overridden or not.
We do not expose the override location, because this is a purely-local information, and show info should never contain such local information (the output of show-info can be shared). Besides, that information is already known by the user, as they passed it to Buildroot via local.mk to begin with.
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 1 + 1 file changed, 1 insertion(+)
Quoted 14 lines
package/pkg-utils.mk define _json-info-pkg · line 103
Quoted 9 lines
package/pkg-utils.mk +1
-0@@ -100,6 +100,7 @@ define _json-info-pkg "virtual": false$(comma) $(call _json-info-pkg-details,$(1)) \ ) + "overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),I don't have a good suggestion, but I'm not sure "overriden" is the most appropriate term. Indeed, the download/extract/patch steps are also replaced by a rsync step for packages that use
_SITE_METHOD= local, and such packages are not "overriden".
It is very unfortunate that we conflate the two conditions.
We can't even reconstruct the override by looking at whether _SITE_METHOD == local, because even local packages may be overriden...
So, is it worth that I try and untangle the tow notions? Given the feedback on the rest of this eries, I don;t want to invest too much time if there is no chance of it getting in...
Regards,
Yann E. MORIN.
Quoted 5 lines
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 5/8 core/show-info: report whether a package is overriden
Quoted 6 lines · On Sat, 11 Apr 2020 11:44:36 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 4 lines
I don't have a good suggestion, but I'm not sure "overriden" is the most appropriate term. Indeed, the download/extract/patch steps are also replaced by a rsync step for packages that use
_SITE_METHOD= local, and such packages are not "overriden".It is very unfortunate that we conflate the two conditions.
Well, from an internal implementation point of view, SITE_METHOD = local and OVERRIDE_SRCDIR are just exactly the same thing.
Perhaps you could use:
"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
as this is really annotates the fact that the package source code is rsynced. You could even make it clearer with "source-rsynced" or something like that.
Quoted 2 lines
We can't even reconstruct the override by looking at whether
_SITE_METHOD== local, because even local packages may be overriden...
Indeed.
Quoted 3 lines
So, is it worth that I try and untangle the tow notions? Given the feedback on the rest of this eries, I don;t want to invest too much time if there is no chance of it getting in...
I think this particular patch is OK, even though admittedly the external tool could just watch for the correct stamp files to show up:
if .stamp_downloaded shows up, we're downloading it normally, if .stamp_rsynced shows up, we have a local or overridden package.
In the design of the tool, it would be good to make sure that top-level parallel build is taken into account.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/8 core/show-info: report whether a package is overriden
Thomas, All,
On 2020-04-11 14:42 +0200, Thomas Petazzoni spake thusly:
Quoted 8 lines
Quoted 5 lines · On Sat, 11 Apr 2020 11:44:36 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 4 lines
I don't have a good suggestion, but I'm not sure "overriden" is the most appropriate term. Indeed, the download/extract/patch steps are also replaced by a rsync step for packages that use
_SITE_METHOD= local, and such packages are not "overriden".
Quoted 4 lines
Quoted 1 line
It is very unfortunate that we conflate the two conditions.
Well, from an internal implementation point of view,
SITE_METHOD= local andOVERRIDE_SRCDIRare just exactly the same thing.
But that is an implementation detail, indeed.
In fact, we should have had local support without override support. And then we should have grafted override suport on top of the local one, while the code as it is makes local re-use the override code...
I'll try to hack a cleanup in that area...
Quoted 3 lines
Perhaps you could use:
"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
Fact is, that still does not reflect what I want to expose. An overriden package is de-facto not reproducible, while a local package may.
But I don't care enough to argue further. "resynced" gets the same value as "overriden", and since we can;t get the semantic of "overriden", it is as good as it is.
Quoted 13 lines
as this is really annotates the fact that the package source code is rsynced. You could even make it clearer with "source-rsynced" or something like that.
Quoted 2 lines
We can't even reconstruct the override by looking at whether
_SITE_METHOD== local, because even local packages may be overriden...Indeed.
Quoted 3 lines
So, is it worth that I try and untangle the tow notions? Given the feedback on the rest of this eries, I don;t want to invest too much time if there is no chance of it getting in...
Quoted 7 lines
I think this particular patch is OK, even though admittedly the external tool could just watch for the correct stamp files to show up:
if .stamp_downloaded shows up, we're downloading it normally, if .stamp_rsynced shows up, we have a local or overridden package.In the design of the tool, it would be good to make sure that top-level parallel build is taken into account.
Which is exactly what I am trying to achieve here: provide all the info about the internal details, so that the tool does not have to guess.
But personally I don't care about such a tool; I'm just trying to make life easier for those (like eeppeliteloop in Cc) who want such tools.
Regards,
Yann E. MORIN.
Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 5/8 core/show-info: report whether a package is overriden
Hello,
Quoted 8 lines · On Sat, 11 Apr 2020 15:22:58 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 2 lines
Well, from an internal implementation point of view,
SITE_METHOD= local andOVERRIDE_SRCDIRare just exactly the same thing.But that is an implementation detail, indeed.
In fact, we should have had local support without override support. And then we should have grafted override suport on top of the local one, while the code as it is makes local re-use the override code...
I am not sure what that would bring, and how that would make things better/clearer. Ultimately, what happens for both overridden packages and local packages is that they are rsynced.
Quoted 6 lines
Quoted 3 lines
Perhaps you could use:
"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),Fact is, that still does not reflect what I want to expose. An overriden package is de-facto not reproducible, while a local package may.
I don't understand why a local package is more reproducible than an overridden package. In both cases, Buildroot cannot guarantee the reproducibility.
Quoted 3 lines
But I don't care enough to argue further. "resynced" gets the same value as "overriden", and since we can;t get the semantic of "overriden", it is as good as it is.
I don't follow your reasoning here. I am fine with having the boolean this patch proposes to add, I was just suggesting another term than "overridden", as that was not really covering the case of local packages.
Quoted 10 lines
Quoted 7 lines
I think this particular patch is OK, even though admittedly the external tool could just watch for the correct stamp files to show up:
if .stamp_downloaded shows up, we're downloading it normally, if .stamp_rsynced shows up, we have a local or overridden package.In the design of the tool, it would be good to make sure that top-level parallel build is taken into account.
Which is exactly what I am trying to achieve here: provide all the info about the internal details, so that the tool does not have to guess.
And I'm not sure it's a worthwhile goal. I think the tool should just have the knowledge of what possible stamp files exist, and what are the possible sequences of stamp file creation.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/8 core/show-info: report whether a package is overriden
Thomas, All,
On 2020-04-11 16:14 +0200, Thomas Petazzoni spake thusly:
Quoted 11 lines
Quoted 6 lines · On Sat, 11 Apr 2020 15:22:58 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 2 lines
Well, from an internal implementation point of view,
SITE_METHOD= local andOVERRIDE_SRCDIRare just exactly the same thing.But that is an implementation detail, indeed.
In fact, we should have had local support without override support. And then we should have grafted override suport on top of the local one, while the code as it is makes local re-use the override code...I am not sure what that would bring, and how that would make things better/clearer. Ultimately, what happens for both overridden packages and local packages is that they are rsynced.
I'd still like to be able to tell the two apart, but that is eventually orthogonal to the current topic.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Perhaps you could use:
"rsynced": $(if $($(1)_OVERRIDE_SRCDIR),true,false),Fact is, that still does not reflect what I want to expose. An overriden package is de-facto not reproducible, while a local package may.
I don't understand why a local package is more reproducible than an overridden package. In both cases, Buildroot cannot guarantee the reproducibility.
If the package is in the same sourec tree as Buildroot (or a git-submodule or something like that), then it is reproducible (as it is in VCS and de-facto referenced by a specific commit that brings both Buildroot and the package sources. That's why a Isaid that a local packge "may" be reproducible.
Quoted 7 lines
Quoted 3 lines
But I don't care enough to argue further. "resynced" gets the same value as "overriden", and since we can;t get the semantic of "overriden", it is as good as it is.
I don't follow your reasoning here. I am fine with having the boolean this patch proposes to add, I was just suggesting another term than "overridden", as that was not really covering the case of local packages.
Yes, you are right here; I got side-tracked. For the stampfiles, we don't need to discrimninate between local or overriden: we just need to know it is rsynced (or downlaoded-extracted-patched).
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines
I think this particular patch is OK, even though admittedly the external tool could just watch for the correct stamp files to show up:
if .stamp_downloaded shows up, we're downloading it normally, if .stamp_rsynced shows up, we have a local or overridden package. In the design of the tool, it would be good to make sure that top-level parallel build is taken into account.Which is exactly what I am trying to achieve here: provide all the info about the internal details, so that the tool does not have to guess.
And I'm not sure it's a worthwhile goal. I think the tool should just have the knowledge of what possible stamp files exist, and what are the possible sequences of stamp file creation.
OK. I'll drop from my series the parts related to that...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 6/8 core/show-info: report package stamp files
In reply to Yann E. MORINPeople (and scripts) who want to report on the build progress, will want to look at the various stamp files.
However, the names of those stamp files are purely an implementation detail (even if they are very unlikely to ever change).
So, expose the names of those stampfiles in show-info, so that interested parties do not have to hard-code them.
--- package/pkg-utils.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
package/pkg-utils.mk +23 -0
@@ -107,6 +107,29 @@ define _json-info-pkg "install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \ "install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \ ) + "stamp_files": { + $(if $($(1)_OVERRIDE_SRCDIR), \ + "rsync": "$(notdir $($(1)_TARGET_RSYNC))"$(comma) \ + , \ + "download": "$(notdir $($(1)_TARGET_SOURCE))"$(comma) \ + "extract": "$(notdir $($(1)_TARGET_EXTRACT))"$(comma) \ + "patch": "$(notdir $($(1)_TARGET_PATCH))"$(comma) \ + ) + "configure": "$(notdir $($(1)_TARGET_CONFIGURE))", + "build": "$(notdir $($(1)_TARGET_BUILD))", + $(if $(filter YES,$($(1)_INSTALL_STAGING)),\ + "install_staging": "$(notdir $($(1)_TARGET_INSTALL_STAGING))"$(comma) \ + ) + $(if $(filter YES,$($(1)_INSTALL_TARGET)),\ + "install_target": "$(notdir $($(1)_TARGET_INSTALL_TARGET))"$(comma) \ + ) + $(if $(filter YES,$($(1)_INSTALL_IMAGES)),\ + "install_images": "$(notdir $($(1)_TARGET_INSTALL_IMAGES))"$(comma) \ + ) + $(if $(filter host,$($(1)_TYPE)), \ + "install": "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \ + ) + }, "dependencies": [ $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES))) ],
Signature · 1 line
2.20.1
PATCH 6/8 core/show-info: report package stamp files
Quoted 12 lines · On Sat, 11 Apr 2020 10:12:31 +0200 "Yann E. MORIN" <[email protected]>…
package/pkg-utils.mk +12
-0+ "stamp_files": { + $(if $($(1)_OVERRIDE_SRCDIR), \ + "rsync": "$(notdir $($(1)_TARGET_RSYNC))"$(comma) \ + , \ + "download": "$(notdir $($(1)_TARGET_SOURCE))"$(comma) \ + "extract": "$(notdir $($(1)_TARGET_EXTRACT))"$(comma) \ + "patch": "$(notdir $($(1)_TARGET_PATCH))"$(comma) \ + ) + "configure": "$(notdir $($(1)_TARGET_CONFIGURE))", + "build": "$(notdir $($(1)_TARGET_BUILD))", + $(if $(filter YES,$($(1)_INSTALL_STAGING)),\ + "install_staging": "$(notdir $($(1)_TARGET_INSTALL_STAGING))"$(comma) \
package/pkg-utils.mk define _json-info-pkg · line 131
Quoted 10 lines
package/pkg-utils.mk +10
-0+ ) + $(if $(filter YES,$($(1)_INSTALL_TARGET)),\ + "install_target": "$(notdir $($(1)_TARGET_INSTALL_TARGET))"$(comma) \ + ) + $(if $(filter YES,$($(1)_INSTALL_IMAGES)),\ + "install_images": "$(notdir $($(1)_TARGET_INSTALL_IMAGES))"$(comma) \ + ) + $(if $(filter host,$($(1)_TYPE)), \ + "install": "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \ + )
Isn't this going to add for each package exactly the same information, over and over again ? I.e, just the names of stamp files, again, again, and again. This seems like a big waste of space, making the JSON output very noisy. I think the tool showing the progress should simply know what the stamp files are.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
In reply to Yann E. MORINPeople (and scritps) who want to report on the build progress, now know where and what to look for (build directory and stampfiles), but they also need to know the ordering of those stampfiles.
The stampfiles item is a dictionary; dictionaries are guaranteed to not be ordered, so even if we output the stampfiles in order, we can expect users of that dictionary to get them back in order.
Expose the build steps in a list, which is guaranteed to be ordered.
--- package/pkg-utils.mk | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
package/pkg-utils.mk +15 -0
@@ -130,6 +130,21 @@ define _json-info-pkg "install": "$(notdir $($(1)_TARGET_INSTALL_HOST))"$(comma) \ ) }, + "build_steps": [ + $(if $($(1)_OVERRIDE_SRCDIR), \ + "rsync"$(comma) \ + , \ + "download"$(comma) \ + "extract"$(comma) \ + "patch"$(comma) \ + ) + "build", + "configure", + $(if $(filter YES,$($(1)_INSTALL_STAGING)),"install_staging"$(comma)) + $(if $(filter YES,$($(1)_INSTALL_TARGET)),"install_target"$(comma)) + $(if $(filter YES,$($(1)_INSTALL_IMAGES)),"install_images"$(comma)) + $(if $(filter host,$($(1)_TYPE)),"install"$(comma)) + ], "dependencies": [ $(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES))) ],
Signature · 1 line
2.20.1
PATCH 7/8 core/show-info: report the ordered list of build steps
Quoted 10 lines · On Sat, 11 Apr 2020 10:12:32 +0200 "Yann E. MORIN" <[email protected]>…
People (and scritps) who want to report on the build progress, now know where and what to look for (build directory and stampfiles), but they also need to know the ordering of those stampfiles.
The stampfiles item is a dictionary; dictionaries are guaranteed to not be ordered, so even if we output the stampfiles in order, we can expect users of that dictionary to get them back in order.
Expose the build steps in a list, which is guaranteed to be ordered.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of additional information in the JSON, almost duplicated for every package, and there is really nothing bad in the "progress showing" tool to have knowledge about these steps. It's not like they are changing all the time.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
Quoted 23 lines · Thomas Petazzoni
Quoted 10 lines · On Sat, 11 Apr 2020 10:12:32 +0200 "Yann E. MORIN" <[email protected]>…
People (and scritps) who want to report on the build progress, now know where and what to look for (build directory and stampfiles), but they also need to know the ordering of those stampfiles.
The stampfiles item is a dictionary; dictionaries are guaranteed to not be ordered, so even if we output the stampfiles in order, we can expect users of that dictionary to get them back in order.
Expose the build steps in a list, which is guaranteed to be ordered.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
Quoted 5 lines
Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of additional information in the JSON, almost duplicated for every package, and there is really nothing bad in the "progress showing" tool to have knowledge about these steps. It's not like they are changing all the time.
Replying for patch 6/8 and this one.
Is the show-info target considered a public API? If so, without a version, it means it can never break. So let me suggest another layout which breaks show-info (name it show-info-2 if you will).
Output example (using YAML only for clarity here):
version: 2
build_steps:
target:
- name: download
stamp_file: .stamp_downloaded
- name: extract
stamp_file: .stamp_extracted
- name: patch
stamp_file: .stamp_patched
- name: build
stamp_file: .stamp_configured
- name: configure
stamp_file: .stamp_built
- name: install_staging
stamp_file: .stamp_staging_installed
- name: install_target
stamp_file: .stamp_target_installed
- name: install_image
stamp_file: .stamp_image_installed
...
host:
- ...
rootfs:
- ...
packages:
target:
libglib2:
virtual: false
version: 2.64.1
licenses: LGPL-2.1+
dl_dir: libglib2
downloads:
- source: glib-2.64.1.tar.xz
uris:
- http+http://ftp.gnome.org/pub/gnome/sources/glib/2.64
- http|urlencode+http://sources.buildroot.net/libglib2
- http|urlencode+http://sources.buildroot.net
overriden: false
build_dir: build/libglib2-2.64.1/
build_steps:
- download
- extract
- patch
- build
- configure
- install_staging
- install_target
dependencies:
- host-libglib2
- host-meson
- host-pkgconf
- host-skeleton
- host-tar
- libffi
- pcre
- skeleton
- toolchain
- zlib
reverse_dependencies:
- cairo
...
host:
...
rootfs:
...
So, changes:
* A root version entry indicates the API version.
* The root build_steps object indicates all the _possible_ (sorted) build steps for each package type.
Each build step in the array has its name and its corresponding stamp file to indicate when it's completed.
Quoted 2 lines · As Yann
However, the names of those stamp files are purely an implementation detail (even if they are very unlikely to ever change).
* The root packages object contains the package information for each package type.
Not strictly related to this patch or the previous one, but instead of repeating the type in each object, you can just group packages by type.
* For each package object, the build_steps array indicate what are the build steps for this package.
Each item is the name of a build step in the corresponding root build_steps object's array.
* For a target package, the install_staging, install_target, and install_images entries are removed.
They are not needed anymore because build_steps offers the same information.
I understand some build steps could be implicit and not listed here, but with my suggestion, you already have what's needed for any build monitoring tool to watch a specific package build process if:
* Some of the implicit build steps become optional in the future for any reason that you don't know now.
* You add new types of build steps: simply describe them in the build_steps root arrays.
Both scenarios won't require an API version bump.
If you're not willing to break show-info and don't want show-info-2, then you can:
* Keep show-info as is:
* Augment a package information with build_steps (like above).
* Keep the install_* entries which become redundant.
* Have another make target, say show-build-steps, which only prints the root build_steps object above.
What do you think?
Phil
Quoted 6 lines
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
Quoted 3 lines · On Sat, 11 Apr 2020 09:41:47 -0400 Philippe Proulx <[email protected]>…
Replying for patch 6/8 and this one.
Is the
show-infotarget considered a public API?
Somewhat yes: the goal of show-info is that people can craft their own tooling on top of it. Breaking the show-info output would break such tools.
Quoted 5 lines
If so, without a
version, it means it can never break. So let me suggest another layout which breaksshow-info(name itshow-info-2if you will).Output example (using YAML only for clarity here):
[...]
This could certainly work, and reduces a bit the duplication of information.
However, I still don't understand why your tool is not capable of having this knowledge about the stamp files. This is something that rarely changes. I think we haven't changed the stamp files since... 5 years? More? If your tool is in the Buildroot tree itself, it can simply be updated if there is ever a change in the sequencing of steps.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
Quoted 20 lines · Thomas Petazzoni
Quoted 3 lines · On Sat, 11 Apr 2020 09:41:47 -0400 Philippe Proulx <[email protected]>…
Replying for patch 6/8 and this one.
Is the
show-infotarget considered a public API?Somewhat yes: the goal of show-info is that people can craft their own tooling on top of it. Breaking the show-info output would break such tools.
Quoted 5 lines
If so, without a
version, it means it can never break. So let me suggest another layout which breaksshow-info(name itshow-info-2if you will).Output example (using YAML only for clarity here):
[...]
Quoted 8 lines
This could certainly work, and reduces a bit the duplication of information.
However, I still don't understand why your tool is not capable of having this knowledge about the stamp files. This is something that rarely changes. I think we haven't changed the stamp files since... 5 years? More? If your tool is in the Buildroot tree itself, it can simply be updated if there is ever a change in the sequencing of steps.
Yes of course.
I was only trying to improve Yann's solution considering your comments about data redundancy.
If you tell me that things such build step order and stamp file names do not change for many years and as such can be considered as interfaces, I'm fine with this.
It's always a balance between adding the same logic to all the viewers and generating less data.
My tool could be part of the Buildroot project, but I doubt you'll want something that depends on PyQt5 in there. However, I could certainly contribute a Python module which offers utilities over the show-info output so as to have a versioned, object-oriented API, and then import it from an external tool. This way, even if show-info changes, the Python module can continue to offer the same (non-breaking) interface.
I still think we somewhat need the build_dir entry though.
Phil
Quoted 8 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
Hello,
Quoted 10 lines · On Sat, 11 Apr 2020 11:06:05 -0400 Philippe Proulx <[email protected]>…
Quoted 5 lines
However, I still don't understand why your tool is not capable of having this knowledge about the stamp files. This is something that rarely changes. I think we haven't changed the stamp files since... 5 years? More? If your tool is in the Buildroot tree itself, it can simply be updated if there is ever a change in the sequencing of steps.
Yes of course.
I was only trying to improve Yann's solution considering your comments about data redundancy.
Sure, and your proposal makes sense if we believe we need to have this build sequencing information exposed in the JSON output.
Quoted 3 lines
If you tell me that things such build step order and stamp file names do not change for many years and as such can be considered as interfaces, I'm fine with this.
I don't think we can really consider these as stable interfaces, but it is sufficiently stable to rely on it.
Quoted 5 lines
It's always a balance between adding the same logic to all the viewers and generating less data.
My tool could be part of the Buildroot project, but I doubt you'll want something that depends on PyQt5 in there.
I don't see why we couldn't have it in the tree. It's not a mandatory tool, so it's fine to have something that depends on PyQt5.
Quoted 7 lines
However, I could certainly contribute a Python module which offers utilities over the
show-infooutput so as to have a versioned, object-oriented API, and then import it from an external tool. This way, even ifshow-infochanges, the Python module can continue to offer the same (non-breaking) interface.I still think we somewhat need the
build_direntry though.
Yes, for the build_dir entry, I definitely agree, it is useful and needed.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 7/8 core/show-info: report the ordered list of build steps
In reply to Philippe ProulxPhilippe, All,
On 2020-04-11 11:06 -0400, Philippe Proulx spake thusly:
Quoted 2 lines
On Sat, Apr 11, 2020 at 10:19 AM Thomas Petazzoni
<[email protected]> wrote:
[--SNIP--]
Quoted 3 lines
If you tell me that things such build step order and stamp file names do not change for many years and as such can be considered as interfaces, I'm fine with this.
They are not an interface, definitely not (IMHO). But they are stable enough that you can build something on them and not expect a breakage that often.
But an interface? No, we can't commit to that.
Quoted 2 lines
It's always a balance between adding the same logic to all the viewers and generating less data.
Since the output of show-info is for machine consumption, and is not geared at human consuption without first being processed, I don;t see a problem with generating "more" data.
Quoted 2 lines
My tool could be part of the Buildroot project, but I doubt you'll want something that depends on PyQt5 in there.
Not a problem, as it is not a mandatory tool. We already have utils/brmake that uses unbuffer, which is not available on most distros by default.
Quoted 5 lines
However, I could certainly
contribute a Python module which offers utilities over theshow-infooutput so as to have a versioned, object-oriented API, and then import it from an external tool. This way, even ifshow-infochanges, the Python module can continue to offer the same (non-breaking) interface.
You can't guarantee that either: some keys may disapear, so such a module could not expose them any more either, so not a stable API.
All you could provide as a stable API, is a way to rerieve basic information, like the list of packages and such. Or a way to query keys, which would not provide much more than what 'import json' would.
Regards,
Yann E. MORIN.
Quoted 12 lines
I still think we somewhat need the
build_direntry though.Phil
Quoted 8 lines
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 7/8 core/show-info: report the ordered list of build steps
In reply to Thomas PetazzoniThomas, All,
On 2020-04-11 16:19 +0200, Thomas Petazzoni spake thusly:
Quoted 7 lines
Quoted 2 lines · On Sat, 11 Apr 2020 09:41:47 -0400 Philippe Proulx <[email protected]>…
Replying for patch 6/8 and this one.
Is theshow-infotarget considered a public API?Somewhat yes: the goal of show-info is that people can craft their own tooling on top of it. Breaking the show-info output would break such tools.
Which in some cases wil not be avoidable, see my previous reply.
Quoted 7 lines
Quoted 5 lines
If so, without a
version, it means it can never break. So let me suggest another layout which breaksshow-info(name itshow-info-2if you will).Output example (using YAML only for clarity here):
This could certainly work, and reduces a bit the duplication of information.
The show-info is not for human consumption, I don't see a problem with its content growing.
Also, as I explained in my previous reply to Philippe, having the information where it is needed (i.e at the package level) allows to process show-info in a stream, without having to memorise anything.
OTOH, having metadata description means the whole of show-info as to be loaded in memory before any rocessing can outcur, since the content of json is not guaranteeed to be ordered.
Yes, we could try and get our bare show-info output would be ordered, but as soon as it gets processed (e.g. with json_pp or whatever), the ordering would get lost.
Quoted 5 lines
However, I still don't understand why your tool is not capable of having this knowledge about the stamp files. This is something that rarely changes. I think we haven't changed the stamp files since... 5 years? More? If your tool is in the Buildroot tree itself, it can simply be updated if there is ever a change in the sequencing of steps.
That's true. But if you want to consider show-info to be a public API and thus be stable,m then you would have to consider the stampfiles to also be public API and thus stable, which is definitely what we want.
But that's also true that it changes so rarely that it would be acceptable to break the tools.
And if we have such tools in-tree, we can fix them as we change the internal details.
Regards,
Yann E. MORIN.
Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 7/8 core/show-info: report the ordered list of build steps
In reply to Philippe ProulxPhilippe, All,
On 2020-04-11 09:41 -0400, Philippe Proulx spake thusly:
Quoted 29 lines
Quoted 21 lines · Thomas Petazzoni
Quoted 10 lines · On Sat, 11 Apr 2020 10:12:32 +0200 "Yann E. MORIN" <[email protected]>…
People (and scritps) who want to report on the build progress, now know where and what to look for (build directory and stampfiles), but they also need to know the ordering of those stampfiles.
The stampfiles item is a dictionary; dictionaries are guaranteed to not be ordered, so even if we output the stampfiles in order, we can expect users of that dictionary to get them back in order.
Expose the build steps in a list, which is guaranteed to be ordered.
Quoted 8 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 15 +++++++++++++++ 1 file changed, 15 insertions(+)Quoted 5 lines
Meh, just like PATCH 6/8, I'm not sure. This is going to be a lot of additional information in the JSON, almost duplicated for every package, and there is really nothing bad in the "progress showing" tool to have knowledge about these steps. It's not like they are changing all the time.
Quoted 3 lines
Replying for patch 6/8 and this one.
Is the
show-infotarget considered a public API?
Somewhat, yes. We can't guaraenntee its stability, though. If there are keys that no longer make sense, then we would no longer be able to expose them. Ditto, if a key semantics were to change, we would have to reflect that in show-info.
For the former case, whem/if we eventually merge target/ and staging/ into one (or rather, we derive target/ from staging/) then there would be no point in reportign the two, as they would both be always true, so we would only need to keep "install_target" and "install_images" (obviously we could have a transition period during which we still report "isntall_staging", but it would be slated for removal long term).
For the latter case, if we eventually have a way to express the licenses in a format more complex that jsut the list of licenses, then the "licenses" key would have to evolve from a simple string to a more complex object.
(Note: the above is just two hypotetical examples, the first being still probable medium-term.)
Quoted 2 lines
If so, without a
version, it means it can never break.
I disagree: we can't guarantee its stability. We can make it as stable as possible, but we can't leave it in stone. Probably what's going to happen is that old keys will stay for as long as they still exist, and only new stuff will be added. But when something no longer exist, we won't be able to expose it anymore...
So, I am totally fine with breaking the output of show-info when we can't do otherwise.
Quoted 16 lines
So let me suggest another layout
which breaksshow-info(name itshow-info-2if you will).Output example (using YAML only for clarity here):
version: 2
build_steps:
target:
- name: download
stamp_file: .stamp_downloaded
- name: extract
stamp_file: .stamp_extracted
- name: patch
stamp_file: .stamp_patched
- name: build
stamp_file: .stamp_configured
Quoted 8 lines
- name: configure
stamp_file: .stamp_built
- name:install_staging
stamp_file: .stamp_staging_installed
- name:install_target
stamp_file: .stamp_target_installed
- name:install_image
stamp_file: .stamp_image_installed
What you're doing here is indirection; I don't like indirection.
By having the information directly in the nodes, it is readily available; the data can be parsed as a stream, with little to no memorisation.
With indirection, it means you have to memorise some data so it can be reused later. With your example, it is very limited, indeed, but it opens a can of worm: except for lists, json is not guaranteed to be ordered, so there is no reason that we can expect the "build_steps" definitions to be read before the rest, so we'd have to buffer the entirety of show-info before we can even start doing anything with it.
[--SNIP--]
Quoted 19 lines
* The root
packagesobject contains the package information for each package type.
Not strictly related to this patch or the previous one, but instead of repeating the type in each object, you can just group packages by type.* For each package object, the
build_stepsarray indicate what are the build steps for this package.
Each item is the name of a build step in the corresponding rootbuild_stepsobject's array.* For a target package, the
install_staging,install_target, andinstall_imagesentries are removed.
They are not needed anymore becausebuild_stepsoffers the same information.
I understand some build steps could be implicit and not listed here, but with my suggestion, you already have what's needed for any build monitoring tool to watch a specific package build process if:
Quoted 7 lines
* Some of the implicit build steps become optional in the future for any reason that you don't know now.
* You add new types of build steps: simply describe them in the
build_stepsroot arrays.Both scenarios won't require an API version bump.
As I understand your proposal, your tool would not have to hard-code the build steps, because they would defined globally, and each package would reference (in an ordered list0 the steps that apply to it.
Well, no need to define the steps globally, since you already have the information at the package level: just construct the build sequence for that package from the build steps listed in that package.
Quoted 6 lines
If you're not willing to break
show-infoand don't wantshow-info-2, then you can:* Keep
show-infoas is:
* Augment a package information withbuild_steps(like above).
* Keep theinstall_*entries which become redundant.
That's exactly what I did: I just augment show-info with new informnation. leaving the legacy install{target,staging,images} keys in place for those that used to use them.
Quoted 3 lines
* Have another make target, say
show-build-steps, which only prints the rootbuild_stepsobject above.
What do you think?
I don't like indirection. ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 8/8 core/show-info: report image name of filesystems
In reply to Yann E. MORINThe rootfses, unlike packages, do not have stampfiles to represent whether they are built or not. Indeed, we always rebuild the rootfs, and there is a single step to do.
Hpwever, people (and scripts) who want to report on the build progress, will want to know whether each rootfs has been built already or not.
Expose in show-info the name of the file that wil contain the rootfs image.
--- package/pkg-utils.mk | 4 ++++ 1 file changed, 4 insertions(+)
package/pkg-utils.mk +4 -0
@@ -174,6 +174,10 @@ define _json-info-pkg-details endef define _json-info-fs + "image_name": $(if $($(1)_FINAL_IMAGE_NAME), \ + "$($(1)_FINAL_IMAGE_NAME)", \ + null \ + ), "dependencies": [ $(call make-comma-list,$(sort $($(1)_DEPENDENCIES))) ]
Signature · 1 line
2.20.1
PATCH 8/8 core/show-info: report image name of filesystems
Quoted 24 lines · On Sat, 11 Apr 2020 10:12:33 +0200 "Yann E. MORIN" <[email protected]>…
The rootfses, unlike packages, do not have stampfiles to represent whether they are built or not. Indeed, we always rebuild the rootfs, and there is a single step to do.
Hpwever, people (and scripts) who want to report on the build progress, will want to know whether each rootfs has been built already or not.
Expose in show-info the name of the file that wil contain the rootfs image.
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-utils.mk +0
-0
package/pkg-utils.mk define _json-info-pkg-details · line 180
Quoted 7 lines
package/pkg-utils.mk +4
-0@@ -174,6 +174,10 @@ define _json-info-pkg-details endef define _json-info-fs + "image_name": $(if $($(1)_FINAL_IMAGE_NAME), \ + "$($(1)_FINAL_IMAGE_NAME)", \ + null \ + ),
Is the condition needed because of the initramfs case?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 8/8 core/show-info: report image name of filesystems
Thomas, All,
On 2020-04-25 15:12 +0200, Thomas Petazzoni spake thusly:
Quoted 8 lines
Quoted 5 lines · On Sat, 11 Apr 2020 10:12:33 +0200 "Yann E. MORIN" <[email protected]>…
The rootfses, unlike packages, do not have stampfiles to represent whether they are built or not. Indeed, we always rebuild the rootfs, and there is a single step to do.
Hpwever, people (and scripts) who want to report on the build progress,
*However
Quoted 21 lines
Quoted 21 lines
will want to know whether each rootfs has been built already or not.
Expose in show-info the name of the file that wil contain the rootfs image.
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc eeppeliteloop at gmail.com--- package/pkg-utils.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-utils.mk +0
-0@@ -174,6 +174,10 @@ define _json-info-pkg-details endef
Quoted 7 lines
package/pkg-utils.mk define _json-info-pkg-details · line 180
Quoted 5 lines
package/pkg-utils.mk +4
-0define _json-info-fs + "image_name": $(if $($(1)_FINAL_IMAGE_NAME), \ + "$($(1)_FINAL_IMAGE_NAME)", \ + null \ + ),Is the condition needed because of the initramfs case?
As far as I remember, yes. I forgot about it when writing the commit log, it seems...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
Arnout Vandecappelle
AppliedRe: [Buildroot] [PATCH 8/8] core/show-info: report image name of filesystems
In reply to Yann E. MORINQuoted 10 lines · Yann E. MORIN
The rootfses, unlike packages, do not have stampfiles to represent whether they are built or not. Indeed, we always rebuild the rootfs, and there is a single step to do.
Hpwever, people (and scripts) who want to report on the build progress, will want to know whether each rootfs has been built already or not.
Expose in show-info the name of the file that wil contain the rootfs image.
Applied to master, thanks.
I extended the commit message with an explanation of the _FINAL_IMAGE_NAME
condition.
Regards,
Arnout
Quoted 15 lines
Signed-off-by Yann E. MORIN <[email protected]>Cc Vadim Kochan <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Arnout Vandecappelle <[email protected]>--- package/pkg-utils.mk | 4 ++++ 1 file changed, 4 insertions(+)package/pkg-utils.mk +0
-0
package/pkg-utils.mk define _json-info-pkg-details · line 183
Quoted 10 lines
package/pkg-utils.mk +4
-0@@ -174,6 +174,10 @@ define _json-info-pkg-details endef define _json-info-fs + "image_name": $(if $($(1)_FINAL_IMAGE_NAME), \ + "$($(1)_FINAL_IMAGE_NAME)", \ + null \ + ), "dependencies": [ $(call make-comma-list,$(sort $($(1)_DEPENDENCIES))) ]
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
cover core/show-info: export extra information about the package build (branch yem/show-info-extras)
In reply to Yann E. MORINQuoted 5 lines · On Sat, 11 Apr 2020 10:12:30 +0200 "Yann E. MORIN" <[email protected]>…
Yann E. MORIN (8):
core/show-info: do not show install types for host packages infra/pkg-generic: don't set INSTALL_{TARGET,STAGING,IMAGES} for host core/show-info: report install types for virtual packages too core/show-info: report the package build directory
So, I've applied these first four patches.
Quoted 3 lines
core/show-info: report whether a package is overriden core/show-info: report package stamp files core/show-info: report the ordered list of build steps
Regarding these 3, as we have discussed, I am not in favor of these changes. I think the "progress" tool should simply have the knowledge of the stamp files and the possible order of steps. It's not like we change that very often.
Quoted 1 line
core/show-info: report image name of filesystems
I just replied to this one to ask a simple question about it. Otherwise, I think it is OK.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com