infra: add solution to dump metadata from packages (branch yem/show-info-2)
v3 cover infra: add solution to dump metadata from packages (branch yem/show-info-2)
Hello All!
This series is a proposal to allow extracting metadata about packages, in a way that makes it reusable for tooling outside of Buildroot, that is both reliable and extendable, without causing too much burden on Buildroot own infrastructure.
This series introduces a global 'show-info' rule, and a per package and per filesystem 'show-info' rule, that dump the package metadata as a JSON blob. JSON is a key-value based serialisation format. As such, it is easy to parse (there are implementations in virtually all languages, and there are even tools like jq, that can be used from shell scripts). But most importantly, it is easy to extend without breakign existing tools (as long as they use proper JSON parsers, that is).
This series first introduces 4 cleanup preparatory patches in the download infra, so that it gets easy to get the list of URIs to report them. Strictly speaking, only patches 1 and 2 are required for this series, but the cleanup was extended to the infra with patches 3 and 4, for consistency.
Then, the series introduces a per package and per filesystem show-info rule. The per filesystem rule is needed to be able to get the info about packages that are a dependency of filesystems.
Finally, a global show-info rule is introduced that generates a JSON array with all the information about a build.
In the end, the output for a packages would look like:
$ make cracklib-show-info |jq . -
{
"cracklib": {
"type": "target",
"virtual": false,
"version": "2.9.7",
"licenses": "LGPL-2.1",
"downloads": [
{
"source": "cracklib-2.9.7.tar.gz",
"URIs": [
"https+https://github.com/cracklib/cracklib/releases/download/v2.9.7",
"http|urlencode+http://sources.buildroot.net/cracklib",
"http|urlencode+http://sources.buildroot.net"
]
},
{
"source": "cracklib-words-2.9.7.gz",
"URIs": [
"https://github.com/cracklib/cracklib/releases/download/v2.9.7",
"http|urlencode+http://sources.buildroot.net/cracklib",
"http|urlencode+http://sources.buildroot.net"
]
}
],
"dependencies": [
"host-cracklib",
"host-skeleton",
"skeleton",
"toolchain"
],
"reverse_dependencies": [
"libpwquality",
"linux-pam"
]
}
}While the whole show-info would look like (elipsed and manually indented for readbility):
$ make show-info
{
"host-gcc-final": { "type": "host", "virtual": false, ... },
"host-binutils": { "type": "host", "virtual": false, ... },
"toolchain-buildroot": { "type": "target", "virtual": true, ... },
"gettext-tiny": { "type": "target", "virtual": false, ... },
"gettext": { "type": "target", "virtual": true, ... },
"ifupdown-scripts": { "type": "target", "virtual": false, ... },
...
"rootfs-common": { "type": "rootfs", ... },
"rootfs-tar": { "type": "rootfs", ... }
}Changes v2 -> v3:
- set the virutal field in a single place (Arnout)
- rename internal macro (Arnout)
- don't add opening and closing {} in clean-json (Arnout)
- add to 'make help'
Changes v1 -> v2:
- DOWNLOAD_URIS uses foreach instead of patsubst (Thomas P.)
- rename dependencies keys (Thomas DS)
- split DOWNLOAD/DOWNLOAD_URIS comments (Thomas DS)
- make-comma-list uses an explicit $(space) (Thomas DS)
- top-level show-info is a dictionary, not an array (Arnout)
- use a macro to generate a JSON element for an item (Arnout)
- drop the 'null' object in download list (Arnout, Thomas DS)
- change graph-depends to use that instead of show-dependency-graph
- drop show-dependency-graph
- fix double list of URIs
- fix typoes
Regards,
Yann E. MORIN.
The following changes since commit a45a3997d8ee0aa592c6a64ce300ee727ad7dc54
package/cjson: security bump to version 1.7.11 (2019-04-15 21:40:14 +0200)
are available in the git repository at:
git://git.buildroot.org/~ymorin/git/buildroot.git
for you to fetch changes up to add67b53daa3c056dad8577f57d483c2bbda3fc6
core: remove show-depednency-tree (2019-04-15 21:46:41 +0200)
----------------------------------------------------------------
Yann E. MORIN (12):
infra/pkg-download: return just a list of URIs infra/pkg-download: make the URI list a callable macro infra/pkg-download: get rid of the FLOCK variable infra/pkg-download: make the DOWNLOAD macro fully parameterised infra/utils: add helper to generate comma-separated lists
fs: introduce variables with name and type
fs: introduce variable with all recursive dependencies
fs: add all recursive dependencies to packages list
core: introduce new global show-info
core: add per-package and per-filesystem show-info support/scripts: use show-info to extract dependency graph
core: remove show-depednency-tree
Makefile | 21 ++++++++++++--- fs/common.mk | 45 +++++++++++++++++++++++++------ package/pkg-download.mk | 46 ++++++++++++++++++------------- package/pkg-generic.mk | 11 ++++---- package/pkg-utils.mk | 64 ++++++++++++++++++++++++++++++++++++++++++++ support/misc/utils.mk | 4 +++ support/scripts/brpkgutil.py | 30 ++++++++++----------- 7 files changed, 168 insertions(+), 53 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 1/12 infra/pkg-download: return just a list of URIs
Currently, the internal DOWNLOAD_URIS variable is set to be a list of options to pass to the download wrapper, with each URI passed as '-u URI'.
This precludes using that variable to get just the list of URIs, in case we need to do something else with it.
Fix the variable to really only contain the list of URIs.
Adapt the caller accordingly.
---
Changes v1 -> v2:
- use foreach instead of patsubst (Thomas)
--- package/pkg-download.mk | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
package/pkg-download.mk +6 -6
@@ -75,17 +75,17 @@ export BR_NO_CHECK_HASH_FOR = ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),) DOWNLOAD_URIS += \ - -u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ - -u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode) + $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ + $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode) endif ifeq ($(BR2_PRIMARY_SITE_ONLY),) DOWNLOAD_URIS += \ - -u $(patsubst %/,%,$(dir $(call qstrip,$(1)))) + $(patsubst %/,%,$(dir $(call qstrip,$(1)))) ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),) DOWNLOAD_URIS += \ - -u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ - -u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode) + $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ + $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode) endif endif
@@ -101,7 +101,7 @@ define DOWNLOAD -N '$($(PKG)_RAWNAME)' \ -o '$($(PKG)_DL_DIR)/$(notdir $(1))' \ $(if $($(PKG)_GIT_SUBMODULES),-r) \ - $(DOWNLOAD_URIS) \ + $(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \ $(QUIET) \ -- \ $($(PKG)_DL_OPTS)
Signature · 1 line
2.14.1
PATCH v3 2/12 infra/pkg-download: make the URI list a callable macro
In reply to Yann E. MORINCurrently, that variable is context-dependent, as it expects the PKG variable to exist and be defined to the current package.
This is not so clean, so change the variable to a callable macro.
Adapt the caller accordingly.
---
Changes v1 -> v2:
- split DOWNLOAD/DOWNLOAD_URIS comment (Thomas DS)
--- package/pkg-download.mk | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
package/pkg-download.mk +13 -5
@@ -63,19 +63,19 @@ github = https://github.com/$(1)/$(2)/archive/$(3) export BR_NO_CHECK_HASH_FOR = ################################################################################ -# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download -# source from: +# DOWNLOAD_URIS - List the candidates URIs where to get the package from: # 1) BR2_PRIMARY_SITE if enabled # 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set # # Argument 1 is the source location +# Argument 2 is the upper-case package name # ################################################################################ ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),) DOWNLOAD_URIS += \ - $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ + $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(2)_DL_SUBDIR)),urlencode) \ $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode) endif
@@ -84,11 +84,19 @@ DOWNLOAD_URIS += \ $(patsubst %/,%,$(dir $(call qstrip,$(1)))) ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),) DOWNLOAD_URIS += \ - $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ + $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(2)_DL_SUBDIR)),urlencode) \ $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode) endif endif +################################################################################ +# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download +# source from the list returned by DOWNLOAD_URIS. +# +# Argument 1 is the source location +# +################################################################################ + define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
@@ -101,7 +109,7 @@ define DOWNLOAD -N '$($(PKG)_RAWNAME)' \ -o '$($(PKG)_DL_DIR)/$(notdir $(1))' \ $(if $($(PKG)_GIT_SUBMODULES),-r) \ - $(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \ + $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \ $(QUIET) \ -- \ $($(PKG)_DL_OPTS)
Signature · 1 line
2.14.1
PATCH v3 3/12 infra/pkg-download: get rid of the FLOCK variable
In reply to Yann E. MORINThe FLOCK variable is context-dependent, and expects the PKG variable to be set to the current package.
This is not so nice. Besides, it is used in a single location.
Get rid of this intermediate variable, and directly use flock where we need it.
--- package/pkg-download.mk | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
package/pkg-download.mk +1 -2
@@ -18,7 +18,6 @@ export SCP := $(call qstrip,$(BR2_SCP)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper -FLOCK = flock $($(PKG)_DL_DIR)/ # DL_DIR may have been set already from the environment ifeq ($(origin DL_DIR),undefined)
@@ -99,7 +98,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) - $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \
Signature · 1 line
2.14.1
PATCH v3 4/12 infra/pkg-download: make the DOWNLOAD macro fully parameterised
In reply to Yann E. MORINCurrently, the DOWNLOAD macro is context-dependent and expects the PKG variable to be set to the current package.
This is not so nice.
Change the macro to expect the upper-case package name as a parameter, rather than expect it from a variable.
Adapt the caller accordingly.
---
changes v1 -> v2:
- DOWNLOAD_URIS was called twice in v1, once with $(PKG) and once with
$(1) (due to an incorrect rebase) --- package/pkg-download.mk | 23 ++++++++++++----------- package/pkg-generic.mk | 4 ++-- 2 files changed, 14 insertions(+), 13 deletions(-)
package/pkg-download.mk +12 -11
@@ -93,23 +93,24 @@ endif # source from the list returned by DOWNLOAD_URIS. # # Argument 1 is the source location +# Argument 2 is the upper-case package name # ################################################################################ define DOWNLOAD - $(Q)mkdir -p $($(PKG)_DL_DIR) - $(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \ - -c '$($(PKG)_DL_VERSION)' \ - -d '$($(PKG)_DL_DIR)' \ + $(Q)mkdir -p $($(2)_DL_DIR) + $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/ $(DL_WRAPPER) \ + -c '$($(2)_DL_VERSION)' \ + -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \ -f '$(notdir $(1))' \ - -H '$($(PKG)_HASH_FILE)' \ - -n '$($(PKG)_BASENAME_RAW)' \ - -N '$($(PKG)_RAWNAME)' \ - -o '$($(PKG)_DL_DIR)/$(notdir $(1))' \ - $(if $($(PKG)_GIT_SUBMODULES),-r) \ - $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \ + -H '$($(2)_HASH_FILE)' \ + -n '$($(2)_BASENAME_RAW)' \ + -N '$($(2)_RAWNAME)' \ + -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_GIT_SUBMODULES),-r) \ + $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \ -- \ - $($(PKG)_DL_OPTS) + $($(2)_DL_OPTS) endef
package/pkg-generic.mk +2 -2
@@ -149,7 +149,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: break ; \ fi ; \ done - $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep)) + $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D) @$(call step_end,download)
@@ -158,7 +158,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: # Retrieve actual source archive, e.g. for prebuilt external toolchains $(BUILD_DIR)/%/.stamp_actual_downloaded: @$(call step_start,actual-download) - $(call DOWNLOAD,$($(PKG)_ACTUAL_SOURCE_SITE)/$($(PKG)_ACTUAL_SOURCE_TARBALL)) + $(call DOWNLOAD,$($(PKG)_ACTUAL_SOURCE_SITE)/$($(PKG)_ACTUAL_SOURCE_TARBALL),$(PKG)) $(Q)mkdir -p $(@D) @$(call step_end,actual-download) $(Q)touch $@
Signature · 1 line
2.14.1
PATCH v3 5/12 infra/utils: add helper to generate comma-separated lists
In reply to Yann E. MORINAdd a helper macro that, from a space-separated list of items, returns a comma-separated list of the quoted items.
This will be useful when we need to generate lists in JSON, later...
Code suggested by Thomas P.
---
Changes v1 -> v2:
- don't rely on implicit space in replacement text, instead use
explicit $(space) (Thmas DS) --- support/misc/utils.mk | 4 ++++ 1 file changed, 4 insertions(+)
support/misc/utils.mk +4 -0
@@ -70,6 +70,10 @@ finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2))) # notfirstword(wordlist): returns all but the first word in wordlist notfirstword = $(wordlist 2,$(words $(1)),$(1)) +# build a comma-separated list of quoted items, from a space-separated +# list of unquoted items: a b c d --> "a", "b", "c", "d" +make-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,"%",$(strip $(1)))) + # Needed for the foreach loops to loop over the list of hooks, so that # each hook call is properly separated by a newline. define sep
Signature · 1 line
2.14.1
PATCH v3 6/12 fs: introduce variables with name and type
In reply to Yann E. MORINThis makes the filesystems resemble packages yet a bit more, and will allow sorting "items" on their type and names, when indexed from the upper-case names.
--- fs/common.mk | 4 ++++ 1 file changed, 4 insertions(+)
fs/common.mk +4 -0
@@ -42,6 +42,8 @@ define ROOTFS_REPRODUCIBLE endef endif +ROOTFS_COMMON_NAME = rootfs-common +ROOTFS_COMMON_TYPE = rootfs ROOTFS_COMMON_DEPENDENCIES = \ host-fakeroot host-makedevs \ $(BR2_TAR_HOST_DEPENDENCY) \
@@ -77,6 +79,8 @@ rootfs-common-show-depends: # all variable references except the arguments must be $$-quoted. define inner-rootfs +ROOTFS_$(2)_NAME = rootfs-$(1) +ROOTFS_$(2)_TYPE = rootfs ROOTFS_$(2)_IMAGE_NAME ?= rootfs.$(1) ROOTFS_$(2)_FINAL_IMAGE_NAME = $$(strip $$(ROOTFS_$(2)_IMAGE_NAME)) ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1)
Signature · 1 line
2.14.1
PATCH v3 7/12 fs: introduce variable with all recursive dependencies
In reply to Yann E. MORINThis allows getting all the recursive dependencies of filesystems, ike we have for packages, and allows us to treat both in a similar fashion.
--- fs/common.mk | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
fs/common.mk +24 -0
@@ -49,6 +49,18 @@ ROOTFS_COMMON_DEPENDENCIES = \ $(BR2_TAR_HOST_DEPENDENCY) \ $(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd) +ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \ + $(if $(filter undefined,$(origin ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X)), \ + $(eval ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X := \ + $(foreach p, \ + $(ROOTFS_COMMON_DEPENDENCIES), \ + $(p) \ + $($(call UPPERCASE,$(p))_FINAL_RECURSIVE_DEPENDENCIES) \ + ) \ + ) \ + ) \ + $(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X)) + rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES)) $(info rootfs-common: host) $(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
@@ -88,6 +100,18 @@ ROOTFS_$(2)_TARGET_DIR = $$(ROOTFS_$(2)_DIR)/target ROOTFS_$(2)_DEPENDENCIES += rootfs-common +ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \ + $$(if $$(filter undefined,$$(origin ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X)), \ + $$(eval ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X := \ + $$(foreach p, \ + $$(ROOTFS_$(2)_DEPENDENCIES), \ + $$(p) \ + $$($$(call UPPERCASE,$$(p))_FINAL_RECURSIVE_DEPENDENCIES) \ + ) \ + ) \ + ) \ + $$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X)) + rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES)) $$(info rootfs-$(1): host) $$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
Signature · 1 line
2.14.1
PATCH v3 8/12 fs: add all recursive dependencies to packages list
In reply to Yann E. MORINCurrently, only first-level dependencies of a filesystem are added to the global list of packages, thus missing all recursive dependencies.
Use the newly introduced recursive variable instead, which already contains the rootfs-common dependencies too.
--- fs/common.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -1
@@ -190,7 +190,7 @@ rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) TARGETS_ROOTFS += rootfs-$(1) -PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES) $$(ROOTFS_COMMON_DEPENDENCIES)) +PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES)) endif # Check for legacy POST_TARGETS rules
Signature · 1 line
2.14.1
PATCH v3 9/12 core: introduce new global show-info
In reply to Yann E. MORINUsers are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package.
Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies...
We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq). It also closely matches Python data structure, which makes it easy to use with our own internal tools as well. Finally, JSON being a key-value store, allows for easy expanding the output without requiring existing consumers to be updated; new, unknown keys are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of the data: virtual packages have no source, version, license or downloads, unlike non-virtual packages. Same goes for filesystems. We use a wrapper macro, show-info, that de-multiplexes unto either the package-related- or filesystem-related macros, and for packages, we also use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially when trying to output an array of objects, like so, where the last item shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of ",}" or ", }" or ",]" or ", ]" with only the respective closing symbol, "}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb that fits on a single line with all spaces squashed (but still with spaces, as it is not possible to differentiate spaces between JSON elements from spaces inside JSON strings).
---
Changes v2 -> v3:
- set the virtual field in a single place (Arnout)
- rename the macros (Arnout)
- don't add opening and closing {} in clean-json (Arnout)
- add to 'make help'
Changes v1 -> v2:
- make it a macro to be called (Arnout)
- make it a top-level rule (Arnout)
--- Makefile | 16 +++++++++++++ package/pkg-utils.mk | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+)
Makefile +16 -0
@@ -903,6 +903,21 @@ check-dependencies: @cd "$(CONFIG_DIR)"; \ $(TOPDIR)/support/scripts/graph-depends -C +.PHONY: show-info +show-info: + @: + $(info $(call clean-json, \ + { $(foreach p, \ + $(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \ + $(i) \ + $($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \ + ) \ + ), \ + $(call json-info,$(call UPPERCASE,$(p)))$(comma) \ + ) } \ + ) \ + ) + else # ifeq ($(BR2_HAVE_DOT_CONFIG),y) # Some subdirectories are also package names. To avoid that "make linux"
@@ -1128,6 +1143,7 @@ help: @echo ' source - download all sources needed for offline-build' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' + @echo ' show-info - generate info about packages, as a JSON blurb' @echo ' printvars - dump internal variables selected with VARS=...' @echo @echo ' make V=0|1 - 0 => quiet build (default), 1 => verbose build'
package/pkg-utils.mk +64 -0
70 lines not on this page · raw message
Signature · 1 line
2.14.1
PATCH v3 10/12 core: add per-package and per-filesystem show-info
In reply to Yann E. MORINSometimes, it is need to quickly get the metadata of a subset of packages, without resorting to a full-blown JSON query.
Introduce a new per-package (and per-filesystem) foo-show-info rule, that otputs a per-entity valid JSON blob.
Note that calling it for multiple packages and.or filesystems at once will not generate a valid JSON blob, as there would be no separator between the JSON elements:
$ make {foo,bar}-show-info
{ "foo": { foo stuff } }
{ "bar": { bar stuff } }However, jq is able to absorb this, with its slurping ability, which generates an array (ellipsed and manualy reformated for readability):
$ make {foo,bar}-show-info |jq -s . -
[
{ "foo": { foo stuff } },
{ "bar": { bar stuff } }
]---
Changes v2 -> v3:
- add to 'make help'
Changes v1 -> v2:
- make it based on the new show-info macro (Arnout)
--- Makefile | 1 + fs/common.mk | 11 ++++++++++- package/pkg-generic.mk | 5 +++++ 3 files changed, 16 insertions(+), 1 deletion(-)
Makefile +1 -0
@@ -1111,6 +1111,7 @@ help: @echo ' <pkg>-depends - Build <pkg>'\''s dependencies' @echo ' <pkg>-configure - Build <pkg> up to the configure step' @echo ' <pkg>-build - Build <pkg> up to the build step' + @echo ' <pkg>-show-info - generate info about <pkg>, as a JSON blurb' @echo ' <pkg>-show-depends - List packages on which <pkg> depends' @echo ' <pkg>-show-rdepends - List packages which have <pkg> as a dependency' @echo ' <pkg>-show-recursive-depends'
fs/common.mk +10 -1
@@ -87,6 +87,11 @@ endif rootfs-common-show-depends: @echo $(ROOTFS_COMMON_DEPENDENCIES) +.PHONY: rootfs-common-show-info +rootfs-common-show-info: + @: + $(info $(call clean-json,{ $(call json-info,ROOTFS_COMMON) })) + # Since this function will be called from within an $(eval ...) # all variable references except the arguments must be $$-quoted. define inner-rootfs
@@ -184,9 +189,13 @@ endif rootfs-$(1)-show-depends: @echo $$(ROOTFS_$(2)_DEPENDENCIES) +rootfs-$(1)-show-info: + @: + $$(info $$(call clean-json,{ $$(call json-info,ROOTFS_$(2)) })) + rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME) -.PHONY: rootfs-$(1) rootfs-$(1)-show-depends +.PHONY: rootfs-$(1) rootfs-$(1)-show-depends rootfs-$(1)-show-info ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) TARGETS_ROOTFS += rootfs-$(1)
package/pkg-generic.mk +5 -0
@@ -867,6 +867,10 @@ $(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_AL $$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION))) $$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES)) +$(1)-show-info: + @: + $$(info $$(call clean-json,{ $$(call json-info,$(2)) })) + $(1)-graph-depends: graph-depends-requirements $(call pkg-graph-depends,$(1),--direct)
@@ -1099,6 +1103,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-dependency-tree \ $(1)-show-depends \ + $(1)-show-info \ $(1)-show-version \ $(1)-source
Signature · 1 line
2.14.1
PATCH v3 11/12 support/scripts: use show-info to extract dependency graph
In reply to Yann E. MORINCurrently, we extract the dependency graph from the aptly named but ad-hoc show-dependency-graph rule.
We now have a better solution to report package information, with show-info.
Since show-dependency-graph never went into a release so far, and show-info does provide the same (and more), swith to using show-info.
Thanks to Adam for suggesting the coding style to have a readable code that is not ugly but still pleases flake8. Thanks to Arnout for suggesting the use of dict.get() to further simplify the code.
Note: we do not use th reverse_dependencies field because it only contains those packages that have a kconfig option, so we'd miss most host packages.
--- support/scripts/brpkgutil.py | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)
support/scripts/brpkgutil.py +14 -16
@@ -1,12 +1,12 @@ # Copyright (C) 2010-2013 Thomas Petazzoni <[email protected]> # Copyright (C) 2019 Yann E. MORIN <[email protected]> +import json import logging import os import subprocess from collections import defaultdict - # This function returns a tuple of four dictionaries, all using package # names as keys: # - a dictionary which values are the lists of packages that are the
@@ -19,7 +19,7 @@ from collections import defaultdict def get_dependency_tree(): logging.info("Getting dependency tree...") - deps = defaultdict(list) + deps = {} rdeps = defaultdict(list) types = {} versions = {}
@@ -29,23 +29,21 @@ def get_dependency_tree(): types['all'] = 'target' versions['all'] = '' - cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"] + cmd = ["make", "-s", "--no-print-directory", "show-info"] with open(os.devnull, 'wb') as devnull: p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull, universal_newlines=True) - output = p.communicate()[0] + pkg_list = json.loads(p.communicate()[0]) - for l in output.splitlines(): - if " -> " in l: - pkg = l.split(" -> ")[0] - deps[pkg] += l.split(" -> ")[1].split() - for p in l.split(" -> ")[1].split(): - rdeps[p].append(pkg) - else: - pkg, type_version = l.split(": ", 1) - t, v = "{} -".format(type_version).split(None, 2)[:2] - deps['all'].append(pkg) - types[pkg] = t - versions[pkg] = v + for pkg in pkg_list: + deps['all'].append(pkg) + types[pkg] = pkg_list[pkg]["type"] + deps[pkg] = pkg_list[pkg].get("dependencies", []) + for p in deps[pkg]: + rdeps[p].append(pkg) + versions[pkg] = \ + None if pkg_list[pkg]["type"] == "rootfs" \ + else "virtual" if pkg_list[pkg]["virtual"] \ + else pkg_list[pkg]["version"] return (deps, rdeps, types, versions)
Signature · 1 line
2.14.1
PATCH v3 11/12 support/scripts: use show-info to extract dependency graph
Arnout, All,
On 2019-04-15 21:47 +0200, Yann E. MORIN spake thusly:
Quoted 33 lines
Currently, we extract the dependency graph from the aptly named but ad-hoc show-dependency-graph rule.
We now have a better solution to report package information, with show-info.
Since show-dependency-graph never went into a release so far, and show-info does provide the same (and more), swith to using show-info.
Thanks to Adam for suggesting the coding style to have a readable code that is not ugly but still pleases flake8. Thanks to Arnout for suggesting the use of
dict.get()to further simplify the code.Note: we do not use th
reverse_dependenciesfield because it only contains those packages that have a kconfig option, so we'd miss most host packages.Reported-by Thomas Petazzoni <[email protected]>Signed-off-by "Yann E. MORIN" <[email protected]>Cc Thomas Petazzoni <[email protected]>Cc Thomas De Schampheleire <[email protected]>Cc Adam Duskett <[email protected]>Cc Arnout Vandecappelle <[email protected]>Cc Ricardo Martincoski <[email protected]>--- support/scripts/brpkgutil.py | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-)support/scripts/brpkgutil.py +0
-0
support/scripts/brpkgutil.py line 9
Quoted 10 lines
support/scripts/brpkgutil.py +1
-1@@ -1,12 +1,12 @@ # Copyright (C) 2010-2013 Thomas Petazzoni <[email protected]> # Copyright (C) 2019 Yann E. MORIN <[email protected]> +import json import logging import os import subprocess from collections import defaultdict -
Spurious line removal... Can you fix when applying, or should I respin?
Regards,
Yann E. MORIN.
Quoted 51 lines
support/scripts/brpkgutil.py +13
-15# This function returns a tuple of four dictionaries, all using package # names as keys: # - a dictionary which values are the lists of packages that are the@@ -19,7 +19,7 @@ from collections import defaultdict def get_dependency_tree(): logging.info("Getting dependency tree...") - deps = defaultdict(list) + deps = {} rdeps = defaultdict(list) types = {} versions = {}@@ -29,23 +29,21 @@ def get_dependency_tree(): types['all'] = 'target' versions['all'] = '' - cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"] + cmd = ["make", "-s", "--no-print-directory", "show-info"] with open(os.devnull, 'wb') as devnull: p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull, universal_newlines=True) - output = p.communicate()[0] + pkg_list = json.loads(p.communicate()[0]) - for l in output.splitlines(): - if " -> " in l: - pkg = l.split(" -> ")[0] - deps[pkg] += l.split(" -> ")[1].split() - for p in l.split(" -> ")[1].split(): - rdeps[p].append(pkg) - else: - pkg, type_version = l.split(": ", 1) - t, v = "{} -".format(type_version).split(None, 2)[:2] - deps['all'].append(pkg) - types[pkg] = t - versions[pkg] = v + for pkg in pkg_list: + deps['all'].append(pkg) + types[pkg] = pkg_list[pkg]["type"] + deps[pkg] = pkg_list[pkg].get("dependencies", []) + for p in deps[pkg]: + rdeps[p].append(pkg) + versions[pkg] = \ + None if pkg_list[pkg]["type"] == "rootfs" \ + else "virtual" if pkg_list[pkg]["virtual"] \ + else pkg_list[pkg]["version"] return (deps, rdeps, types, versions)Signature · 1 line
2.14.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 11/12 support/scripts: use show-info to extract dependency graph
In reply to Yann E. MORINHello,
One question related to an output that changes with this patch and also some nits.
Quoted 8 lines · Yann E. MORIN
Currently, we extract the dependency graph from the aptly named but ad-hoc show-dependency-graph rule.
We now have a better solution to report package information, with show-info.
Since show-dependency-graph never went into a release so far, and show-info does provide the same (and more), swith to using show-info.
s/swith/switch/
Quoted 6 lines
Thanks to Adam for suggesting the coding style to have a readable code that is not ugly but still pleases flake8. Thanks to Arnout for suggesting the use of
dict.get()to further simplify the code.Note: we do not use th
reverse_dependenciesfield because it only
s/th/the/
Quoted 2 lines
contains those packages that have a kconfig option, so we'd miss most host packages.
When I do this before and after this patch:
$ make qemu_arm_versatile_defconfig $ make graph-depends The color for "rootfs-common" and "rootfs-ext2" changes. Can you reproduce it? Is this intended? If not intended, do we care enough?
[snip]
support/scripts/brpkgutil.py def get_dependency_tree(): · line 47
Quoted 4 lines
support/scripts/brpkgutil.py +4
-0+ versions[pkg] = \ + None if pkg_list[pkg]["type"] == "rootfs" \ + else "virtual" if pkg_list[pkg]["virtual"] \ + else pkg_list[pkg]["version"]
Why not the straightforward version?
if pkg_list[pkg]["type"] == "rootfs":
versions[pkg] = None
elif pkg_list[pkg]["virtual"]:
versions[pkg] = "virtual"
else:
versions[pkg] = pkg_list[pkg]["version"]
Regards,
Ricardo
PATCH v3 11/12 support/scripts: use show-info to extract dependency graph
Ricardo, All,
[Typoes fixed, thanks.]
On 2019-04-21 21:14 -0300, Ricardo Martincoski spake thusly:
Quoted 3 lines · Yann E. MORIN
Quoted 2 lines
Since show-dependency-graph never went into a release so far, and show-info does provide the same (and more), swith to using show-info.
[--SNIP--]
Quoted 5 lines
When I do this before and after this patch:
$ make qemu_arm_versatile_defconfig $ make graph-depends The color for "rootfs-common" and "rootfs-ext2" changes. Can you reproduce it?
Yes.
Quoted 1 line
Is this intended?
More or less, yes.
Quoted 1 line
If not intended, do we care enough?
IMHO, not really.
However, I still have further improvements to do about the graphs, and one of them was to add a new 'colour' to rootfs items.
Quoted 7 lines
[snip]
support/scripts/brpkgutil.py def get_dependency_tree(): · line 47
Quoted 4 lines
support/scripts/brpkgutil.py +4
-0+ versions[pkg] = \ + None if pkg_list[pkg]["type"] == "rootfs" \ + else "virtual" if pkg_list[pkg]["virtual"] \ + else pkg_list[pkg]["version"]
Quoted 7 lines
Why not the straightforward version?
if pkg_list[pkg]["type"] == "rootfs":
versions[pkg] = None
elif pkg_list[pkg]["virtual"]:
versions[pkg] = "virtual"
else:
versions[pkg] = pkg_list[pkg]["version"]
Because this is not pythonic!
Seriously, yes, the main reason is to write python scripts in the most pythonic way I can, for two reasons:
1- I try to use the idioms and best practices of the language I write
in, so that those that come later and are pofficient in the language can maintain it without having to cleanup before;
2- I learn python along the way...
So yes, this looks very more pythonic than the if-else-blocs which look much more like C or shell...
And it takes 6 lines instead of 4! ;-)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v3 12/12 core: remove show-depednency-tree
In reply to Yann E. MORINshow-depednency-tree was introduced in this release cycle, as a way to quickly and easily provide the dependency tree to geaph-depends.
show-dependency-tree is no longer used, no that graph-depends has been switched over to using the more versatile show-info.
Beside, show-dependency-tree has never been part of a release.
Drop it.
--- Makefile | 4 ---- fs/common.mk | 8 -------- package/pkg-generic.mk | 6 ------ 3 files changed, 18 deletions(-)
Makefile +0 -4
@@ -876,10 +876,6 @@ graph-depends-requirements: @dot -? >/dev/null 2>&1 || \ { echo "ERROR: The 'dot' program from Graphviz is needed for graph-depends" >&2; exit 1; } -.PHONY: show-dependency-tree -show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(PACKAGES) $(TARGETS_ROOTFS)) - @: - .PHONY: graph-depends graph-depends: graph-depends-requirements @$(INSTALL) -d $(GRAPHS_DIR)
fs/common.mk +0 -8
@@ -61,10 +61,6 @@ ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \ ) \ $(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X)) -rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES)) - $(info rootfs-common: host) - $(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d))) - .PHONY: rootfs-common rootfs-common: $(ROOTFS_COMMON_DEPENDENCIES) target-finalize @$(call MESSAGE,"Generating root filesystems common tables")
@@ -117,10 +113,6 @@ ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \ ) \ $$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X)) -rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES)) - $$(info rootfs-$(1): host) - $$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d))) - ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y) ROOTFS_$(2)_COMPRESS_EXT = .gz ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
package/pkg-generic.mk +0 -6
@@ -862,11 +862,6 @@ $(1)-show-build-order: $$(patsubst %,%-show-build-order,$$($(2)_FINAL_ALL_DEPEND @: $$(info $(1)) -$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_ALL_DEPENDENCIES)) - @: - $$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION))) - $$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES)) - $(1)-show-info: @: $$(info $$(call clean-json,{ $$(call json-info,$(2)) }))
@@ -1101,7 +1096,6 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-reconfigure \ $(1)-reinstall \ $(1)-rsync \ - $(1)-show-dependency-tree \ $(1)-show-depends \ $(1)-show-info \ $(1)-show-version \
Signature · 1 line
2.14.1
v3 cover infra: add solution to dump metadata from packages (branch yem/show-info-2)
In reply to Yann E. MORINQuoted 11 lines · Yann E. MORIN
Yann E. MORIN (12):
infra/pkg-download: return just a list of URIs infra/pkg-download: make the URI list a callable macro infra/pkg-download: get rid of the FLOCK variable infra/pkg-download: make the DOWNLOAD macro fully parameterised infra/utils: add helper to generate comma-separated lists
fs: introduce variables with name and type
fs: introduce variable with all recursive dependencies
fs: add all recursive dependencies to packages list
core: introduce new global show-info
core: add per-package and per-filesystem show-info
1-10 applied to master, thanks.
Quoted 2 lines
support/scripts: use show-info to extract dependency graph
core: remove show-depednency-tree
I never looked at these before so that might take some time still.
Regards,
Arnout