Skip to contentSkip to conversation

infra: add solution to dump metadata from packages (branch yem/show-info-2)

PATCH Superseded Yann E. MORIN 4 participants 24 messages Apr 14, 2019 – Apr 16, 2019

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover infra: add solution to dump metadata from packages (branch yem/show-info-2)

Hello All!

This series is a proposal to allow extracting metadata about packages, in a way that makes it reusable for tooling outside of Buildroot, that is both reliable and extendable, without causing too much burden on Buildroot own infrastructure.

This series introduces a global 'show-info' rule, and a per package and per filesystem 'show-info' rule, that dump the package metadata as a JSON blob. JSON is a key-value based serialisation format. As such, it is easy to parse (there are implementations in virtually all languages, and there are even tools like jq, that can be used from shell scripts). But most importantly, it is easy to extend without breakign existing tools (as long as they use proper JSON parsers, that is).

This series first introduces 4 cleanup preparatory patches in the download infra, so that it gets easy to get the list of URIs to report them. Strictly speaking, only patches 1 and 2 are required for this series, but the cleanup was extended to the infra with patches 3 and 4, for consistency.

Then, the series introduces a per package and per filesystem show-info rule. The per filesystem rule is needed to be able to get the info about packages that are a dependency of filesystems.

Finally, a global show-info rule is introduced that generates a JSON array with all the information about a build.

In the end, the output for a packages would look like:

    $ make cracklib-show-info |jq . -
    {
      "cracklib": {
        "type": "target",
        "virtual": false,
        "version": "2.9.7",
        "licenses": "LGPL-2.1",
        "downloads": [
          {
            "source": "cracklib-2.9.7.tar.gz",
            "URIs": [
              "https+https://github.com/cracklib/cracklib/releases/download/v2.9.7",
              "http|urlencode+http://sources.buildroot.net/cracklib",
              "http|urlencode+http://sources.buildroot.net"
            ]
          },
          {
            "source": "cracklib-words-2.9.7.gz",
            "URIs": [
              "https://github.com/cracklib/cracklib/releases/download/v2.9.7",
              "http|urlencode+http://sources.buildroot.net/cracklib",
              "http|urlencode+http://sources.buildroot.net"
            ]
          }
        ],
        "dependencies": [
          "host-cracklib",
          "host-skeleton",
          "skeleton",
          "toolchain"
        ],
        "reverse_dependencies": [
          "libpwquality",
          "linux-pam"
        ]
      }
    }

While the whole show-info would look like (elipsed and manually indented for readbility):

    $ make show-info
    {
      "host-gcc-final": { "type": "host", "virtual": false, ... },
      "host-binutils": { "type": "host", "virtual": false, ... },
      "toolchain-buildroot": { "type": "target", "virtual": true, ... },
      "gettext-tiny": { "type": "target", "virtual": false, ... },
      "gettext": { "type": "target", "virtual": true, ... },
      "ifupdown-scripts": { "type": "target", "virtual": false, ... },
      ...
      "rootfs-common": { "type": "rootfs", ... },
      "rootfs-tar": { "type": "rootfs", ... }
    }

Changes v1 -> v2:
- DOWNLOAD_URIS uses foreach instead of patsubst (Thomas P.)
- rename dependencies keys (Thomas DS)
- split DOWNLOAD/DOWNLOAD_URIS comments (Thomas DS)
- make-comma-list uses an explicit $(space) (Thomas DS)
- top-level show-info is a dictionary, not an array (Arnout)
- use a macro to generate a JSON element for an item (Arnout)
- drop the 'null' object in download list (Arnout, Thomas DS)
- change graph-depends to use that instead of show-dependency-graph
- drop show-dependency-graph
- fix double list of URIs
- fix typoes

Regards,
Yann E. MORIN.

The following changes since commit 46cd7061228720ddb93f540bc733359304d8118f

package/dacapo: fix indentation (2019-04-14 17:20:26 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 3f4a75c33a5f817dcd7a3576086ffca4c18e522b

core: remove show-depednency-tree (2019-04-14 17:56:08 +0200)

----------------------------------------------------------------
Yann E. MORIN (12):
infra/pkg-download: return just a list of URIs infra/pkg-download: make the URI list a callable macro infra/pkg-download: get rid of the FLOCK variable infra/pkg-download: make the DOWNLOAD macro fully parameterised infra/utils: add helper to generate comma-separated lists
fs: introduce variables with name and type
fs: introduce variable with all recursive dependencies
fs: add all recursive dependencies to packages list
core: introduce new global show-info
core: add per-package and per-filesystem show-info support/scripts: use show-info to extract dependency graph
core: remove show-depednency-tree

 Makefile                     | 19 ++++++++++---
 fs/common.mk                 | 45 +++++++++++++++++++++++------
 package/pkg-download.mk      | 46 +++++++++++++++++-------------
 package/pkg-generic.mk       |  9 +++---
 package/pkg-utils.mk         | 68 ++++++++++++++++++++++++++++++++++++++++++++
 support/misc/utils.mk        |  4 +++
 support/scripts/brpkgutil.py | 31 +++++++++-----------
 7 files changed, 169 insertions(+), 53 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v2 1/12 infra/pkg-download: return just a list of URIs

Currently, the internal DOWNLOAD_URIS variable is set to be a list of options to pass to the download wrapper, with each URI passed as '-u URI'.

This precludes using that variable to get just the list of URIs, in case we need to do something else with it.

Fix the variable to really only contain the list of URIs.

Adapt the caller accordingly.

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- use foreach instead of patsubst (Thomas)

---
 package/pkg-download.mk | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
package/pkg-download.mk +6 -6
index 7cd87c38ff..4163333998 100644
@@ -75,17 +75,17 @@ export BR_NO_CHECK_HASH_FOR =
 
 ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
 DOWNLOAD_URIS += \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
 endif
 
 ifeq ($(BR2_PRIMARY_SITE_ONLY),)
 DOWNLOAD_URIS += \
-	-u $(patsubst %/,%,$(dir $(call qstrip,$(1))))
+	$(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
-	-u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
 endif
 endif
 
@@ -101,7 +101,7 @@ define DOWNLOAD
 		-N '$($(PKG)_RAWNAME)' \
 		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(DOWNLOAD_URIS) \
+		$(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_DL_OPTS)
Signature · 1 line
2.14.1
#2 lore raw

PATCH v2 2/12 infra/pkg-download: make the URI list a callable macro

In reply to Yann E. MORIN

Currently, that variable is context-dependent, as it expects the PKG variable to exist and be defined to the current package.

This is not so clean, so change the variable to a callable macro.

Adapt the caller accordingly.

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- split DOWNLOAD/DOWNLOAD_URIS comment (Thomas DS)

---
 package/pkg-download.mk | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)
package/pkg-download.mk +13 -5
index 4163333998..623caf9325 100644
@@ -63,19 +63,19 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
 export BR_NO_CHECK_HASH_FOR =
 
 ################################################################################
-# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
-# source from:
+# DOWNLOAD_URIS - List the candidates URIs where to get the package from:
 # 1) BR2_PRIMARY_SITE if enabled
 # 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
 # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
 #
 # Argument 1 is the source location
+# Argument 2 is the upper-case package name
 #
 ################################################################################
 
 ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
 DOWNLOAD_URIS += \
-	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(2)_DL_SUBDIR)),urlencode) \
 	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
 endif
 
@@ -84,11 +84,19 @@ DOWNLOAD_URIS += \
 	$(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
-	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \
+	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(2)_DL_SUBDIR)),urlencode) \
 	$(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode)
 endif
 endif
 
+################################################################################
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from the list returned by DOWNLOAD_URIS.
+#
+# Argument 1 is the source location
+#
+################################################################################
+
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
 	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
@@ -101,7 +109,7 @@ define DOWNLOAD
 		-N '$($(PKG)_RAWNAME)' \
 		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
 		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(foreach uri,$(DOWNLOAD_URIS),-u $(uri)) \
+		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \
 		$(QUIET) \
 		-- \
 		$($(PKG)_DL_OPTS)
Signature · 1 line
2.14.1
#3 lore raw

PATCH v2 3/12 infra/pkg-download: get rid of the FLOCK variable

In reply to Yann E. MORIN

The FLOCK variable is context-dependent, and expects the PKG variable to be set to the current package.

This is not so nice. Besides, it is used in a single location.

Get rid of this intermediate variable, and directly use flock where we need it.

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Maxime Hadjinlian <[email protected]>
---
 package/pkg-download.mk | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
package/pkg-download.mk +1 -2
index 623caf9325..96a12b1f2e 100644
@@ -18,7 +18,6 @@ export SCP := $(call qstrip,$(BR2_SCP))
 export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
 
 DL_WRAPPER = support/download/dl-wrapper
-FLOCK = flock $($(PKG)_DL_DIR)/
 
 # DL_DIR may have been set already from the environment
 ifeq ($(origin DL_DIR),undefined)
@@ -99,7 +98,7 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(PKG)_DL_DIR)
-	$(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \
 		-c '$($(PKG)_DL_VERSION)' \
 		-d '$($(PKG)_DL_DIR)' \
 		-D '$(DL_DIR)' \
Signature · 1 line
2.14.1
#4 lore raw

PATCH v2 4/12 infra/pkg-download: make the DOWNLOAD macro fully parameterised

In reply to Yann E. MORIN

Currently, the DOWNLOAD macro is context-dependent and expects the PKG variable to be set to the current package.

This is not so nice.

Change the macro to expect the upper-case package name as a parameter, rather than expect it from a variable.

Adapt the caller accordingly.

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
changes v1 -> v2:
- DOWNLOAD_URIS was called twice in v1, once with $(PKG) and once with

    $(1) (due to an incorrect rebase)
---
 package/pkg-download.mk | 23 ++++++++++++-----------
 package/pkg-generic.mk  |  2 +-
 2 files changed, 13 insertions(+), 12 deletions(-)
package/pkg-download.mk +12 -11
index 96a12b1f2e..de619ba90a 100644
@@ -93,23 +93,24 @@ endif
 # source from the list returned by DOWNLOAD_URIS.
 #
 # Argument 1 is the source location
+# Argument 2 is the upper-case package name
 #
 ################################################################################
 
 define DOWNLOAD
-	$(Q)mkdir -p $($(PKG)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(PKG)_DL_DIR)/ $(DL_WRAPPER) \
-		-c '$($(PKG)_DL_VERSION)' \
-		-d '$($(PKG)_DL_DIR)' \
+	$(Q)mkdir -p $($(2)_DL_DIR)
+	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/ $(DL_WRAPPER) \
+		-c '$($(2)_DL_VERSION)' \
+		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
 		-f '$(notdir $(1))' \
-		-H '$($(PKG)_HASH_FILE)' \
-		-n '$($(PKG)_BASENAME_RAW)' \
-		-N '$($(PKG)_RAWNAME)' \
-		-o '$($(PKG)_DL_DIR)/$(notdir $(1))' \
-		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(PKG)),-u $(uri)) \
+		-H '$($(2)_HASH_FILE)' \
+		-n '$($(2)_BASENAME_RAW)' \
+		-N '$($(2)_RAWNAME)' \
+		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_GIT_SUBMODULES),-r) \
+		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
 		-- \
-		$($(PKG)_DL_OPTS)
+		$($(2)_DL_OPTS)
 endef
package/pkg-generic.mk +1 -1
index a83813e28d..f78ff2e665 100644
@@ -149,7 +149,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 			break ; \
 		fi ; \
 	done
-	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep))
+	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
 	$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
 	@$(call step_end,download)
Signature · 1 line
2.14.1
#5 lore raw

PATCH v2 4/12 infra/pkg-download: make the DOWNLOAD macro fully parameterised

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_downloaded: · line 155

Quoted 13 lines · Yann E. MORIN
package/pkg-generic.mk +1 -1
index a83813e28d..f78ff2e665 100644
@@ -149,7 +149,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 			break ; \
 		fi ; \
 	done
-	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep))
+	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
 	$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
 	@$(call step_end,download)

Maybe this is the thing you forgot to amend, but: the DOWNLOAD macro is also
used a couple of lines lower for the actual_source.

Regards,
Arnout

#6 lore raw

PATCH v2 4/12 infra/pkg-download: make the DOWNLOAD macro fully parameterised

Arnout, All,

On 2019-04-15 11:08 +0200, Arnout Vandecappelle spake thusly:

Quoted 9 lines · Yann E. MORIN
Quoted 8 lines
package/pkg-generic.mk +0 -0
index a83813e28d..f78ff2e665 100644
@@ -149,7 +149,7 @@ $(BUILD_DIR)/%/.stamp_downloaded:
 			break ; \
 		fi ; \
 	done
Quoted 8 lines

package/pkg-generic.mk $(BUILD_DIR)/%/.stamp_downloaded: · line 155

Quoted 5 lines
package/pkg-generic.mk +1 -1
-	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep))
+	$(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p),$(PKG))$(sep))
 	$(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
 	@$(call step_end,download)

Maybe this is the thing you forgot to amend, but: the DOWNLOAD macro is also
used a couple of lines lower for the actual_source.

Good catch. And no, that was not the thing I missed...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH v2 5/12 infra/utils: add helper to generate comma-separated lists

In reply to Yann E. MORIN

Add a helper macro that, from a space-separated list of items, returns a comma-separated list of the quoted items.

This will be useful when we need to generate lists in JSON, later...

Code suggested by Thomas P.

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- don't rely on implicit space in replacement text, instead use

    explicit $(space)  (Thmas DS)
---
 support/misc/utils.mk | 4 ++++
 1 file changed, 4 insertions(+)
support/misc/utils.mk +4 -0
index c44319338e..b422c67d76 100644
@@ -70,6 +70,10 @@ finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
 # notfirstword(wordlist): returns all but the first word in wordlist
 notfirstword = $(wordlist 2,$(words $(1)),$(1))
 
+# build a comma-separated list of quoted items, from a space-separated
+# list of unquoted items:   a b c d  -->  "a", "b", "c", "d"
+make-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,"%",$(strip $(1))))
+
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep
Signature · 1 line
2.14.1
#8 lore raw

PATCH v2 6/12 fs: introduce variables with name and type

In reply to Yann E. MORIN

This makes the filesystems resemble packages yet a bit more, and will allow sorting "items" on their type and names, when indexed from the upper-case names.

Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 fs/common.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/common.mk +4 -0
index 4ad51fdd0a..286d671d06 100644
@@ -42,6 +42,8 @@ define ROOTFS_REPRODUCIBLE
 endef
 endif
 
+ROOTFS_COMMON_NAME = rootfs-common
+ROOTFS_COMMON_TYPE = rootfs
 ROOTFS_COMMON_DEPENDENCIES = \
 	host-fakeroot host-makedevs \
 	$(BR2_TAR_HOST_DEPENDENCY) \
@@ -77,6 +79,8 @@ rootfs-common-show-depends:
 # all variable references except the arguments must be $$-quoted.
 define inner-rootfs
 
+ROOTFS_$(2)_NAME = rootfs-$(1)
+ROOTFS_$(2)_TYPE = rootfs
 ROOTFS_$(2)_IMAGE_NAME ?= rootfs.$(1)
 ROOTFS_$(2)_FINAL_IMAGE_NAME = $$(strip $$(ROOTFS_$(2)_IMAGE_NAME))
 ROOTFS_$(2)_DIR = $$(FS_DIR)/$(1)
Signature · 1 line
2.14.1
#9 lore raw

PATCH v2 7/12 fs: introduce variable with all recursive dependencies

In reply to Yann E. MORIN

This allows getting all the recursive dependencies of filesystems, ike we have for packages, and allows us to treat both in a similar fashion.

Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 fs/common.mk | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
fs/common.mk +24 -0
index 286d671d06..5ec28ca183 100644
@@ -49,6 +49,18 @@ ROOTFS_COMMON_DEPENDENCIES = \
 	$(BR2_TAR_HOST_DEPENDENCY) \
 	$(if $(PACKAGES_USERS)$(ROOTFS_USERS_TABLES),host-mkpasswd)
 
+ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \
+	$(if $(filter undefined,$(origin ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X)), \
+		$(eval ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X := \
+			$(foreach p, \
+				$(ROOTFS_COMMON_DEPENDENCIES), \
+				$(p) \
+				$($(call UPPERCASE,$(p))_FINAL_RECURSIVE_DEPENDENCIES) \
+			) \
+		) \
+	) \
+	$(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X))
+
 rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES))
 	$(info rootfs-common: host)
 	$(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
@@ -88,6 +100,18 @@ ROOTFS_$(2)_TARGET_DIR = $$(ROOTFS_$(2)_DIR)/target
 
 ROOTFS_$(2)_DEPENDENCIES += rootfs-common
 
+ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \
+	$$(if $$(filter undefined,$$(origin ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X)), \
+		$$(eval ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X := \
+			$$(foreach p, \
+				$$(ROOTFS_$(2)_DEPENDENCIES), \
+				$$(p) \
+				$$($$(call UPPERCASE,$$(p))_FINAL_RECURSIVE_DEPENDENCIES) \
+			) \
+		) \
+	) \
+	$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X))
+
 rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES))
 	$$(info rootfs-$(1): host)
 	$$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
Signature · 1 line
2.14.1
#10 lore raw

PATCH v2 8/12 fs: add all recursive dependencies to packages list

In reply to Yann E. MORIN

Currently, only first-level dependencies of a filesystem are added to the global list of packages, thus missing all recursive dependencies.

Use the newly introduced recursive variable instead, which already contains the rootfs-common dependencies too.

Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +1 -1
index 5ec28ca183..f7989eac57 100644
@@ -190,7 +190,7 @@ rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
-PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES) $$(ROOTFS_COMMON_DEPENDENCIES))
+PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES))
 endif
 
 # Check for legacy POST_TARGETS rules
Signature · 1 line
2.14.1
#11 lore raw

PATCH v2 8/12 fs: add all recursive dependencies to packages list

Quoted 19 lines · Yann E. MORIN

Currently, only first-level dependencies of a filesystem are added to the global list of packages, thus missing all recursive dependencies.

Use the newly introduced recursive variable instead, which already contains the rootfs-common dependencies too.

Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/common.mk +0 -0
index 5ec28ca183..f7989eac57 100644

fs/common.mk rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME) · line 193

Quoted 5 lines
fs/common.mk +1 -1
@@ -190,7 +190,7 @@ rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
-PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES) $$(ROOTFS_COMMON_DEPENDENCIES))
+PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES))

I don't like that we do this for rootfs but not for packages...

So maybe we should indeed change that in the package infra itself:

-PACKAGES += $(1)
+PACKAGES += $$($(2)_FINAL_RECURSIVE_DEPENDENCIES)

(The $$ is as usual not really needed since it will anyway be expanded
immediately because PACKAGES is defined as immediately-expanded (i.e. := instead of =)).

And then of course also change the top-level Makefile to remove the use of
_FINAL_ALL_DEPENDENCIES there.

And as a bonus, you can remove -all-source, -all-external-deps and
-all-legal-info because PACKAGES is already complete.

But this is diverging a little bit from the show-info integration :-)

In other words, this patch is fine as is.

Regards,
Arnout

Quoted 4 lines
fs/common.mk +0 -0
 endif
 
 # Check for legacy POST_TARGETS rules
#12 lore raw

PATCH v2 9/12 core: introduce new global show-info

In reply to Yann E. MORIN

Users are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package.

Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies...

We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq). It also closely matches Python data structure, which makes it easy to use with our own internal tools as well. Finally, JSON being a key-value store, allows for easy expanding the output without requiring existing consumers to be updated; new, unknown keys are simply ignored by those (as long as they are true JSON parsers).

The complex part of this change was the conditional output of parts of the data: virtual packages have no source, version, license or downloads, unlike non-virtual packages. Same goes for filesystems. We use a wrapper macro, show-info, that de-multiplexes unto either the package-related- or filesystem-related macros, and for packages, we also use a detailed macro for non-virtual packages.

It is non-trivial to properly output correct JSON blurbs, especially when trying to output an array of objects, like so, where the last item shall not be followed by a comma: [ { ... }, { ... } ]

So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of ",}" or ", }" or ",]" or ", ]" with only the respective closing symbol, "}" or "]".

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb that fits on a single line with all spaces squashed (but still with spaces, as it is not possible to differentiate spaces between JSON elements from spaces inside JSON strings).

Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- make it a macro to be called (Arnout)
- make it a top-level rule (Arnout)

---
 Makefile             | 15 ++++++++++++
 package/pkg-utils.mk | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
Makefile +15 -0
index 60bf7d7d08..33215f91e5 100644
@@ -903,6 +903,21 @@ check-dependencies:
 	@cd "$(CONFIG_DIR)"; \
 	$(TOPDIR)/support/scripts/graph-depends -C
 
+.PHONY: show-info
+show-info:
+	@:
+	$(info $(call clean-json, \
+			$(foreach p, \
+				$(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \
+						$(i) \
+						$($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \
+					) \
+				), \
+				$(call show-info,$(call UPPERCASE,$(p)))$(comma) \
+			) \
+		) \
+	)
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
package/pkg-utils.mk +68 -0

74 lines not on this page · raw message

Signature · 1 line
2.14.1
#13 lore raw

PATCH v2 9/12 core: introduce new global show-info

Quoted 8 lines · Yann E. MORIN

Users are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package.

Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is.

Quoted 7 lines

Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies...

We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq).

Actually, OpenWrt has a couple of bash functions to convert JSON into bash
variables. If we start making extensive use of show-info, it might be worth copying that.

Quoted 17 lines

It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally, JSON being a key-value store, allows for easy expanding the output without requiring existing consumers to be updated; new, unknown keys are simply ignored by those (as long as they are true JSON parsers).

The complex part of this change was the conditional output of parts of the data: virtual packages have no source, version, license or downloads, unlike non-virtual packages. Same goes for filesystems. We use a wrapper macro, show-info, that de-multiplexes unto either the package-related- or filesystem-related macros, and for packages, we also use a detailed macro for non-virtual packages.

It is non-trivial to properly output correct JSON blurbs, especially when trying to output an array of objects, like so, where the last item shall not be followed by a comma: [ { ... }, { ... } ]

Quoted 8 lines

So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of ",}" or ", }" or ",]" or ", ]" with only the respective closing symbol, "}" or "]".

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb that fits on a single line with all spaces squashed (but still with spaces, as it is not possible to differentiate spaces between JSON elements from spaces inside JSON strings).

The implicit assumption here is that show-info will never include anything
where whitespace is significant. Which I think is a valid assumption (I don't expect any whitespace at all anywhere within the keys or values, except for the license list where the whitespace is usually a list separator anyway).

Quoted 22 lines
Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- make it a macro to be called (Arnout)
- make it a top-level rule (Arnout)

---
 Makefile             | 15 ++++++++++++
 package/pkg-utils.mk | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
Makefile +0 -0
index 60bf7d7d08..33215f91e5 100644
@@ -903,6 +903,21 @@ check-dependencies:
 	@cd "$(CONFIG_DIR)"; \

Makefile check-dependencies: · line 913

Quoted 10 lines
Makefile +8 -0
 	$(TOPDIR)/support/scripts/graph-depends -C
 
+.PHONY: show-info
+show-info:
+	@:
+	$(info $(call clean-json, \
+			$(foreach p, \
+				$(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \
+						$(i) \
+						$($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \

So, with the comment I made in the earlier patch, this line won't be needed
anymore because packages already contains *_FINAL_RECURSIVE_DEPENDENCIES. Boy, this is too good to be true...

Quoted 15 lines
Makefile +7 -0
+					) \
+				), \
+				$(call show-info,$(call UPPERCASE,$(p)))$(comma) \
+			) \
+		) \
+	)
+
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
 # Some subdirectories are also package names. To avoid that "make linux"
package/pkg-utils.mk +0 -0
index bffd79dfb0..00d1d6bcac 100644

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 68

Quoted 7 lines
package/pkg-utils.mk +4 -0
@@ -62,6 +62,74 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
 endif
 endef
 
+# show-info -- return package or filesystem metadata formatted as an entry
+#              of a JSON dictionnary
+# $(1): upper-case package or filesystem name
+define show-info

Really nitpicking here: this function is not showing anything. So maybe
'json-info' is better. Same for the internal functions obviously.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 70

Quoted 2 lines
package/pkg-utils.mk +2 -0
+	"$($(1)_NAME)": {
+		"type": "$($(1)_TYPE)",

I may be exaggerating here, but I am getting a bit confused between commas that
are intrepreted by make and the JSON commas. Maybe we should consistently use $(comma) for the commas that go into the output, even if it is not needed like here?

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 73

Quoted 3 lines
package/pkg-utils.mk +3 -0
+		$(if $(filter rootfs,$($(1)_TYPE)), \
+			$(call _show-info-fs,$(1)), \
+			$(call _show-info-pkg,$(1)), \

Very elegant!

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 82

Quoted 9 lines
package/pkg-utils.mk +9 -0
+		)
+	}
+endef
+
+# _show-info-pkg, _show-info-pkg-details, _show-info-fs: private helpers
+# for show-info, above
+define _show-info-pkg
+	$(if $($(1)_IS_VIRTUAL), \
+		"virtual": true$(comma),

Again, I may be exaggerating, but I would find it more aesthetically pleasing
to put the

		"virtual": false$(comma)

here rather than in the pkg-details.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 91

Quoted 9 lines
package/pkg-utils.mk +9 -0
+		$(call _show-info-pkg-details,$(1)) \
+	)
+	"dependencies": [
+		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+	],
+	"reverse_dependencies": [
+		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+	]
+endef

Add an empty line here.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 95

Quoted 4 lines
package/pkg-utils.mk +4 -0
+define _show-info-pkg-details
+	"virtual": false,
+	"version": "$($(1)_DL_VERSION)",
+	"licenses": "$($(1)_LICENSE)",

I'm torn here...

The licenses could be converted into a list - actually, it is already a list,
just missing the quotes. But clearly, converting that into a JSON list in make syntax is... a challenge.

However, I'm actually dreaming of making the license statement a SPDX
expression instead of a list (i.e. conjoined with AND and OR, and with parenthesis; unfortunately, SPDX has no syntax to express that it only applies to a part, e.g. GPL (programs) is not valid SPDX). Then it is no longer a list.

And we should make a decision now, we're defining "userspace ABI" here so it's
difficult to switch the string to a list or vice versa later.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 100

Quoted 5 lines
package/pkg-utils.mk +5 -0
+	"downloads": [
+	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
+		{
+			"source": "$(notdir $(dl))",
+			"URIs": [

I would prefer all keys to be consistent case, i.e. lowercase.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 110

Quoted 10 lines
package/pkg-utils.mk +10 -0
+				$(call make-comma-list,
+					$(subst \|,|,
+						$(call DOWNLOAD_URIS,$(dl),$(1))
+					)
+				)
+			]
+		},
+	)
+	],
+endef

Empty line again.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 118

Quoted 8 lines
package/pkg-utils.mk +8 -0
+define _show-info-fs
+	"dependencies": [
+		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
+	]
+endef
+
+# clean-json -- cleanup pseudo-json into clean json:
+#  - adds opening { and closing }

I don't think it's appropriate that clean-json does that. Without it, you can
call clean-json as often as you like.

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 121

Quoted 3 lines
package/pkg-utils.mk +3 -0
+#  - remove commas before closing ] and }
+#  - minify with $(strip)
+clean-json = $(strip \

Why strip a second time?

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 122

Quoted 1 line
package/pkg-utils.mk +1 -0
+	$(subst $(comma)},, \
                          ^}

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 124

Quoted 2 lines
package/pkg-utils.mk +2 -0
+		$(subst $(comma)$(space)},$(space)}, \
+			$(subst $(comma)],, \
                                          ^]

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 127

Quoted 3 lines
package/pkg-utils.mk +3 -0
+				$(subst $(comma)$(space)],$(space)], \
+					{ $(strip $(1)) } \
+				) \

Even though incorrect, I think this would be more readable and maintainable by
not indenting the nested subst's:

	$(subst $(comma)},},
	$(subst $(comma)],],
	$(subst $(comma)$(space)},$(space)},
	$(subst $(comma)$(space)],$(space)],
		$(strip $(1))
	))))

Regards,
Arnout

Quoted 9 lines
package/pkg-utils.mk +5 -0
+			) \
+		) \
+	) \
+)
+
 #
 # legal-info helper functions
 #
#14 lore raw

PATCH v2 9/12 core: introduce new global show-info

Quoted 7 lines · On Mon, 15 Apr 2019 14:17:13 +0200 Arnout Vandecappelle <[email protected]> wrote:

However, I'm actually dreaming of making the license statement a SPDX
expression instead of a list (i.e. conjoined with AND and OR, and with parenthesis; unfortunately, SPDX has no syntax to express that it only applies to a part, e.g. GPL (programs) is not valid SPDX). Then it is no longer a list.

And we should make a decision now, we're defining "userspace ABI" here so it's
difficult to switch the string to a list or vice versa later.

Well, you can always have "licenses" be a string corresponding directly to <pkg>_LICENSES as proposed by Yann, and if later on we feel like having the same information in a different format is useful as a list, we could introduce a separate "licenses-list" JSON property.

It is not nice, but it illustrates that even though we are defining userspace ABI, it doesn't mean that we have to find the perfect solution right now.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#15 lore raw

PATCH v2 9/12 core: introduce new global show-info

In reply to Arnout Vandecappelle

Arnout, All,

(note: I won't reply to comments I agree with, so they got elided from my reply, unless I had more to state about them)

On 2019-04-15 14:17 +0200, Arnout Vandecappelle spake thusly:

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 8 lines
Quoted 4 lines

The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb that fits on a single line with all spaces squashed (but still with spaces, as it is not possible to differentiate spaces between JSON elements from spaces inside JSON strings).

The implicit assumption here is that show-info will never include anything
where whitespace is significant. Which I think is a valid assumption (I don't expect any whitespace at all anywhere within the keys or values, except for the license list where the whitespace is usually a list separator anyway).

Indeed. I shall add that to the commit log, then. Thanks!

Quoted 23 lines
Quoted 23 lines
Cc Thomas De Schampheleire <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>

---
Changes v1 -> v2:
- make it a macro to be called (Arnout)
- make it a top-level rule (Arnout)

---
 Makefile             | 15 ++++++++++++
 package/pkg-utils.mk | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
Makefile +0 -0
index 60bf7d7d08..33215f91e5 100644
@@ -903,6 +903,21 @@ check-dependencies:
 	@cd "$(CONFIG_DIR)"; \
 	$(TOPDIR)/support/scripts/graph-depends -C
 
Quoted 12 lines

Makefile check-dependencies: · line 913

Quoted 8 lines
Makefile +8 -0
+.PHONY: show-info
+show-info:
+	@:
+	$(info $(call clean-json, \
+			$(foreach p, \
+				$(sort $(foreach i,$(PACKAGES) $(TARGETS_ROOTFS), \
+						$(i) \
+						$($(call UPPERCASE,$(i))_FINAL_RECURSIVE_DEPENDENCIES) \

So, with the comment I made in the earlier patch, this line won't be needed
anymore because packages already contains *_FINAL_RECURSIVE_DEPENDENCIES. Boy, this is too good to be true...

Won't do that in this series. Can be refined later. ;-)

Quoted 6 lines

Makefile check-dependencies: · line 915

Quoted 2 lines
Makefile +2 -0
+	"$($(1)_NAME)": {
+		"type": "$($(1)_TYPE)",

I may be exaggerating here, but I am getting a bit confused between commas that
are intrepreted by make and the JSON commas. Maybe we should consistently use $(comma) for the commas that go into the output, even if it is not needed like here?

This is a macro definition, not a macro call, so commas are not interpreted.

Quoted 10 lines

Makefile check-dependencies: · line 920

Quoted 5 lines
Makefile +5 -0
+# _show-info-pkg, _show-info-pkg-details, _show-info-fs: private helpers
+# for show-info, above
+define _show-info-pkg
+	$(if $($(1)_IS_VIRTUAL), \
+		"virtual": true$(comma),

Again, I may be exaggerating, but I would find it more aesthetically pleasing
to put the

		"virtual": false$(comma)

here rather than in the pkg-details.

Can doo, OK.

Also, since that would be the else-clause of the if, the folowing commas would *not* be interpreted as separators of the if clause. E.g.:

    all:
        @: $(info $(if ,ignored,shown, too))

would print:

shown, too

Yet, it looks hackish to do so...

Quoted 11 lines

Makefile check-dependencies:

Quoted 4 lines
Makefile +4 -0
+define _show-info-pkg-details
+	"virtual": false,
+	"version": "$($(1)_DL_VERSION)",
+	"licenses": "$($(1)_LICENSE)",

I'm torn here...

The licenses could be converted into a list - actually, it is already a list,
just missing the quotes. But clearly, converting that into a JSON list in make syntax is... a challenge.

Quoted 7 lines

However, I'm actually dreaming of making the license statement a SPDX
expression instead of a list (i.e. conjoined with AND and OR, and with parenthesis; unfortunately, SPDX has no syntax to express that it only applies to a part, e.g. GPL (programs) is not valid SPDX). Then it is no longer a list.

And we should make a decision now, we're defining "userspace ABI" here so it's
difficult to switch the string to a list or vice versa later.

Sorry, two things:

1. I don't think we should treat the _LICENSE fields as a list, becasue it is *very* difficult to split:
FOO_LICENSE = GPL (tools, utils, doc), LGPL (libs)

2. The reason to use JSON is to actually be able to expand it later. We can add and "spdx" object later when we are confident we can generate something reliable.

Quoted 11 lines

Makefile

Quoted 8 lines
Makefile +8 -0
+define _show-info-fs
+	"dependencies": [
+		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
+	]
+endef
+
+# clean-json -- cleanup pseudo-json into clean json:
+#  - adds opening { and closing }

I don't think it's appropriate that clean-json does that. Without it, you can
call clean-json as often as you like.

On the other hand, everywhere we call clean-json, we enclose the result between { }. I.e. the purpose of that macro *is* to output a clean json object.

But OK, I can switch over (that was what I did previously, so I don't have a strong feeling either...)

Quoted 4 lines

Makefile

Quoted 3 lines
Makefile +3 -0
+#  - remove commas before closing ] and }
+#  - minify with $(strip)
+clean-json = $(strip \

Why strip a second time?

First is to eliminate new lines and duplicate spaces, second is to remove duplicate spaces and newlines introduced by the macro itself. Try without either, it is not pretty... ;-)

Quoted 2 lines

Makefile

Quoted 1 line
Makefile +1 -0
+	$(subst $(comma)},, \
                          ^}

This is the part I lost during a borked rebase. :-(

Quoted 3 lines

Makefile

Quoted 2 lines
Makefile +2 -0
+		$(subst $(comma)$(space)},$(space)}, \
+			$(subst $(comma)],, \
                                          ^]

Ditto.

Quoted 7 lines

Makefile

Quoted 3 lines
Makefile +3 -0
+				$(subst $(comma)$(space)],$(space)], \
+					{ $(strip $(1)) } \
+				) \

Even though incorrect, I think this would be more readable and maintainable by
not indenting the nested subst's:

Quoted 6 lines
	$(subst $(comma)},},
	$(subst $(comma)],],
	$(subst $(comma)$(space)},$(space)},
	$(subst $(comma)$(space)],$(space)],
		$(strip $(1))
	))))

Ah, I was not sure this would be appropriate to do so. I even thought of doing:

    clean-json = $(strip \
        $(subst $(comma)},}, $(subst $(comma)$(space)},$(space)},
        $(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
            $(strip $(1)) \
        )))) \
    )

Regards,
Yann E. MORIN.

Quoted 12 lines

Regards,
Arnout

Quoted 9 lines
Makefile +5 -0
+			) \
+		) \
+	) \
+)
+
 #
 # legal-info helper functions
 #
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#16 lore raw

PATCH v2 9/12 core: introduce new global show-info

On 15/04/2019 19:34, Yann E. MORIN wrote:
[snip]

Quoted 9 lines
Quoted 6 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 70

Quoted 2 lines
package/pkg-utils.mk +2 -0
+	"$($(1)_NAME)": {
+		"type": "$($(1)_TYPE)",

I may be exaggerating here, but I am getting a bit confused between commas that
are intrepreted by make and the JSON commas. Maybe we should consistently use $(comma) for the commas that go into the output, even if it is not needed like here?

This is a macro definition, not a macro call, so commas are not interpreted.

I know, of course. My point is that it is not immediately apparent. My mind is
jumbling the JSON separators and the macro call separators together. My thought
was: if we use $(comma) everywhere to mark the JSON separators, things might become more readable.

In fact, in the part you snipped, one line lower there is a macro-separator:

		"type": "$($(1)_TYPE)",
		$(if $(filter rootfs,$($(1)_TYPE)),

You see where I'm coming from?

Now, I can imagine that sprinkling this code with $(comma)s is not going to
help readability one bit. But I thought I'd just float the idea.

Quoted 17 lines
Quoted 11 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 82

Quoted 6 lines
package/pkg-utils.mk +5 -0
+# _show-info-pkg, _show-info-pkg-details, _show-info-fs: private helpers
+# for show-info, above
+define _show-info-pkg
+	$(if $($(1)_IS_VIRTUAL), \
+		"virtual": true$(comma),

Again, I may be exaggerating, but I would find it more aesthetically pleasing
to put the

		"virtual": false$(comma)

here rather than in the pkg-details.

Can doo, OK.

Also, since that would be the else-clause of the if, the folowing commas would *not* be interpreted as separators of the if clause. E.g.:

Quoted 8 lines
    all:
        @: $(info $(if ,ignored,shown, too))

would print:

shown, too

Yet, it looks hackish to do so...

No shit.

[snip]

Quoted 8 lines
Quoted 4 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 121

Quoted 3 lines
package/pkg-utils.mk +3 -0
+#  - remove commas before closing ] and }
+#  - minify with $(strip)
+clean-json = $(strip \

Why strip a second time?

First is to eliminate new lines and duplicate spaces, second is to remove duplicate spaces and newlines introduced by the macro itself. Try without either, it is not pretty... ;-)

Of course, the macro itself inserts additional newlines, I forgot about that.

Quoted 28 lines
Quoted 2 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 122

Quoted 1 line
package/pkg-utils.mk +1 -0
+	$(subst $(comma)},, \
                          ^}

This is the part I lost during a borked rebase. :-(

Quoted 3 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 124

Quoted 2 lines
package/pkg-utils.mk +2 -0
+		$(subst $(comma)$(space)},$(space)}, \
+			$(subst $(comma)],, \
                                          ^]

Ditto.

Quoted 7 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 127

Quoted 3 lines
package/pkg-utils.mk +3 -0
+				$(subst $(comma)$(space)],$(space)], \
+					{ $(strip $(1)) } \
+				) \

Even though incorrect, I think this would be more readable and maintainable by
not indenting the nested subst's:

Quoted 6 lines
	$(subst $(comma)},},
	$(subst $(comma)],],
	$(subst $(comma)$(space)},$(space)},
	$(subst $(comma)$(space)],$(space)],
		$(strip $(1))
	))))

Ah, I was not sure this would be appropriate to do so. I even thought of doing:

Quoted 6 lines
    clean-json = $(strip \
        $(subst $(comma)},}, $(subst $(comma)$(space)},$(space)},
        $(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
            $(strip $(1)) \
        )))) \
    )

It's up to you, really. Same as for the $(comma) thing: you can decide what
looks best for you and I'll accept it.

Regards,
Arnout

#17 lore raw

PATCH v2 9/12 core: introduce new global show-info

Arnout, All,

On 2019-04-15 19:51 +0200, Arnout Vandecappelle spake thusly:

Quoted 17 lines · Yann E. MORIN

[snip]

Quoted 9 lines
Quoted 6 lines

package/pkg-utils.mk $$(error Package error: use $(2) instead of $(1). Please fix your .mk file) · line 70

Quoted 2 lines
package/pkg-utils.mk +2 -0
+	"$($(1)_NAME)": {
+		"type": "$($(1)_TYPE)",

I may be exaggerating here, but I am getting a bit confused between commas that
are intrepreted by make and the JSON commas. Maybe we should consistently use $(comma) for the commas that go into the output, even if it is not needed like here?

This is a macro definition, not a macro call, so commas are not interpreted.

I know, of course. My point is that it is not immediately apparent. My mind is
jumbling the JSON separators and the macro call separators together. My thought
was: if we use $(comma) everywhere to mark the JSON separators, things might become more readable.

Quoted 6 lines

In fact, in the part you snipped, one line lower there is a macro-separator:

		"type": "$($(1)_TYPE)",
		$(if $(filter rootfs,$($(1)_TYPE)),

You see where I'm coming from?

Yeah, I see now.

Quoted 2 lines

Now, I can imagine that sprinkling this code with $(comma)s is not going to
help readability one bit. But I thought I'd just float the idea.

To be honest, I tried to use $(comma) in place of actual commas. But seriously, that is totally unreadable... :-(

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#18 lore raw

PATCH v2 9/12 core: introduce new global show-info

In reply to Arnout Vandecappelle
Quoted 15 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Quoted 12 lines · Yann E. MORIN

Users are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package.

Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies...

Quoted 7 lines
Quoted 3 lines

We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq).

Actually, OpenWrt has a couple of bash functions to convert JSON into bash
variables. If we start making extensive use of show-info, it might be worth copying that.

That should be pretty trivial to do with jq, E.G. something like:

https://stackoverflow.com/questions/25378013/how-to-convert-a-json-object-to-key-value-format-in-jq

Signature · 1 line
Bye, Peter Korsgaard
#19 lore raw

PATCH v2 9/12 core: introduce new global show-info

Quoted 19 lines · Peter Korsgaard
Quoted 18 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Quoted 15 lines · Yann E. MORIN

Users are increasingly trying to extract information about packages. For example, they might need to get the list of URIs, or the dependencies of a package.

Although we do have a bunch of rules to generate some of that, this is done in ad-hoc way, with most of the output formats just ad-hoc, raw, unformatted blurbs, mostly internal data dumped as-is.

Introduce a new rule, show-info, that provides a properly formatted output of all the meta-information about packages: name, type, version, licenses, dependencies...

We choose to use JSON as the output format, because it is pretty versatile, has parsers in virtually all languages, has tools to parse from the shell (jq).

Quoted 7 lines
Quoted 3 lines

Actually, OpenWrt has a couple of bash functions to convert JSON into bash
variables. If we start making extensive use of show-info, it might be worth copying that.

That should be pretty trivial to do with jq, E.G. something like:

https://stackoverflow.com/questions/25378013/how-to-convert-a-json-object-to-key-value-format-in-jq

Yes, but then jq needs to be installed. The point is to be able to use the
show-info output in our own shell scripts without requiring an additional package to be installed.

Regards,
Arnout

#20 lore raw

PATCH v2 9/12 core: introduce new global show-info

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Arnout" == Arnout Vandecappelle <[email protected]> writes:

Hi,

Quoted 7 lines
Quoted 3 lines

That should be pretty trivial to do with jq, E.G. something like:

https://stackoverflow.com/questions/25378013/how-to-convert-a-json-object-to-key-value-format-in-jq

Yes, but then jq needs to be installed. The point is to be able to use the
show-info output in our own shell scripts without requiring an additional package to be installed.

Ok, but we do have a host-jq package and jq has no extra dependencies.

Signature · 1 line
Bye, Peter Korsgaard
#21 lore raw

PATCH v2 10/12 core: add per-package and per-filesystem show-info

In reply to Yann E. MORIN

Sometimes, it is need to quickly get the metadata of a subset of packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule, that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once will not generate a valid JSON blob, as there would be no separator between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 fs/common.mk           | 11 ++++++++++-
 package/pkg-generic.mk |  5 +++++
 2 files changed, 15 insertions(+), 1 deletion(-)
fs/common.mk +10 -1
index f7989eac57..ce17c35c59 100644
@@ -87,6 +87,11 @@ endif
 rootfs-common-show-depends:
 	@echo $(ROOTFS_COMMON_DEPENDENCIES)
 
+.PHONY: rootfs-common-show-info
+rootfs-common-show-info:
+	@:
+	$(info $(call clean-json,$(call show-info,ROOTFS_COMMON)))
+
 # Since this function will be called from within an $(eval ...)
 # all variable references except the arguments must be $$-quoted.
 define inner-rootfs
@@ -184,9 +189,13 @@ endif
 rootfs-$(1)-show-depends:
 	@echo $$(ROOTFS_$(2)_DEPENDENCIES)
 
+rootfs-$(1)-show-info:
+	@:
+	$$(info $$(call clean-json,$$(call show-info,ROOTFS_$(2))))
+
 rootfs-$(1): $$(BINARIES_DIR)/$$(ROOTFS_$(2)_FINAL_IMAGE_NAME)
 
-.PHONY: rootfs-$(1) rootfs-$(1)-show-depends
+.PHONY: rootfs-$(1) rootfs-$(1)-show-depends rootfs-$(1)-show-info
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
package/pkg-generic.mk +5 -0
index f78ff2e665..25bc2e527a 100644
@@ -867,6 +867,10 @@ $(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_AL
 	$$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION)))
 	$$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES))
 
+$(1)-show-info:
+	@:
+	$$(info $$(call clean-json,$$(call show-info,$(2))))
+
 $(1)-graph-depends: graph-depends-requirements
 	$(call pkg-graph-depends,$(1),--direct)
 
@@ -1099,6 +1103,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-rsync \
 	$(1)-show-dependency-tree \
 	$(1)-show-depends \
+	$(1)-show-info \
 	$(1)-show-version \
 	$(1)-source
 
Signature · 1 line
2.14.1
#22 lore raw

PATCH v2 11/12 support/scripts: use show-info to extract dependency graph

In reply to Yann E. MORIN

Currently, we extract the dependency graph from the aptly named but ad-hoc show-dependency-graph rule.

We now have a better solution to report package information, with show-info.

Since show-dependency-graph never went into a release so far, and show-info does provide the same (and more), swith to using show-info.

Thanks to Adam for suggesting the coding style to have a readable code that is not ugly but still pleases flake8. Thanks to Arnout for suggesting the use of dict.get() to further simplify the code.

Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Adam Duskett <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Ricardo Martincoski <[email protected]>
---
 support/scripts/brpkgutil.py | 31 ++++++++++++++-----------------
 1 file changed, 14 insertions(+), 17 deletions(-)
support/scripts/brpkgutil.py +14 -17
index f65068d348..22ceab2b10 100644
@@ -1,10 +1,10 @@
 # Copyright (C) 2010-2013 Thomas Petazzoni <[email protected]>
 # Copyright (C) 2019 Yann E. MORIN <[email protected]>
 
+import json
 import logging
 import os
 import subprocess
-from collections import defaultdict
 
 
 # This function returns a tuple of four dictionaries, all using package
@@ -19,8 +19,8 @@ from collections import defaultdict
 def get_dependency_tree():
     logging.info("Getting dependency tree...")
 
-    deps = defaultdict(list)
-    rdeps = defaultdict(list)
+    deps = {}
+    rdeps = {}
     types = {}
     versions = {}
 
@@ -29,23 +29,20 @@ def get_dependency_tree():
     types['all'] = 'target'
     versions['all'] = ''
 
-    cmd = ["make", "-s", "--no-print-directory", "show-dependency-tree"]
+    cmd = ["make", "-s", "--no-print-directory", "show-info"]
     with open(os.devnull, 'wb') as devnull:
         p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull,
                              universal_newlines=True)
-        output = p.communicate()[0]
+        pkg_list = json.loads(p.communicate()[0])
 
-    for l in output.splitlines():
-        if " -> " in l:
-            pkg = l.split(" -> ")[0]
-            deps[pkg] += l.split(" -> ")[1].split()
-            for p in l.split(" -> ")[1].split():
-                rdeps[p].append(pkg)
-        else:
-            pkg, type_version = l.split(": ", 1)
-            t, v = "{} -".format(type_version).split(None, 2)[:2]
-            deps['all'].append(pkg)
-            types[pkg] = t
-            versions[pkg] = v
+    for pkg in pkg_list:
+        deps['all'].append(pkg)
+        types[pkg] = pkg_list[pkg]["type"]
+        deps[pkg] = pkg_list[pkg].get("dependencies", [])
+        rdeps[pkg] = pkg_list[pkg].get("reverse_dependencies", [])
+        versions[pkg] = \
+            None if pkg_list[pkg]["type"] == "rootfs" \
+            else "virtual" if pkg_list[pkg]["virtual"] \
+            else pkg_list[pkg]["version"]
 
     return (deps, rdeps, types, versions)
Signature · 1 line
2.14.1
#23 lore raw

PATCH v2 12/12 core: remove show-depednency-tree

In reply to Yann E. MORIN

show-depednency-tree was introduced in this release cycle, as a way to quickly and easily provide the dependency tree to geaph-depends.

show-dependency-tree is no longer used, no that graph-depends has been switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
---
 Makefile               | 4 ----
 fs/common.mk           | 8 --------
 package/pkg-generic.mk | 6 ------
 3 files changed, 18 deletions(-)
Makefile +0 -4
index 33215f91e5..d9f2f6e645 100644
@@ -876,10 +876,6 @@ graph-depends-requirements:
 	@dot -? >/dev/null 2>&1 || \
 		{ echo "ERROR: The 'dot' program from Graphviz is needed for graph-depends" >&2; exit 1; }
 
-.PHONY: show-dependency-tree
-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(PACKAGES) $(TARGETS_ROOTFS))
-	@:
-
 .PHONY: graph-depends
 graph-depends: graph-depends-requirements
 	@$(INSTALL) -d $(GRAPHS_DIR)
fs/common.mk +0 -8
index ce17c35c59..0fcd1cdbdc 100644
@@ -61,10 +61,6 @@ ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES = $(sort \
 	) \
 	$(ROOTFS_COMMON_FINAL_RECURSIVE_DEPENDENCIES__X))
 
-rootfs-common-show-dependency-tree: $(patsubst %,%-show-dependency-tree,$(ROOTFS_COMMON_DEPENDENCIES))
-	$(info rootfs-common: host)
-	$(info rootfs-common -> $(foreach d,$(ROOTFS_COMMON_DEPENDENCIES),$(d)))
-
 .PHONY: rootfs-common
 rootfs-common: $(ROOTFS_COMMON_DEPENDENCIES) target-finalize
 	@$(call MESSAGE,"Generating root filesystems common tables")
@@ -117,10 +113,6 @@ ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES = $$(sort \
 	) \
 	$$(ROOTFS_$(2)_FINAL_RECURSIVE_DEPENDENCIES__X))
 
-rootfs-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$(ROOTFS_$(2)_DEPENDENCIES))
-	$$(info rootfs-$(1): host)
-	$$(info rootfs-$(1) -> $$(foreach d,$$(ROOTFS_$(2)_DEPENDENCIES),$$(d)))
-
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
 ROOTFS_$(2)_COMPRESS_EXT = .gz
 ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
package/pkg-generic.mk +0 -6
index 25bc2e527a..e5b75bc7b6 100644
@@ -862,11 +862,6 @@ $(1)-show-build-order: $$(patsubst %,%-show-build-order,$$($(2)_FINAL_ALL_DEPEND
 	@:
 	$$(info $(1))
 
-$(1)-show-dependency-tree: $$(patsubst %,%-show-dependency-tree,$$($(2)_FINAL_ALL_DEPENDENCIES))
-	@:
-	$$(info $(1): $(4) $$(if $$($(2)_IS_VIRTUAL),virtual,$$($(2)_DL_VERSION)))
-	$$(info $(1) -> $$($(2)_FINAL_ALL_DEPENDENCIES))
-
 $(1)-show-info:
 	@:
 	$$(info $$(call clean-json,$$(call show-info,$(2))))
@@ -1101,7 +1096,6 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
 	$(1)-reconfigure \
 	$(1)-reinstall \
 	$(1)-rsync \
-	$(1)-show-dependency-tree \
 	$(1)-show-depends \
 	$(1)-show-info \
 	$(1)-show-version \
Signature · 1 line
2.14.1