core: add show-vars, a json-formatted equivalent to printvars (branch yem/show-vars)

PATCH Applied Yann E. MORIN 3 participants 31 messages Nov 13, 2021 – Dec 31, 2021

Subsystems (2)

Kbuild Arm/Amlogic Meson SoC support

lore

Share & tools

lore

Yann E. MORIN

author

2021-11-13 13:28 UTC #0 lore raw

v2 cover core: add show-vars, a json-formatted equivalent to printvars (branch yem/show-vars)

Hello All!

This series introduces a new top-level command, show-vars, which like printvars, prints all the variables of Buildroot, and like show-info, does so in a JSON-formatted output.

The first 8 patches are preparatory cleanups in the core infras. They were previously posted separately, but their raison d'être is only really obvious with the introduction of this new show-vars.

Then the following 4 patches prepare the existing show-info helpers for more genericity.

The 3 next patches add a few more items to show-info (I'm sneaking them in this series, am I not devious?). There are hints in the commit logs about something that will come in the future, that will make use of those new fields (and now am I not teasing too much?).

Eventually, we introduce show-vars, which is pretty trivial in and off itelf once all the preparatory work has been done...

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (16):
core/show-info: report build_dir relative to CONFIG_DIR package/kodi-*: run pkg-config at build time, not at parse time support/dependencies: avoid spurious warning on print-vars package/infras: do not conditionally set empty variables package/pkg-python: do not set empty variables core/pkg-generic: also list files installed in images/
Makefile: really comment syntax colouring
core/show-info: 'name' only applies to packages support/utils: make-comma-list does just that, not quoting support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b package/pkg-utils: introduce helper to properly json-escape a string package/pkg-utils: generate proper JSON strings where they are emitted core/show-info: report whether a package is overriden core/show-info: also export source and stamp directories in show-info core/show-info: report package build steps and stamp files
Makefile: introduce show-vars, a json-formatted equivalent to printvars

 Makefile                                           | 23 +++++-
 .../kodi-screensaver-asterwave.mk                  |  4 +-
 .../kodi-screensaver-cpblobs.mk                    |  4 +-
 .../kodi-screensaver-matrixtrails.mk               |  4 +-
 package/pkg-autotools.mk                           |  4 -
 package/pkg-cmake.mk                               |  4 -
 package/pkg-generic.mk                             | 29 +------
 package/pkg-kconfig.mk                             |  3 -
 package/pkg-meson.mk                               |  4 -
 package/pkg-python.mk                              |  7 --
 package/pkg-qmake.mk                               |  4 -
 package/pkg-utils.mk                               | 90 +++++++++++++++++-----
 package/pkg-waf.mk                                 |  5 --
 support/dependencies/dependencies.mk               |  2 +-
 support/misc/utils.mk                              | 14 +++-
 15 files changed, 111 insertions(+), 90 deletions(-)

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:28 UTC #1 lore raw

PATCH v2 1/16 core/show-info: report build_dir relative to CONFIG_DIR

Currently, the build_dir field is reported relative to $(BASE_DIR), to avoid leaking local paths.

However, BASE_DIR is not a directory that is very convenient: for in-tree builds, it is $(CONFIG_DIR)/output/, while for out-of-tree builds, it is $(CONFIG_DIR). This difference is purely an idiosyncracy of how out-of-tree builds have been implemented in Buildroot, and is not under the control of the user.

What the user is in control of, however, is where the .config file is located. This, really, is the directory we should base relative paths on.

Reported-by Thomas Petazzoni <[email protected]>

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-utils.mk +1 -1

index ae3c7f9da9..b2f0b8cebf 100644
@@ -113,7 +113,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
-	"build_dir": "$(patsubst $(BASE_DIR)/%,%,$($(1)_BUILDDIR))",
+	"build_dir": "$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))",
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:28 UTC #2 lore raw

PATCH v2 2/16 package/kodi-*: run pkg-config at build time, not at parse time

In reply to Yann E. MORIN

When calling 'printvars' on a just-configured tree, the output contains many spurious warnings about pkg-config being not found:

    make[1]: [...]/host/bin/pkg-config: Command not found

This is partly because a few packages call pkg-config at the time the Makefile is parsed (really, right when make evaluates the recipe before executing it).

It is customary, instead, to defer the call to pkg-config to the actual shell running the recipe's commands, like all our other packages do.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Bernd Kuhls <[email protected]>

---
 .../kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk  | 4 ++--
 package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk  | 4 ++--
 .../kodi-screensaver-matrixtrails.mk                          | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

package/kodi-screensaver-asterwave/kodi-screensaver-asterwave.mk +2 -2

index b78646b9a9..21def8ca4c 100644
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_ASTERWAVE_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_ASTERWAVE_DEPENDENCIES = glm kodi
 
 KODI_SCREENSAVER_ASTERWAVE_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))

package/kodi-screensaver-cpblobs/kodi-screensaver-cpblobs.mk +2 -2

index c6dd26a89b..6f462e7d27 100644
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_CPBLOBS_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_CPBLOBS_DEPENDENCIES = glm kodi
 
 KODI_SCREENSAVER_CPBLOBS_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))

package/kodi-screensaver-matrixtrails/kodi-screensaver-matrixtrails.mk +2 -2

index ac4303a719..45d3e46b4a 100644
@@ -11,7 +11,7 @@ KODI_SCREENSAVER_MATRIXTRAILS_LICENSE_FILES = LICENSE.md
 KODI_SCREENSAVER_MATRIXTRAILS_DEPENDENCIES = kodi
 
 KODI_SCREENSAVER_MATRIXTRAILS_CONF_OPTS += \
-	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)" \
-	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)"
+	-DCMAKE_C_FLAGS="$(TARGET_CFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`" \
+	-DCMAKE_CXX_FLAGS="$(TARGET_CXXFLAGS) `$(PKG_CONFIG_HOST_BINARY) --cflags egl`"
 
 $(eval $(cmake-package))

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:29 UTC #3 lore raw

PATCH v2 3/16 support/dependencies: avoid spurious warning on print-vars

In reply to Yann E. MORIN

When calling 'printvars', the 'suitable-host-package' macro is printed (a macro is just a variable like the others, after all, just with some parameters). Because it is printed as a variable, it is missing its parameters, but it still tries to evaluate the $(shell) construct.

This causes spurious warning:

make[1]: support/dependencies/check-host-.sh: Command not found

Only try and call the script if there is actually a tool to check for.

Signed-off-by Yann E. MORIN <[email protected]>

---
 support/dependencies/dependencies.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

support/dependencies/dependencies.mk +1 -1

index 4fac5c731b..a7bb1a6080 100644
@@ -15,7 +15,7 @@ else
 # script should use 'which' to find a candidate. The script should return
 # the path to the suitable host tool, or nothing if no suitable tool was found.
 define suitable-host-package
-$(shell support/dependencies/check-host-$(1).sh $(2))
+$(if $(1),$(shell support/dependencies/check-host-$(1).sh $(2)))
 endef
 endif
 # host utilities needs host-tar to extract the source code tarballs, so

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:29 UTC #4 lore raw

PATCH v2 4/16 package/infras: do not conditionally set empty variables

In reply to Yann E. MORIN

Setting an unset variable to an empty value is useless in make; an unset variable just expands to an empty string anyway. So what we do currently has no side effect:

    variable set and not empty   -> variable not modified
    variable set and empty       -> variable not modified
    variable unset               -> set to an empty string

However, additional variables do have an impact on the parsing time of the Makefiles, and the more variables, the more collisions in the hash table used internally by make, which slows down the parsing.

By dropping those conditionally-set-empty variables, we gain about 3%:

    Run     Before  After
    1       5.572   5.325
    2       5.434   5.354
    3       5.490   5.320
    4       5.525   5.330
    5       5.476   5.330
    6       5.511   5.434
    7       5.498   5.388
    8       5.524   5.371
    9       5.479   5.346
    10      5.637   5.324
    Mean:   5.515   5.352

Yeah, 0.163s does not look like much, and this does not make autocompletion any more usable. Still, that 3% gain is not to be ashamed of either.

Note that there are 3 others case where we do set empty variables, but those are unconditional and serve other purposes:

- pkg-virtual: this is done on purpose to avoid a bug when the environment may have TOOLCHAIN_VERSION or _SOURCE set, and we really want those to be empty, so the assignment is not conditional;

- pkg-python: the reason for setting those to empty is dubious at best; it's been there since the inception of the python infra, back in 2013; still, the case is different than this patch addresses;

- pkg-toolchain-external: this is the case for a toolchain already installed, so indeed we want to set _SOURCE and _VERSION to empty.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-autotools.mk |  4 ----
 package/pkg-cmake.mk     |  4 ----
 package/pkg-generic.mk   | 27 ---------------------------
 package/pkg-kconfig.mk   |  3 ---
 package/pkg-meson.mk     |  4 ----
 package/pkg-python.mk    |  4 ----
 package/pkg-qmake.mk     |  4 ----
 package/pkg-waf.mk       |  5 -----
 8 files changed, 55 deletions(-)

package/pkg-autotools.mk +0 -4

index daa688dab6..717ff81593 100644
@@ -160,10 +160,6 @@ ifeq ($(4),host)
  $(2)_AUTORECONF_OPTS ?= $$($(3)_AUTORECONF_OPTS)
 endif
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_OPTS                ?= install
 $(2)_INSTALL_STAGING_OPTS	?= DESTDIR=$$(STAGING_DIR) install
 $(2)_INSTALL_TARGET_OPTS		?= DESTDIR=$$(TARGET_DIR) install

package/pkg-cmake.mk +0 -4

index 4ee100a0c6..3b1db35fb6 100644
@@ -51,11 +51,7 @@ endif
 
 define inner-cmake-package
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
 $(2)_MAKE			?= $$(MAKE)
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_OPTS		?= install
 $(2)_INSTALL_STAGING_OPTS	?= DESTDIR=$$(STAGING_DIR) install/fast
 $(2)_INSTALL_TARGET_OPTS	?= DESTDIR=$$(TARGET_DIR) install/fast

package/pkg-generic.mk +0 ~~-27~~

index ded5176428..ab48240283 100644
@@ -538,8 +538,6 @@ $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASENAME)
 ifndef $(2)_SUBDIR
  ifdef $(3)_SUBDIR
   $(2)_SUBDIR = $$($(3)_SUBDIR)
- else
-  $(2)_SUBDIR ?=
  endif
 endif

@@ -836,31 +834,6 @@ $(2)_EXTRACT_CMDS ?= \
 		$$(TAR_OPTIONS) -)
 
 # pre/post-steps hooks
-$(2)_PRE_DOWNLOAD_HOOKS         ?=
-$(2)_POST_DOWNLOAD_HOOKS        ?=
-$(2)_PRE_EXTRACT_HOOKS          ?=
-$(2)_POST_EXTRACT_HOOKS         ?=
-$(2)_PRE_RSYNC_HOOKS            ?=
-$(2)_POST_RSYNC_HOOKS           ?=
-$(2)_PRE_PATCH_HOOKS            ?=
-$(2)_POST_PATCH_HOOKS           ?=
-$(2)_PRE_CONFIGURE_HOOKS        ?=
-$(2)_POST_CONFIGURE_HOOKS       ?=
-$(2)_PRE_BUILD_HOOKS            ?=
-$(2)_POST_BUILD_HOOKS           ?=
-$(2)_PRE_INSTALL_HOOKS          ?=
-$(2)_POST_INSTALL_HOOKS         ?=
-$(2)_PRE_INSTALL_STAGING_HOOKS  ?=
-$(2)_POST_INSTALL_STAGING_HOOKS ?=
-$(2)_PRE_INSTALL_TARGET_HOOKS   ?=
-$(2)_POST_INSTALL_TARGET_HOOKS  ?=
-$(2)_PRE_INSTALL_IMAGES_HOOKS   ?=
-$(2)_POST_INSTALL_IMAGES_HOOKS  ?=
-$(2)_PRE_LEGAL_INFO_HOOKS       ?=
-$(2)_POST_LEGAL_INFO_HOOKS      ?=
-$(2)_TARGET_FINALIZE_HOOKS      ?=
-$(2)_ROOTFS_PRE_CMD_HOOKS       ?=
-
 $(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)

package/pkg-kconfig.mk +0 -3

index 23a291fccf..32dcfea0bc 100644
@@ -81,9 +81,6 @@ define inner-kconfig-package
 # Default values
 $(2)_MAKE ?= $$(MAKE)
 $(2)_KCONFIG_EDITORS ?= menuconfig
-$(2)_KCONFIG_OPTS ?=
-$(2)_KCONFIG_FIXUP_CMDS ?=
-$(2)_KCONFIG_FRAGMENT_FILES ?=
 $(2)_KCONFIG_DOTCONFIG ?= .config
 $(2)_KCONFIG_SUPPORTS_DEFCONFIG ?= YES

package/pkg-meson.mk +0 -4

index 16a90d157b..ec535174b6 100644
@@ -45,10 +45,6 @@ NINJA_OPTS	= $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
 define inner-meson-package
 
-$(2)_CONF_ENV		?=
-$(2)_CONF_OPTS		?=
-$(2)_NINJA_ENV		?=
-
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file. And take care of the differences between host and target

package/pkg-python.mk +0 -4

index e6b81bdfd3..ce0fc97283 100644
@@ -108,10 +108,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 
 define inner-python-package
 
-$(2)_ENV         ?=
-$(2)_BUILD_OPTS   ?=
-$(2)_INSTALL_OPTS ?=
-
 ifndef $(2)_SETUP_TYPE
  ifdef $(3)_SETUP_TYPE
   $(2)_SETUP_TYPE = $$($(3)_SETUP_TYPE)

package/pkg-qmake.mk +0 -4

index deb033c1d6..4dcaf6dd62 100644
@@ -40,10 +40,6 @@ endef
 
 define inner-qmake-package
 
-$(2)_CONF_ENV			?=
-$(2)_CONF_OPTS			?=
-$(2)_MAKE_ENV			?=
-$(2)_MAKE_OPTS			?=
 $(2)_INSTALL_STAGING_OPTS	?= install
 $(2)_INSTALL_TARGET_OPTS	?= $$($(2)_INSTALL_STAGING_OPTS)

package/pkg-waf.mk +0 -5

index 038d89454e..a593476a69 100644
@@ -49,11 +49,6 @@ else
 $(2)_WAF ?= ./waf
 endif
 
-$(2)_BUILD_OPTS				?=
-$(2)_INSTALL_STAGING_OPTS		?=
-$(2)_INSTALL_TARGET_OPTS		?=
-$(2)_WAF_OPTS				?=
-
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file.

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:43 UTC #5 lore raw

Re: [Buildroot] [PATCH 04/16 v2] package/infras: do not conditionally set empty variables

Quoted 48 lines · On Sat, 13 Nov 2021 14:28:15 +0100 "Yann E. MORIN" <[email protected]>…

Setting an unset variable to an empty value is useless in make; an unset variable just expands to an empty string anyway. So what we do currently has no side effect:
    variable set and not empty   -> variable not modified
    variable set and empty       -> variable not modified
    variable unset               -> set to an empty string
However, additional variables do have an impact on the parsing time of the Makefiles, and the more variables, the more collisions in the hash table used internally by make, which slows down the parsing.
By dropping those conditionally-set-empty variables, we gain about 3%:
    Run     Before  After
    1       5.572   5.325
    2       5.434   5.354
    3       5.490   5.320
    4       5.525   5.330
    5       5.476   5.330
    6       5.511   5.434
    7       5.498   5.388
    8       5.524   5.371
    9       5.479   5.346
    10      5.637   5.324
    Mean:   5.515   5.352
Yeah, 0.163s does not look like much, and this does not make autocompletion any more usable. Still, that 3% gain is not to be ashamed of either.
Note that there are 3 others case where we do set empty variables, but those are unconditional and serve other purposes:
- pkg-virtual: this is done on purpose to avoid a bug when the environment may have TOOLCHAIN_VERSION or _SOURCE set, and we really want those to be empty, so the assignment is not conditional;
- pkg-python: the reason for setting those to empty is dubious at best; it's been there since the inception of the python infra, back in 2013; still, the case is different than this patch addresses;
- pkg-toolchain-external: this is the case for a toolchain already installed, so indeed we want to set _SOURCE and _VERSION to empty.
Signed-off-by Yann E. MORIN <[email protected]>
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Quoted 10 lines

---
 package/pkg-autotools.mk |  4 ----
 package/pkg-cmake.mk     |  4 ----
 package/pkg-generic.mk   | 27 ---------------------------
 package/pkg-kconfig.mk   |  3 ---
 package/pkg-meson.mk     |  4 ----
 package/pkg-python.mk    |  4 ----
 package/pkg-qmake.mk     |  4 ----
 package/pkg-waf.mk       |  5 -----
 8 files changed, 55 deletions(-)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:29 UTC #6 lore raw

PATCH v2 5/16 package/pkg-python: do not set empty variables

In reply to Yann E. MORIN

Similarly to what we just did for conditionally-set-empty variables, unconditionally setting empty variables serves no purpose in Makefiles, as unset variables are just expanded as empty...

Even though we have numerous python packages, the speed gain was not measurable, with delta much less than the noise.

Still, for consistency, we just do not set those variables.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-python.mk | 3 ---
 1 file changed, 3 deletions(-)

package/pkg-python.mk +0 -3

index ce0fc97283..4cc416cea0 100644
@@ -127,7 +127,6 @@ $(2)_BASE_INSTALL_STAGING_OPTS = $$(PKG_PYTHON_DISTUTILS_INSTALL_STAGING_OPTS)
 else
 $(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_DISTUTILS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_OPTS = $$(HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS)
 endif
 # Setuptools

@@ -135,13 +134,11 @@ else ifeq ($$($(2)_SETUP_TYPE),setuptools)
 ifeq ($(4),target)
 $(2)_BASE_ENV         = $$(PKG_PYTHON_SETUPTOOLS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_TARGET_OPTS  = $$(PKG_PYTHON_SETUPTOOLS_INSTALL_TARGET_OPTS)
 $(2)_BASE_INSTALL_STAGING_OPTS = $$(PKG_PYTHON_SETUPTOOLS_INSTALL_STAGING_OPTS)
 else
 $(2)_BASE_ENV         = $$(HOST_PKG_PYTHON_SETUPTOOLS_ENV)
 $(2)_BASE_BUILD_TGT   = build
-$(2)_BASE_BUILD_OPTS   =
 $(2)_BASE_INSTALL_OPTS = $$(HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS)
 endif
 else

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:43 UTC #7 lore raw

Re: [Buildroot] [PATCH 05/16 v2] package/pkg-python: do not set empty variables

Quoted 7 lines · On Sat, 13 Nov 2021 14:28:16 +0100 "Yann E. MORIN" <[email protected]>…

Similarly to what we just did for conditionally-set-empty variables, unconditionally setting empty variables serves no purpose in Makefiles, as unset variables are just expanded as empty...
Even though we have numerous python packages, the speed gain was not measurable, with delta much less than the noise.

Quoted 8 lines

Still, for consistency, we just do not set those variables.
Signed-off-by Yann E. MORIN <[email protected]>
Cc Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
 package/pkg-python.mk | 3 ---
 1 file changed, 3 deletions(-)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:29 UTC #8 lore raw

PATCH v2 6/16 core/pkg-generic: also list files installed in images/

In reply to Yann E. MORIN

Some packages install nothing in target nor staging, but install images (like the kernel vmlinux, or a bootloader boot blob...)

If we want to appropriately account for the files installed by each package, we also need to take images/ into account.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Herve Codina <[email protected]>

---
Changes v1 -> v2:
- fix typo in commit log (Hervé)

---
 package/pkg-generic.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-generic.mk +2 -0

index ab48240283..5626af1d87 100644
@@ -276,6 +276,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
+	@$(call pkg_size_before,$(BINARIES_DIR),-images)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))

@@ -402,6 +403,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(TARGET_DIR))
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
+	@$(call pkg_size_after,$(BINARIES_DIR),-images)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
 	$(Q)touch $@

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:29 UTC #9 lore raw

PATCH v2 7/16 Makefile: really comment syntax colouring

In reply to Yann E. MORIN

The unmatched escaped single-quote lies in the middle of a few function calls, so they too must be fake-closed to properly fix colour highlighting in some editors.

Signed-off-by Yann E. MORIN <[email protected]>

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Makefile +1 -1

index 4062883cc2..6f6bb72de3 100644
@@ -1068,7 +1068,7 @@ printvars:
 		$(if $(QUOTED_VARS),\
 			$(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \
 			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
-# ' Syntax colouring...
+# ')))) # Syntax colouring...
 
 .PHONY: clean
 clean:

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:30 UTC #10 lore raw

PATCH v2 8/16 core/show-info: 'name' only applies to packages

In reply to Yann E. MORIN

Commit 0cfa1659487f (package/pkg-utils.mk: introduce "name" field in show-info output) did what it said, but did so in the generic show-info part, thus it was also added to filesystems (rootfs), the other kind of entity that show-info reports on.

Only packages have a "name"; filesystems do not. Instead, they already have an 'image_name'.

Move the 'name' field to the package-related part of show-info.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---
 package/pkg-utils.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-utils.mk +1 -1

index b2f0b8cebf..973eabe437 100644
@@ -96,7 +96,6 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
-		"name": "$($(1)_RAWNAME)",
 		"type": "$($(1)_TYPE)",
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \

@@ -108,6 +107,7 @@ endef
 # _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
 # for json-info, above
 define _json-info-pkg
+	"name": "$($(1)_RAWNAME)",
 	$(if $($(1)_IS_VIRTUAL), \
 		"virtual": true$(comma),
 		"virtual": false$(comma)

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Peter Korsgaard

2021-12-12 22:18 UTC #11 lore raw

Re: [Buildroot] [PATCH 08/16 v2] core/show-info: 'name' only applies to packages

Quoted 7 lines

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Yann" == Yann E MORIN <[email protected]> writes:
Commit 0cfa1659487f (package/pkg-utils.mk: introduce "name" field in show-info output) did what it said, but did so in the generic show-info part, thus it was also added to filesystems (rootfs), the other kind of entity that show-info reports on.

Quoted 7 lines

Only packages have a "name"; filesystems do not. Instead, they already have an 'image_name'.
Move the 'name' field to the package-related part of show-info.
Signed-off-by Yann E. MORIN <[email protected]>
Cc Thomas Petazzoni <[email protected]>

Committed to 2021.02.x and 2021.08.x, thanks.

Signature · 5 lines

Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:30 UTC #12 lore raw

PATCH v2 9/16 support/utils: make-comma-list does just that, not quoting

In reply to Yann E. MORIN

Currently, we have two functions that build a comma-separated list of items; one is double-quoting the items, while the other is single-quoting them. Their naming is not very consistent.

Besides, in a followup change, we will need to build a comma-separated list of items that are already double-quoted.

Introduce a macro that does just build a comma-separated list, and use that in the two other macros; rename the existing macro so the naming is consistent.

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk  | 10 +++++-----
 support/misc/utils.mk | 12 ++++++++----
 2 files changed, 13 insertions(+), 9 deletions(-)

package/pkg-utils.mk +5 -5

index 973eabe437..94e806963b 100644
@@ -120,17 +120,17 @@ define _json-info-pkg
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
 	"dependencies": [
-		$(call make-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
 	],
 	"reverse_dependencies": [
-		$(call make-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_RDEPENDENCIES)))
 	]
 	$(if $($(1)_CPE_ID_VALID), \
 		$(comma) "cpe-id": "$($(1)_CPE_ID)" \
 	)
 	$(if $($(1)_IGNORE_CVES),
 		$(comma) "ignore_cves": [
-			$(call make-comma-list,$(sort $($(1)_IGNORE_CVES)))
+			$(call make-dq-comma-list,$(sort $($(1)_IGNORE_CVES)))
 		]
 	)
 endef

@@ -144,7 +144,7 @@ define _json-info-pkg-details
 		{
 			"source": "$(notdir $(dl))",
 			"uris": [
-				$(call make-comma-list,
+				$(call make-dq-comma-list,
 					$(subst \|,|,
 						$(call DOWNLOAD_URIS,$(dl),$(1))
 					)

@@ -161,7 +161,7 @@ define _json-info-fs
 				null \
 			),
 	"dependencies": [
-		$(call make-comma-list,$(sort $($(1)_DEPENDENCIES)))
+		$(call make-dq-comma-list,$(sort $($(1)_DEPENDENCIES)))
 	]
 endef

support/misc/utils.mk +8 -4

index dc60cad979..fdc22a2eef 100644
@@ -84,13 +84,17 @@ finddirclauses = $(call notfirstword,$(patsubst %,-o -path '$(1)/%',$(2)))
 # notfirstword(wordlist): returns all but the first word in wordlist
 notfirstword = $(wordlist 2,$(words $(1)),$(1))
 
-# build a comma-separated list of quoted items, from a space-separated
+# build a comma-separated list of items, from a space-separated
+# list of items:   a b c d  -->  a, b, c, d
+make-comma-list = $(subst $(space),$(comma)$(space),$(strip $(1)))
+
+# build a comma-separated list of double-quoted items, from a space-separated
 # list of unquoted items:   a b c d  -->  "a", "b", "c", "d"
-make-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,"%",$(strip $(1))))
+make-dq-comma-list = $(call make-comma-list,$(patsubst %,"%",$(strip $(1))))
 
-# build a comma-separated list of single quoted items, from a space-separated
+# build a comma-separated list of single-quoted items, from a space-separated
 # list of unquoted items:   a b c d  -->  'a', 'b', 'c', 'd'
-make-sq-comma-list = $(subst $(space),$(comma)$(space),$(patsubst %,'%',$(strip $(1))))
+make-sq-comma-list = $(call make-comma-list,$(patsubst %,'%',$(strip $(1))))
 
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:43 UTC #13 lore raw

Re: [Buildroot] [PATCH 09/16 v2] support/utils: make-comma-list does just that, not quoting

Quoted 11 lines · On Sat, 13 Nov 2021 14:28:20 +0100 "Yann E. MORIN" <[email protected]>…

Currently, we have two functions that build a comma-separated list of items; one is double-quoting the items, while the other is single-quoting them. Their naming is not very consistent.
Besides, in a followup change, we will need to build a comma-separated list of items that are already double-quoted.
Introduce a macro that does just build a comma-separated list, and use that in the two other macros; rename the existing macro so the naming is consistent.

Quoted 5 lines

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk  | 10 +++++-----
 support/misc/utils.mk | 12 ++++++++----
 2 files changed, 13 insertions(+), 9 deletions(-)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:30 UTC #14 lore raw

PATCH v2 10/16 support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b

In reply to Yann E. MORIN

Signed-off-by Yann E. MORIN <[email protected]>

---
 support/misc/utils.mk | 2 ++
 1 file changed, 2 insertions(+)

support/misc/utils.mk +2 -0

index fdc22a2eef..4d0efcced0 100644
@@ -13,6 +13,8 @@ qstrip = $(strip $(subst ",,$(1)))
 comma := ,
 empty :=
 space := $(empty) $(empty)
+tab := $(empty)	$(empty)
+escape := $(shell printf '\x1b')
 
 # make 4.3:
 # https://lwn.net/Articles/810071/

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:44 UTC #15 lore raw

Re: [Buildroot] [PATCH 10/16 v2] support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b

Quoted 4 lines · On Sat, 13 Nov 2021 14:28:21 +0100 "Yann E. MORIN" <[email protected]>…

Signed-off-by Yann E. MORIN <[email protected]>
---
 support/misc/utils.mk | 2 ++
 1 file changed, 2 insertions(+)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:30 UTC #16 lore raw

PATCH v2 11/16 package/pkg-utils: introduce helper to properly json-escape a string

In reply to Yann E. MORIN

In quite a few places, we need to generate string that are proper JSON values or keys.

However, JSON is very strict on what constitute a string, and most JSON parsers (like jq or python3's json module) are very picky when parsing a string; any deviation from the spec is immediately sanctioned by a hard error (jq aborts, python3's json module raise an exception).

Introduce a macro that properly prepares a Makefile value into a valid JSON string:

- backslash '\' must be escaped;

- double-quotes need to be escaped of course, as they are the string

    delimiter in JSON;

- anything in the range [0x00..0x1F] must be escaped; in practice, we only ever need to escape \n, \t, and ESC (we could add more in the future if need be);

- finally, we also escape the space, \x20, so that we can call $(strip) on a JSON blurb (like we do for example do build a comma-separated list, or when we sanitise the JSON) without losing multiple spaces where they make sense.

It would have been nice if we had been able to split the macro on multiple lines, but spaces creep in from everywhere in that case, and getting rid of them is getting quite nasty... We could introduce intermediate macros, but meh...

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

package/pkg-utils.mk +11 -0

index 94e806963b..15e3e14e1b 100644
@@ -176,6 +176,17 @@ clean-json = $(strip \
 	))))) \
 )
 
+# mk-json-str -- escape and double-quote a string to make it a valid json string
+#  - escape \
+#  - escape "
+#  - escape \n
+#  - escape \t
+#  - escape ESC
+#  - escape SPACE (so that we can $(strip) a JSON blurb without squashing multiple spaces)
+# This unfortunately has to be on a single line...
+mk-json-str = "$(subst $(space),\u0020,$(subst $(escape),\u001b,$(subst $(tab),\t,$(subst $(sep),\n,$(subst ",\",$(subst \,\\,$(1)))))))"
+# )))))" # Syntax colouring
+
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 # rsync the contents of per-package directories
 # $1: space-separated list of packages to rsync from

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:44 UTC #17 lore raw

Re: [Buildroot] [PATCH 11/16 v2] package/pkg-utils: introduce helper to properly json-escape a string

Quoted 30 lines · On Sat, 13 Nov 2021 14:28:22 +0100 "Yann E. MORIN" <[email protected]>…

In quite a few places, we need to generate string that are proper JSON values or keys.
However, JSON is very strict on what constitute a string, and most JSON parsers (like jq or python3's json module) are very picky when parsing a string; any deviation from the spec is immediately sanctioned by a hard error (jq aborts, python3's json module raise an exception).
Introduce a macro that properly prepares a Makefile value into a valid JSON string:
- backslash '\' must be escaped;
- double-quotes need to be escaped of course, as they are the string
    delimiter in JSON;
- anything in the range [0x00..0x1F] must be escaped; in practice, we only ever need to escape \n, \t, and ESC (we could add more in the future if need be);
- finally, we also escape the space, \x20, so that we can call $(strip) on a JSON blurb (like we do for example do build a comma-separated list, or when we sanitise the JSON) without losing multiple spaces where they make sense.
It would have been nice if we had been able to split the macro on multiple lines, but spaces creep in from everywhere in that case, and getting rid of them is getting quite nasty... We could introduce intermediate macros, but meh...

Quoted 4 lines

Signed-off-by Yann E. MORIN <[email protected]>
---
 package/pkg-utils.mk | 11 +++++++++++
 1 file changed, 11 insertions(+)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:30 UTC #18 lore raw

PATCH v2 12/16 package/pkg-utils: generate proper JSON strings where they are emitted

In reply to Yann E. MORIN

Currently, our clean-json macro does two things:
- remove excessive commas before a closing list or dictionary;
- escape the backslash.

We are going to need the comma-drop feature of clean-json, but on a JSON blurb where all the characters, backslash included, are already all properly escaped, so that we do not need further escape.

So, we drop the backslash escaping from clean-json, and use the newly introduced mk-json-str helper in every locations where we turn a Makefile variable into a JSON string.

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk | 53 ++++++++++++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 19 deletions(-)

package/pkg-utils.mk +34 ~~-19~~

index 15e3e14e1b..0b0ff77419 100644
@@ -96,7 +96,7 @@ endef
 # $(1): upper-case package or filesystem name
 define json-info
 	"$($(1)_NAME)": {
-		"type": "$($(1)_TYPE)",
+		"type": $(call mk-json-str,$($(1)_TYPE)),
 		$(if $(filter rootfs,$($(1)_TYPE)), \
 			$(call _json-info-fs,$(1)), \
 			$(call _json-info-pkg,$(1)), \

@@ -107,47 +107,59 @@ endef
 # _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
 # for json-info, above
 define _json-info-pkg
-	"name": "$($(1)_RAWNAME)",
+	"name": $(call mk-json-str,$($(1)_RAWNAME)),
 	$(if $($(1)_IS_VIRTUAL), \
 		"virtual": true$(comma),
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
-	"build_dir": "$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))",
+	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
 	"dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	],
 	"reverse_dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_RDEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_RDEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	]
 	$(if $($(1)_CPE_ID_VALID), \
-		$(comma) "cpe-id": "$($(1)_CPE_ID)" \
+		$(comma) "cpe-id": $(call mk-json-str,$($(1)_CPE_ID)) \
 	)
 	$(if $($(1)_IGNORE_CVES),
 		$(comma) "ignore_cves": [
-			$(call make-dq-comma-list,$(sort $($(1)_IGNORE_CVES)))
+			$(call make-comma-list, \
+				$(foreach cve,$(sort $($(1)_IGNORE_CVES)), \
+					$(call mk-json-str,$(cve)) \
+				) \
+			)
 		]
 	)
 endef
 
 define _json-info-pkg-details
-	"version": "$($(1)_DL_VERSION)",
-	"licenses": "$($(1)_LICENSE)",
-	"dl_dir": "$($(1)_DL_SUBDIR)",
+	"version": $(call mk-json-str,$($(1)_DL_VERSION)),
+	"licenses": $(call mk-json-str,$($(1)_LICENSE)),
+	"dl_dir": $(call mk-json-str,$($(1)_DL_SUBDIR)),
 	"downloads": [
 	$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
 		{
-			"source": "$(notdir $(dl))",
+			"source": $(call mk-json-str,$(notdir $(dl))),
 			"uris": [
-				$(call make-dq-comma-list,
-					$(subst \|,|,
-						$(call DOWNLOAD_URIS,$(dl),$(1))
-					)
+				$(call make-comma-list, \
+					$(foreach uri,$(call DOWNLOAD_URIS,$(dl),$(1)), \
+						$(call mk-json-str,$(subst \|,|,$(uri))) \
+					) \
 				)
 			]
 		},

@@ -157,11 +169,15 @@ endef
 
 define _json-info-fs
 	"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
-				"$($(1)_FINAL_IMAGE_NAME)", \
+				$(call mk-json-str,$($(1)_FINAL_IMAGE_NAME)), \
 				null \
 			),
 	"dependencies": [
-		$(call make-dq-comma-list,$(sort $($(1)_DEPENDENCIES)))
+		$(call make-comma-list, \
+			$(foreach dep,$(sort $($(1)_DEPENDENCIES)), \
+				$(call mk-json-str,$(dep)) \
+			) \
+		)
 	]
 endef

@@ -171,9 +187,8 @@ endef
 clean-json = $(strip \
 	$(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \
 	$(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
-	$(subst \,\\, \
 		$(strip $(1)) \
-	))))) \
+	)))) \
 )
 
 # mk-json-str -- escape and double-quote a string to make it a valid json string

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:44 UTC #19 lore raw

Re: [Buildroot] [PATCH 12/16 v2] package/pkg-utils: generate proper JSON strings where they are emitted

Quoted 8 lines · On Sat, 13 Nov 2021 14:28:23 +0100 "Yann E. MORIN" <[email protected]>…

Currently, our clean-json macro does two things:
- remove excessive commas before a closing list or dictionary;
- escape the backslash.
We are going to need the comma-drop feature of clean-json, but on a JSON blurb where all the characters, backslash included, are already all properly escaped, so that we do not need further escape.

Quoted 8 lines

So, we drop the backslash escaping from clean-json, and use the newly introduced mk-json-str helper in every locations where we turn a Makefile variable into a JSON string.
Signed-off-by Yann E. MORIN <[email protected]>
---
 package/pkg-utils.mk | 53 ++++++++++++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 19 deletions(-)

Applied to master, thanks.

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:31 UTC #20 lore raw

PATCH v2 13/16 core/show-info: report whether a package is overriden

In reply to Yann E. MORIN

The show-info output can be used in quite some ways, like to assess the validity of the current configuration, like a production build in a CI would do for example. One such assessment would be to ensure that no package is overriden.

Additionally, when we will (later!) want to check whether we can re-use cached artefacts of a package, we will not want to do so for packages that are overridden (nor would we want to actually cache such a result either).

We do not expose the override location, because this is a purely-local information, and show info should never contain such local information (the output of show-info can be shared). Besides, that information is already known by the user, as they passed it to Buildroot via local.mk to begin with.

Signed-off-by Yann E. MORIN <[email protected]>

Cc Vadim Kochan <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc [email protected]

---
 package/pkg-utils.mk | 1 +
 1 file changed, 1 insertion(+)

package/pkg-utils.mk +1 -0

index 0b0ff77419..da123a6ed2 100644
@@ -113,6 +113,7 @@ define _json-info-pkg
 		"virtual": false$(comma)
 		$(call _json-info-pkg-details,$(1)) \
 	)
+	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
 	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

2021-12-30 20:45 UTC #21 lore raw

Re: [Buildroot] [PATCH 13/16 v2] core/show-info: report whether a package is overriden

Quoted 16 lines · On Sat, 13 Nov 2021 14:28:24 +0100 "Yann E. MORIN" <[email protected]>…

The show-info output can be used in quite some ways, like to assess the validity of the current configuration, like a production build in a CI would do for example. One such assessment would be to ensure that no package is overriden.
Additionally, when we will (later!) want to check whether we can re-use cached artefacts of a package, we will not want to do so for packages that are overridden (nor would we want to actually cache such a result either).
We do not expose the override location, because this is a purely-local information, and show info should never contain such local information (the output of show-info can be shared). Besides, that information is already known by the user, as they passed it to Buildroot via local.mk to begin with.

Quoted 5 lines

Signed-off-by Yann E. MORIN <[email protected]>
Cc Vadim Kochan <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc [email protected]

I did not apply this one, because when you set <pkg>_SITE_METHOD = local, then <pkg>_OVERRIDE_SRCDIR will be assigned, but the package isn't really "overridden". It's using locally available source code, as part of the package definition itself.

So I found that advertising this package as "overridden" was not semantically correct.

I'm of course ready to discuss this, and see other arguments :)

Best regards,

Thomas

Signature · 7 lines

Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-12-31 17:28 UTC #22 lore raw

Re: [Buildroot] [PATCH 13/16 v2] core/show-info: report whether a package is overriden

Thomas, All,

On 2021-12-30 21:45 +0100, Thomas Petazzoni spake thusly:

Quoted 6 lines

Quoted 4 lines · On Sat, 13 Nov 2021 14:28:24 +0100 "Yann E. MORIN" <[email protected]>…
The show-info output can be used in quite some ways, like to assess the validity of the current configuration, like a production build in a CI would do for example. One such assessment would be to ensure that no package is overriden.

[--SNIP--]

Quoted 7 lines

I did not apply this one, because when you set <pkg>_SITE_METHOD = local, then <pkg>_OVERRIDE_SRCDIR will be assigned, but the package isn't really "overridden". It's using locally available source code, as part of the package definition itself.
So I found that advertising this package as "overridden" was not semantically correct.

Indeed, I'll see if we can easily discriminate the two, to really report overriden non-local packages.

Regards,
Yann E. MORIN.

Quoted 13 lines

I'm of course ready to discuss this, and see other arguments :)

Best regards,

Thomas

Signature · 7 lines

Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:31 UTC #23 lore raw

PATCH v2 14/16 core/show-info: also export source and stamp directories in show-info

In reply to Yann E. MORIN

External tools may need to peek into the source tree (to check what the list of patches that were applied), or in the stamp directory (to check and report on the progress of a build)

Currently, both locations are identical, but semantically different and an internal implementation detail. Exposing both separately will llow us to change either without breaking users' scripts. Hopefully.

Signed-off-by Yann E. MORIN <[email protected]>

---
 package/pkg-utils.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-utils.mk +2 -0

index da123a6ed2..37a4d244d7 100644
@@ -114,6 +114,8 @@ define _json-info-pkg
 		$(call _json-info-pkg-details,$(1)) \
 	)
 	"overriden": $(if $($(1)_OVERRIDE_SRCDIR),true,false),
+	"stamp_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
+	"source_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
 	"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
 	$(if $(filter target,$($(1)_TYPE)), \
 		"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

2021-12-30 20:45 UTC #24 lore raw

Re: [Buildroot] [PATCH 14/16 v2] core/show-info: also export source and stamp directories in show-info

Quoted 9 lines · On Sat, 13 Nov 2021 14:28:25 +0100 "Yann E. MORIN" <[email protected]>…

External tools may need to peek into the source tree (to check what the list of patches that were applied), or in the stamp directory (to check and report on the progress of a build)
Currently, both locations are identical, but semantically different and an internal implementation detail. Exposing both separately will llow us to change either without breaking users' scripts. Hopefully.
Signed-off-by Yann E. MORIN <[email protected]>

I was not too sure on this one, but applied anyway.

Thomas

Signature · 7 lines

Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:31 UTC #25 lore raw

PATCH v2 15/16 core/show-info: report package build steps and stamp files

In reply to Yann E. MORIN

People (and their scripts!) whi want to report on the progress of the build, will need to know what steps a package follows in what order.

This is handled internally by Buildroot itself already, by use of stamp files.

However, the names of those stamp files are purely an implementation detail (even if they are very unlikely to ever change).

Expose the build steps in a list, which is guaranteed to be ordered, ad for each step, its name and stamp file. The output (once filtered through jq, elided for bevity) will look like:

    [...]
    "install_target": true,
    "install_staging": false,
    "install_images": false,
    "build_steps": [
      {
        "step": "download",
        "stamp_file": ".stamp_downloaded"
      },
      {
        "step": "extract",
        "stamp_file": ".stamp_extracted"
      },
      {
        "step": "patch",
        "stamp_file": ".stamp_patched"
      },
      {
        "step": "configure",
        "stamp_file": ".stamp_configured"
      },
      {
        "step": "build",
        "stamp_file": ".stamp_built"
      },
      {
        "step": "install_target",
        "stamp_file": ".stamp_target_installed"
      }
    ],
    [...]

(Note: the first three fields already existed before this patch.)

Signed-off-by Yann E. MORIN <[email protected]>

Cc Vadim Kochan <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc [email protected]

---
 package/pkg-utils.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

package/pkg-utils.mk +23 -0

index 37a4d244d7..d7cd8f9775 100644
@@ -122,6 +122,29 @@ define _json-info-pkg
 		"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
 		"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
 	)
+	"build_steps": [
+		$(if $($(1)_OVERRIDE_SRCDIR), \
+			{ "step": "rsync"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_RSYNC))) }$(comma) \
+			, \
+			{ "step": "download"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_SOURCE))) }$(comma) \
+			{ "step": "extract"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_EXTRACT))) }$(comma) \
+			{ "step": "patch"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_PATCH))) }$(comma) \
+		)
+		{ "step": "configure", "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_CONFIGURE))) },
+		{ "step": "build", "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_BUILD))) },
+		$(if $(filter YES,$($(1)_INSTALL_STAGING)), \
+			{ "step": "install_staging"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_STAGING))) }$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_TARGET)), \
+			{ "step": "install_target"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_TARGET))) }$(comma) \
+		)
+		$(if $(filter YES,$($(1)_INSTALL_IMAGES)), \
+			{ "step": "install_images"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_IMAGES))) }$(comma) \
+		)
+		$(if $(filter host,$($(1)_TYPE)), \
+			{ "step": "install"$(comma) "stamp_file": $(call mk-json-str,$(notdir $($(1)_TARGET_INSTALL_HOST))) } \
+		)
+	],
 	"dependencies": [
 		$(call make-comma-list, \
 			$(foreach dep,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)), \

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

2021-12-30 20:46 UTC #26 lore raw

Re: [Buildroot] [PATCH 15/16 v2] core/show-info: report package build steps and stamp files

Quoted 45 lines · On Sat, 13 Nov 2021 14:28:26 +0100 "Yann E. MORIN" <[email protected]>…

People (and their scripts!) whi want to report on the progress of the build, will need to know what steps a package follows in what order.
This is handled internally by Buildroot itself already, by use of stamp files.
However, the names of those stamp files are purely an implementation detail (even if they are very unlikely to ever change).
Expose the build steps in a list, which is guaranteed to be ordered, ad for each step, its name and stamp file. The output (once filtered through jq, elided for bevity) will look like:
    [...]
    "install_target": true,
    "install_staging": false,
    "install_images": false,
    "build_steps": [
      {
        "step": "download",
        "stamp_file": ".stamp_downloaded"
      },
      {
        "step": "extract",
        "stamp_file": ".stamp_extracted"
      },
      {
        "step": "patch",
        "stamp_file": ".stamp_patched"
      },
      {
        "step": "configure",
        "stamp_file": ".stamp_configured"
      },
      {
        "step": "build",
        "stamp_file": ".stamp_built"
      },
      {
        "step": "install_target",
        "stamp_file": ".stamp_target_installed"
      }
    ],
    [...]

Quoted 7 lines

(Note: the first three fields already existed before this patch.)
Signed-off-by Yann E. MORIN <[email protected]>
Cc Vadim Kochan <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc [email protected]

This one I really found excessive. There's pretty a 1:1 relationship between the steps and the stamp files, so anything that has knowledge of the internal steps of a package can also know about the stamp files.

As it's more controversial than the rest of the series, I left it aside. I'll for now mark it as Rejected, but I'm also ready to revisit that if need be.

Thomas

Signature · 7 lines

Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-12-31 18:03 UTC #27 lore raw

Re: [Buildroot] [PATCH 15/16 v2] core/show-info: report package build steps and stamp files

Thomas, All,

On 2021-12-30 21:46 +0100, Thomas Petazzoni spake thusly:

Quoted 10 lines

Quoted 8 lines · On Sat, 13 Nov 2021 14:28:26 +0100 "Yann E. MORIN" <[email protected]>…
People (and their scripts!) whi want to report on the progress of the build, will need to know what steps a package follows in what order.
This is handled internally by Buildroot itself already, by use of stamp files.
However, the names of those stamp files are purely an implementation detail (even if they are very unlikely to ever change).

[--SNIP--]

Quoted 3 lines

This one I really found excessive. There's pretty a 1:1 relationship between the steps and the stamp files, so anything that has knowledge of the internal steps of a package can also know about the stamp files.

The thing is, those utilities do not even need to have the knowledge of the internal steps, because the steps are all listed explicitly, in order, with their names and associated stamp-files.

So, such a tool could just do (python-esque):

for step in packages[pkg_name]['build_steps']:
print('{}: {}'.format(step['step'], os.path.exists(step['stamp_file'])))

and they do not even have to concern themselves with whether we add or remove steps, or whether we change the ordering or what.

(Yeah, the key for the step name could be changed from 'step' to 'name'; naming is hard.)

The only nitpick however, is that with TLPB, installation in images/ is not ordered wrt the installation in target/ or staging/ (target/ is always after staging/ even with TLPB).

Regards,
Yann E. MORIN.

Quoted 13 lines

As it's more controversial than the rest of the series, I left it aside. I'll for now mark it as Rejected, but I'm also ready to revisit that if need be.
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-11-13 13:31 UTC #28 lore raw

PATCH v2 16/16 Makefile: introduce show-vars, a json-formatted equivalent to printvars

In reply to Yann E. MORIN

The current printvars output suffers from a serious design flaw:
variables are not delimited, which makes it impossible to reliably retrieve the value of variables; only variables that are known to not contain a \n can be relatively safely extracted.

However, in some cases, it is important to be able to retrieve the multi-line value of a variable, notably the CMDS or the hooks. One such use-case (to follow in an unscheduled future) would be to hash the variables that make up a package "configuration", and cache or extract the files for that package to speed up the build.

Modeled after printvars and show-info, we introduce show-vars (what a lack of imagination here) that outputs a json dictionary which keys are the variable names, and for each variable, provides the raw and expanded values.

Unlike printvars, we do not provide a way to get either the raw or expanded value; both are systematically printed; a user will get just the one is needs. Additionally, strings in JSON are quoted, so there is no need to provide a way to quote variables; that would not make sense.

Note: for printvars, we require that the user provides an explicit pattern to filter variables on. This is historical (see fd5bd12379dc,
Makefile: printvars: don't print anything when VARS is not set). The underlying reasoning was that printvars is too "raw", and variables are not well delimited, so printvars was mostly used to extract a few values here and there, from scripts, or to quickly inspect a specific package's variables during debugging.

But show-vars, although technically plain-text, being JSON, is not very human-readable, and is mostly aimed at tools that will parse it with a real JSON parser, and which will want to have a complete view of a lot of variables at once. As such, and contrary to printvars, it makes sense to report on all variables by default, unless the user explicitly requested a subset.

As a final note: a lot of our variables only make sense in the context of an actual make target. For example, a variable of package foo, that contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is because our CMDS and hooks are expanded as the recipe of a stamp file that lies in the package build directory.

But for show-info, this falls flat on its face: it is not the stamp file of a package, so there is no package directory, and show-info itself has not directory part, so $(@D) expands to '.' (dot).

Additionally, some variables may contain calls to $(shell) (e.g. to call pkg-config), and this also does not work with show-info.

These two issues make it impossible to emit the correct expanded value of variables. To be noted: printvars has the exact same limitations for the exact same reasons.

Signed-off-by Yann E. MORIN <[email protected]>

---
 Makefile | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

Makefile +20 -1

index 6f6bb72de3..0a24f7c5be 100644
@@ -141,7 +141,7 @@ nobuild_targets := source %-source \
 	clean distclean help show-targets graph-depends \
 	%-graph-depends %-show-depends %-show-version \
 	graph-build graph-size list-defconfigs \
-	savedefconfig update-defconfig printvars
+	savedefconfig update-defconfig printvars show-vars
 ifeq ($(MAKECMDGOALS),)
 BR_BUILDING = y
 else ifneq ($(filter-out $(nobuild_targets),$(MAKECMDGOALS)),)

@@ -1058,6 +1058,7 @@ endif
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
 # displayed.
+# show-vars does the same, but as a JSON dictionnary.
 .PHONY: printvars
 printvars:
 	@:

@@ -1070,6 +1071,22 @@ printvars:
 			$(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))
 # ')))) # Syntax colouring...
 
+.PHONY: show-vars
+show-vars: VARS?=%
+show-vars:
+	@:
+	$(info $(call clean-json, { \
+			$(foreach V, \
+				$(sort $(filter $(VARS),$(.VARIABLES))), \
+				$(if $(filter-out environment% default automatic, $(origin $V)), \
+					"$V": { \
+						"expanded": $(call mk-json-str,$($V))$(comma) \
+						"raw": $(call mk-json-str,$(value $V)) \
+					}$(comma) \
+				) \
+			) \
+	} ))
+
 .PHONY: clean
 clean:
 	rm -rf $(BASE_TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) $(HOST_DIR_SYMLINK) \

@@ -1162,6 +1179,8 @@ help:
 	@echo '  pkg-stats              - generate info about packages as JSON and HTML'
 	@echo '  missing-cpe            - generate XML snippets for missing CPE identifiers'
 	@echo '  printvars              - dump internal variables selected with VARS=...'
+	@echo '  show-vars              - dump all internal variables as a JSON blurb; use VARS=...'
+	@echo '                             to limit the list to variables names matching that pattern'
 	@echo
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'

Signature · 6 lines

2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Thomas Petazzoni

Applied

2021-12-30 20:47 UTC #29 lore raw

Re: [Buildroot] [PATCH 16/16 v2] Makefile: introduce show-vars, a json-formatted equivalent to printvars

Quoted 49 lines · On Sat, 13 Nov 2021 14:28:27 +0100 "Yann E. MORIN" <[email protected]>…

The current printvars output suffers from a serious design flaw:
variables are not delimited, which makes it impossible to reliably retrieve the value of variables; only variables that are known to not contain a \n can be relatively safely extracted.
However, in some cases, it is important to be able to retrieve the multi-line value of a variable, notably the CMDS or the hooks. One such use-case (to follow in an unscheduled future) would be to hash the variables that make up a package "configuration", and cache or extract the files for that package to speed up the build.
Modeled after printvars and show-info, we introduce show-vars (what a lack of imagination here) that outputs a json dictionary which keys are the variable names, and for each variable, provides the raw and expanded values.
Unlike printvars, we do not provide a way to get either the raw or expanded value; both are systematically printed; a user will get just the one is needs. Additionally, strings in JSON are quoted, so there is no need to provide a way to quote variables; that would not make sense.
Note: for printvars, we require that the user provides an explicit pattern to filter variables on. This is historical (see fd5bd12379dc,
Makefile: printvars: don't print anything when VARS is not set). The underlying reasoning was that printvars is too "raw", and variables are not well delimited, so printvars was mostly used to extract a few values here and there, from scripts, or to quickly inspect a specific package's variables during debugging.
But show-vars, although technically plain-text, being JSON, is not very human-readable, and is mostly aimed at tools that will parse it with a real JSON parser, and which will want to have a complete view of a lot of variables at once. As such, and contrary to printvars, it makes sense to report on all variables by default, unless the user explicitly requested a subset.
As a final note: a lot of our variables only make sense in the context of an actual make target. For example, a variable of package foo, that contains $(@D)/bar, would expand to .../build/FOO-VERSION/bar. This is because our CMDS and hooks are expanded as the recipe of a stamp file that lies in the package build directory.
But for show-info, this falls flat on its face: it is not the stamp file of a package, so there is no package directory, and show-info itself has not directory part, so $(@D) expands to '.' (dot).
Additionally, some variables may contain calls to $(shell) (e.g. to call pkg-config), and this also does not work with show-info.

Quoted 8 lines

These two issues make it impossible to emit the correct expanded value of variables. To be noted: printvars has the exact same limitations for the exact same reasons.
Signed-off-by Yann E. MORIN <[email protected]>
---
 Makefile | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

Applied to master, thanks.

However, there are two things that are missing:

(1) Update to the Buildroot manual

(2) At least one test case in supporting/testing/

Thanks!

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

Yann E. MORIN

author

2021-12-04 19:13 UTC #30 lore raw

In reply to Yann E. MORIN

All,

On 2021-11-13 14:28 +0100, Yann E. MORIN spake thusly:

Quoted 25 lines

Hello All!
This series introduces a new top-level command, show-vars, which like printvars, prints all the variables of Buildroot, and like show-info, does so in a JSON-formatted output.
The first 8 patches are preparatory cleanups in the core infras. They were previously posted separately, but their raison d'être is only really obvious with the introduction of this new show-vars.
Then the following 4 patches prepare the existing show-info helpers for more genericity.
The 3 next patches add a few more items to show-info (I'm sneaking them in this series, am I not devious?). There are hints in the commit logs about something that will come in the future, that will make use of those new fields (and now am I not teasing too much?).
Eventually, we introduce show-vars, which is pretty trivial in and off itelf once all the preparatory work has been done...
Regards,
Yann E. MORIN.

Quoted 5 lines

----------------------------------------------------------------
Yann E. MORIN (16):
core/show-info: report build_dir relative to CONFIG_DIR package/kodi-*: run pkg-config at build time, not at parse time support/dependencies: avoid spurious warning on print-vars

Those three patches are fixes, so applied to master.

Quoted 2 lines

package/infras: do not conditionally set empty variables package/pkg-python: do not set empty variables

Those two patches, not applied

Quoted 3 lines

core/pkg-generic: also list files installed in images/
Makefile: really comment syntax colouring core/show-info: 'name' only applies to packages

Those three patches are fixes, so applied to master.

Quoted 8 lines

support/utils: make-comma-list does just that, not quoting support/misc/utils: introduce $(tab)=\t and $(escape)=\x1b package/pkg-utils: introduce helper to properly json-escape a string package/pkg-utils: generate proper JSON strings where they are emitted core/show-info: report whether a package is overriden core/show-info: also export source and stamp directories in show-info core/show-info: report package build steps and stamp files
Makefile: introduce show-vars, a json-formatted equivalent to printvars

The rest of the series is not about fixes, so will have to wait for another round.

Regards,
Yann E. MORIN.

Quoted 25 lines

 Makefile                                           | 23 +++++-
 .../kodi-screensaver-asterwave.mk                  |  4 +-
 .../kodi-screensaver-cpblobs.mk                    |  4 +-
 .../kodi-screensaver-matrixtrails.mk               |  4 +-
 package/pkg-autotools.mk                           |  4 -
 package/pkg-cmake.mk                               |  4 -
 package/pkg-generic.mk                             | 29 +------
 package/pkg-kconfig.mk                             |  3 -
 package/pkg-meson.mk                               |  4 -
 package/pkg-python.mk                              |  7 --
 package/pkg-qmake.mk                               |  4 -
 package/pkg-utils.mk                               | 90 +++++++++++++++++-----
 package/pkg-waf.mk                                 |  5 --
 support/dependencies/dependencies.mk               |  2 +-
 support/misc/utils.mk                              | 14 +++-
 15 files changed, 111 insertions(+), 90 deletions(-)

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Signature · 10 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot