Skip to contentSkip to conversation

br2-external: Alow to include toolchain from external tree

RFC New Vadim Kochan 3 participants 10 messages Apr 17, 2019 – Apr 22, 2019 Kbuild

lore
Share & tools
lore
#0 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Add possibility to select toolchain from br2-external tree which allows to easy use custom external toolchains by selecting them via menuconfig as if they were integrated into buildroot.

The br2-external tree should contain:

	${br2-external}/external-toolchain/Config.in

file to be included into external toolchains list.

All such picked toolchains are sourced in:

toolchain/toolchain-external/Config.in

from auto-generated file ${O}/build/.br2-external.in.toolchain.

Added new '-t' option in support/scripts/br2-external to generate kconfig for the found toolchains from the all specified external trees.

---

This is just PoC to rise the discussion about this concept.

 Makefile                               |  6 +++++-
 support/scripts/br2-external           | 36 ++++++++++++++++++++++++++++++++--
 toolchain/toolchain-external/Config.in |  2 ++
 3 files changed, 41 insertions(+), 3 deletions(-)
Makefile +5 -1
index 522c0b0606..4eceb88813 100644
@@ -938,7 +938,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in $(BUILD_DIR)/.br2-external.in.toolchain
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -1042,6 +1042,10 @@ endif
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
+.PHONY: $(BUILD_DIR)/.br2-external.in.toolchain
+$(BUILD_DIR)/.br2-external.in.toolchain: $(BUILD_DIR)
+	$(Q)support/scripts/br2-external -t -o "$(@)" $(BR2_EXTERNAL)
+
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
support/scripts/br2-external +34 -2
index 00cb57d1ed..3ec332d93e 100755
@@ -16,10 +16,11 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
-    while getopts :hkmo: OPT; do
+    while getopts :htkmo: OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
+        t)  ofmt="toolchain";;
         k)  ofmt="kconfig";;
         m)  ofmt="mk";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -30,7 +31,7 @@ main() {
     shift $((OPTIND-1))
 
     case "${ofmt}" in
-    mk|kconfig)
+    mk|kconfig|toolchain)
         ;;
     *)  error "no output format specified (-m/-k)\n";;
     esac
@@ -188,6 +189,37 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
+# Generate the toolchain kconfig snippet for the br2-external tree.
+do_toolchain() {
+    local br2_name br2_ext
+
+    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+    printf '\n'
+
+    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+        printf '# No br2-external tree defined.\n'
+        return
+    fi
+
+    for br2_name in "${BR2_EXT_NAMES[@]}"; do
+        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+
+        if [ ! -f "${br2_ext}/toolchain-external/Config.in" ]; then
+            continue
+	fi
+
+        # we have to duplicate it here too because otherwise BR2_EXTERNAL_*
+	# is not evaluated in Config.in
+        printf 'config BR2_EXTERNAL_%s_PATH\n' "${br2_name}"
+        printf '\tstring\n'
+        printf '\tdefault "%s"\n' "${br2_ext}"
+
+        printf 'source "%s/toolchain-external/Config.in"\n' "${br2_ext}"
+        printf '\n'
+    done
+}
+
 help() {
     cat <<-_EOF_
 	Usage:
toolchain/toolchain-external/Config.in +2 -0
index d234c1c552..70e8a89e5e 100644
@@ -50,6 +50,8 @@ source "toolchain/toolchain-external/toolchain-external-codesourcery-amd64/Confi
 # architecture.
 source "toolchain/toolchain-external/toolchain-external-custom/Config.in"
 
+source "$BR2_BUILD_DIR/.br2-external.in.toolchain"
+
 endchoice
 
 choice
Signature · 1 line
2.14.1
#1 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Quoted 8 lines · Vadim Kochan

Add possibility to select toolchain from br2-external tree which allows to easy use custom external toolchains by selecting them via menuconfig as if they were integrated into buildroot.

The br2-external tree should contain:

	${br2-external}/external-toolchain/Config.in
Quoted 7 lines

file to be included into external toolchains list.

All such picked toolchains are sourced in:

toolchain/toolchain-external/Config.in

from auto-generated file ${O}/build/.br2-external.in.toolchain.

Bikeshedding time: I would call the file .br2-external-toolchain.in.

Quoted 28 lines

Added new '-t' option in support/scripts/br2-external to generate kconfig for the found toolchains from the all specified external trees.

---

This is just PoC to rise the discussion about this concept.

 Makefile                               |  6 +++++-
 support/scripts/br2-external           | 36 ++++++++++++++++++++++++++++++++--
 toolchain/toolchain-external/Config.in |  2 ++
 3 files changed, 41 insertions(+), 3 deletions(-)
Makefile +1 -1
index 522c0b0606..4eceb88813 100644
@@ -938,7 +938,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in $(BUILD_DIR)/.br2-external.in.toolchain
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog

Makefile endif · line 1047

Quoted 6 lines
Makefile +3 -0
@@ -1042,6 +1042,10 @@ endif
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
+.PHONY: $(BUILD_DIR)/.br2-external.in.toolchain
+$(BUILD_DIR)/.br2-external.in.toolchain: $(BUILD_DIR)
+	$(Q)support/scripts/br2-external -t -o "$(@)" $(BR2_EXTERNAL)

I don't like much that it's a separate target. However, it's consistent with
how .br2-external.in is called, so it's OK as it is.

Quoted 42 lines
Makefile +0 -0
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
support/scripts/br2-external +12 -2
index 00cb57d1ed..3ec332d93e 100755
@@ -16,10 +16,11 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
-    while getopts :hkmo: OPT; do
+    while getopts :htkmo: OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
+        t)  ofmt="toolchain";;
         k)  ofmt="kconfig";;
         m)  ofmt="mk";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -30,7 +31,7 @@ main() {
     shift $((OPTIND-1))
 
     case "${ofmt}" in
-    mk|kconfig)
+    mk|kconfig|toolchain)
         ;;
     *)  error "no output format specified (-m/-k)\n";;
     esac
@@ -188,6 +189,37 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
+# Generate the toolchain kconfig snippet for the br2-external tree.
+do_toolchain() {
+    local br2_name br2_ext
+
+    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+    printf '\n'
+
+    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+        printf '# No br2-external tree defined.\n'

support/scripts/br2-external do_kconfig() { · line 210

Quoted 10 lines
support/scripts/br2-external +10 -0
+        return
+    fi
+
+    for br2_name in "${BR2_EXT_NAMES[@]}"; do
+        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+
+        if [ ! -f "${br2_ext}/toolchain-external/Config.in" ]; then
+            continue
+	fi

There's some whitespace confusion here; this file uses spaces only.

support/scripts/br2-external do_kconfig() { · line 213

Quoted 3 lines
support/scripts/br2-external +3 -0
+
+        # we have to duplicate it here too because otherwise BR2_EXTERNAL_*
+	# is not evaluated in Config.in

Argh, that's mightily annoying... But I don't really see a better solution. The
reason is that .br2-external.in only gets included after all the other files have already been included. Normally Kconfig is indepdent of the order in which you declare things, but not when you use a variable in a source directive...

I don't see a good way around it. We could have yet another
.br2-external-variables.in that gets included *before* all the rest, but that's also pretty ugly...

Yann, any ideas?

Assuming there is no better solution than this one, the patch looks pretty much
OK except for the whitespace. I believe it covers all cases. It would be nice if the file would also contain *something* if there are some externals but none of them have a toolchain-external/Config.in, but I don't think it's worth spending time on that.

For the final patch there should also be documentation of course.

Regards,
Arnout

Quoted 26 lines
support/scripts/br2-external +9 -0
+        printf 'config BR2_EXTERNAL_%s_PATH\n' "${br2_name}"
+        printf '\tstring\n'
+        printf '\tdefault "%s"\n' "${br2_ext}"
+
+        printf 'source "%s/toolchain-external/Config.in"\n' "${br2_ext}"
+        printf '\n'
+    done
+}
+
 help() {
     cat <<-_EOF_
 	Usage:
toolchain/toolchain-external/Config.in +2 -0
index d234c1c552..70e8a89e5e 100644
@@ -50,6 +50,8 @@ source "toolchain/toolchain-external/toolchain-external-codesourcery-amd64/Confi
 # architecture.
 source "toolchain/toolchain-external/toolchain-external-custom/Config.in"
 
+source "$BR2_BUILD_DIR/.br2-external.in.toolchain"
+
 endchoice
 
 choice
#2 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Arnout, Vadim, All,

On 2019-04-17 23:26 +0200, Arnout Vandecappelle spake thusly:

Quoted 11 lines · Vadim Kochan
Quoted 10 lines

Add possibility to select toolchain from br2-external tree which allows to easy use custom external toolchains by selecting them via menuconfig as if they were integrated into buildroot.

The br2-external tree should contain:

	${br2-external}/external-toolchain/Config.in

file to be included into external toolchains list.

Quoted 7 lines
Quoted 5 lines

All such picked toolchains are sourced in:

toolchain/toolchain-external/Config.in

from auto-generated file ${O}/build/.br2-external.in.toolchain.

Bikeshedding time: I would call the file .br2-external-toolchain.in.

Ditto.

Quoted 30 lines
Quoted 30 lines

Added new '-t' option in support/scripts/br2-external to generate kconfig for the found toolchains from the all specified external trees.

---

This is just PoC to rise the discussion about this concept.

 Makefile                               |  6 +++++-
 support/scripts/br2-external           | 36 ++++++++++++++++++++++++++++++++--
 toolchain/toolchain-external/Config.in |  2 ++
 3 files changed, 41 insertions(+), 3 deletions(-)
Makefile +1 -1
index 522c0b0606..4eceb88813 100644
@@ -938,7 +938,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in $(BUILD_DIR)/.br2-external.in.toolchain
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -1042,6 +1042,10 @@ endif
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
Quoted 6 lines

Makefile endif · line 1047

Quoted 3 lines
Makefile +3 -0
+.PHONY: $(BUILD_DIR)/.br2-external.in.toolchain
+$(BUILD_DIR)/.br2-external.in.toolchain: $(BUILD_DIR)
+	$(Q)support/scripts/br2-external -t -o "$(@)" $(BR2_EXTERNAL)

I don't like much that it's a separate target. However, it's consistent with
how .br2-external.in is called, so it's OK as it is.

If at all, I would have tried to make it a single target as well.

However, it would require a bit of reachitecting the file, and so maybe it is not worht the effort...

But now, the 'kconfig' -k option is misleading, because it does no longer generate the whole kconfig stuff; it only generates parts of it. But I don't havea better idea either... :-(

Oh, what about removing the '-o' opiton, and requiring each type to expect the output file as arg :

- $(BUILD_DIR)/.br2-external.mk is generated with a call like:

        support/scripts/br2-external -m /path/to/.br2-external.mk $(BR2_EXTERNAL)

- $(BUILD_DIR)/.br2-external.in and $(BUILD_DIR)/.br2-external-toolchain.in are generated with a single call like so:

support/scripts/br2-external -k /path/to/.br2-external.in -t /path/to/.br2-external-toolchain.in $(BR2_EXTERNAL)

Quoted 11 lines
Quoted 11 lines
Makefile +0 -0
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
support/scripts/br2-external +0 -0
index 00cb57d1ed..3ec332d93e 100755
@@ -16,10 +16,11 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
Quoted 6 lines

support/scripts/br2-external main() { · line 23

Quoted 6 lines
support/scripts/br2-external +2 -1
-    while getopts :hkmo: OPT; do
+    while getopts :htkmo: OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
+        t)  ofmt="toolchain";;

Keep options in alphabetical order, except help which goes first.

Quoted 1 line

support/scripts/br2-external main() { · line 24

Quoted 1 line
support/scripts/br2-external +0 -0
         k)  ofmt="kconfig";;

And thus you'd have to be smart here, like so:

    do_kconfig=false
    do_mk=faile
    do_toolchain=false
    while getopts :ht:k:m: OPT; do
        case "${1}" in
        (m)     do_kconfig=true; kconfig_file="${OPTARG}";;
        (k)     do_mk=true; mk_file="${OPTARG}";;
        (t)     do_toolchain=true; toolchain_file="${OPTARG}";;
        esac
    done

This is missing the error case and help, but you get the idea. You could also check that do_toolchain and do_kconfig must be identical (both false or both true) and that it do_mk is incompatible with do_kconfig.

It would then be used as:

    if ${do_kconfig}; then gen_kconfig >"${kconfig_file}"; fi
    if ${do_mk}; then gen_mk >"${mk_file}"; fi
    if ${do_toolchain}; then gen_toolchain >"${toolchain_file}"; fi
Quoted 46 lines
Quoted 24 lines
support/scripts/br2-external +10 -1
         m)  ofmt="mk";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -30,7 +31,7 @@ main() {
     shift $((OPTIND-1))
 
     case "${ofmt}" in
-    mk|kconfig)
+    mk|kconfig|toolchain)
         ;;
     *)  error "no output format specified (-m/-k)\n";;
     esac
@@ -188,6 +189,37 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
+# Generate the toolchain kconfig snippet for the br2-external tree.
+do_toolchain() {
+    local br2_name br2_ext
+
+    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+    printf '\n'
+
+    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+        printf '# No br2-external tree defined.\n'

support/scripts/br2-external do_kconfig() { · line 210

Quoted 10 lines
support/scripts/br2-external +10 -0
+        return
+    fi
+
+    for br2_name in "${BR2_EXT_NAMES[@]}"; do
+        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+
+        if [ ! -f "${br2_ext}/toolchain-external/Config.in" ]; then
+            continue
+	fi

There's some whitespace confusion here; this file uses spaces only.

support/scripts/br2-external do_kconfig() { · line 213

Quoted 3 lines
support/scripts/br2-external +3 -0
+
+        # we have to duplicate it here too because otherwise BR2_EXTERNAL_*
+	# is not evaluated in Config.in

Argh, that's mightily annoying... But I don't really see a better solution. The
reason is that .br2-external.in only gets included after all the other files have already been included. Normally Kconfig is indepdent of the order in which you declare things, but not when you use a variable in a source directive...

Quoted 5 lines

I don't see a good way around it. We could have yet another
.br2-external-variables.in that gets included *before* all the rest, but that's also pretty ugly...

Yann, any ideas?

Gaaahhh... I don't have a very impressive solution either... :-(

Building on my proposal above, what about expanding to:

    -m -> .mk file

    -k -> kconfig variables _only_, included very early, probably in
          Config.in, before line 105 (source "arch/Config.in")

    -t -> toolchains

    -M -> kconfig menu only  (but uppercase 'M' is not nice, since we
          already have lowercase 'm'... maybe switch them?)

If you have a better idea, be my guest and shout it out loud. ;-)

Quoted 7 lines

Assuming there is no better solution than this one, the patch looks pretty much
OK except for the whitespace. I believe it covers all cases. It would be nice if the file would also contain *something* if there are some externals but none of them have a toolchain-external/Config.in, but I don't think it's worth spending time on that.

For the final patch there should also be documentation of course.

Documentation, what's that? ;-)

Yes, the manual should explain that, when this file exist in a br2-external tree, it will be included in the toolchain choice, so it should only contain the boolean options that define the external toolchains.

Regards,
Yann E. MORIN.

Quoted 30 lines

Regards,
Arnout

Quoted 26 lines
support/scripts/br2-external +9 -0
+        printf 'config BR2_EXTERNAL_%s_PATH\n' "${br2_name}"
+        printf '\tstring\n'
+        printf '\tdefault "%s"\n' "${br2_ext}"
+
+        printf 'source "%s/toolchain-external/Config.in"\n' "${br2_ext}"
+        printf '\n'
+    done
+}
+
 help() {
     cat <<-_EOF_
 	Usage:
toolchain/toolchain-external/Config.in +2 -0
index d234c1c552..70e8a89e5e 100644
@@ -50,6 +50,8 @@ source "toolchain/toolchain-external/toolchain-external-codesourcery-amd64/Confi
 # architecture.
 source "toolchain/toolchain-external/toolchain-external-custom/Config.in"
 
+source "$BR2_BUILD_DIR/.br2-external.in.toolchain"
+
 endchoice
 
 choice
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Hi Yann, Arnout, Thomas, All

Quoted 106 lines · Yann E. MORIN

Arnout, Vadim, All,

On 2019-04-17 23:26 +0200, Arnout Vandecappelle spake thusly:

Quoted 11 lines · Vadim Kochan
Quoted 10 lines

Add possibility to select toolchain from br2-external tree which allows to easy use custom external toolchains by selecting them via menuconfig as if they were integrated into buildroot.

The br2-external tree should contain:

	${br2-external}/external-toolchain/Config.in

file to be included into external toolchains list.

Quoted 7 lines
Quoted 5 lines

All such picked toolchains are sourced in:

toolchain/toolchain-external/Config.in

from auto-generated file ${O}/build/.br2-external.in.toolchain.

Bikeshedding time: I would call the file .br2-external-toolchain.in.

Ditto.

Quoted 30 lines
Quoted 30 lines

Added new '-t' option in support/scripts/br2-external to generate kconfig for the found toolchains from the all specified external trees.

---

This is just PoC to rise the discussion about this concept.

 Makefile                               |  6 +++++-
 support/scripts/br2-external           | 36 ++++++++++++++++++++++++++++++++--
 toolchain/toolchain-external/Config.in |  2 ++
 3 files changed, 41 insertions(+), 3 deletions(-)
Makefile +1 -1
index 522c0b0606..4eceb88813 100644
@@ -938,7 +938,7 @@ HOSTCFLAGS = $(CFLAGS_FOR_BUILD)
 export HOSTCFLAGS
 
 .PHONY: prepare-kconfig
-prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in
+prepare-kconfig: outputmakefile $(BUILD_DIR)/.br2-external.in $(BUILD_DIR)/.br2-external.in.toolchain
 
 $(BUILD_DIR)/buildroot-config/%onf:
 	mkdir -p $(@D)/lxdialog
@@ -1042,6 +1042,10 @@ endif
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
Quoted 6 lines

Makefile endif · line 1047

Quoted 3 lines
Makefile +3 -0
+.PHONY: $(BUILD_DIR)/.br2-external.in.toolchain
+$(BUILD_DIR)/.br2-external.in.toolchain: $(BUILD_DIR)
+	$(Q)support/scripts/br2-external -t -o "$(@)" $(BR2_EXTERNAL)

I don't like much that it's a separate target. However, it's consistent with
how .br2-external.in is called, so it's OK as it is.

If at all, I would have tried to make it a single target as well.

However, it would require a bit of reachitecting the file, and so maybe it is not worht the effort...

But now, the 'kconfig' -k option is misleading, because it does no longer generate the whole kconfig stuff; it only generates parts of it. But I don't havea better idea either... :-(

Oh, what about removing the '-o' opiton, and requiring each type to expect the output file as arg :

- $(BUILD_DIR)/.br2-external.mk is generated with a call like:

        support/scripts/br2-external -m /path/to/.br2-external.mk $(BR2_EXTERNAL)

- $(BUILD_DIR)/.br2-external.in and $(BUILD_DIR)/.br2-external-toolchain.in are generated with a single call like so:

support/scripts/br2-external -k /path/to/.br2-external.in -t /path/to/.br2-external-toolchain.in $(BR2_EXTERNAL)

Quoted 11 lines
Quoted 11 lines
Makefile +0 -0
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
support/scripts/br2-external +0 -0
index 00cb57d1ed..3ec332d93e 100755
@@ -16,10 +16,11 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
Quoted 6 lines

support/scripts/br2-external main() { · line 23

Quoted 6 lines
support/scripts/br2-external +2 -1
-    while getopts :hkmo: OPT; do
+    while getopts :htkmo: OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
+        t)  ofmt="toolchain";;

Keep options in alphabetical order, except help which goes first.

Quoted 1 line

support/scripts/br2-external main() { · line 24

Quoted 1 line
support/scripts/br2-external +0 -0
         k)  ofmt="kconfig";;

And thus you'd have to be smart here, like so:

Quoted 10 lines
    do_kconfig=false
    do_mk=faile
    do_toolchain=false
    while getopts :ht:k:m: OPT; do
        case "${1}" in
        (m)     do_kconfig=true; kconfig_file="${OPTARG}";;
        (k)     do_mk=true; mk_file="${OPTARG}";;
        (t)     do_toolchain=true; toolchain_file="${OPTARG}";;
        esac
    done

What about to use '-o' (or better '-O' ?) arg for output folder and use default names for output files, but also add possibility to specify a custom file names per each output file (if it is needed actually) ? then the invocation might look like:

for kconfig:

    $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
    $(Q)support/scripts/br2-external -k -o "$(BUILD_DIR)" $(BR2_EXTERNAL)

for mk:

    $(shell support/scripts/br2-external \
	-m -o '$(BASE_DIR)' $(BR2_EXTERNAL))

in this case '-k' still makes sense and it will generate both toolchain and external kconfig files.

The following options might be used for specifying output file names:

-M - *.mk file

-E - main external kconfig

-T - toolchain file name

-C - common kconfig file name

(I really think this is not needed)

I think that it is ok that script aware about default output file names.

Quoted 89 lines

This is missing the error case and help, but you get the idea. You could also check that do_toolchain and do_kconfig must be identical (both false or both true) and that it do_mk is incompatible with do_kconfig.

It would then be used as:

    if ${do_kconfig}; then gen_kconfig >"${kconfig_file}"; fi
    if ${do_mk}; then gen_mk >"${mk_file}"; fi
    if ${do_toolchain}; then gen_toolchain >"${toolchain_file}"; fi
Quoted 46 lines
Quoted 24 lines
support/scripts/br2-external +10 -1
         m)  ofmt="mk";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -30,7 +31,7 @@ main() {
     shift $((OPTIND-1))
 
     case "${ofmt}" in
-    mk|kconfig)
+    mk|kconfig|toolchain)
         ;;
     *)  error "no output format specified (-m/-k)\n";;
     esac
@@ -188,6 +189,37 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
+# Generate the toolchain kconfig snippet for the br2-external tree.
+do_toolchain() {
+    local br2_name br2_ext
+
+    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+    printf '\n'
+
+    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+        printf '# No br2-external tree defined.\n'

support/scripts/br2-external do_kconfig() { · line 210

Quoted 10 lines
support/scripts/br2-external +10 -0
+        return
+    fi
+
+    for br2_name in "${BR2_EXT_NAMES[@]}"; do
+        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+
+        if [ ! -f "${br2_ext}/toolchain-external/Config.in" ]; then
+            continue
+	fi

There's some whitespace confusion here; this file uses spaces only.

support/scripts/br2-external do_kconfig() { · line 213

Quoted 3 lines
support/scripts/br2-external +3 -0
+
+        # we have to duplicate it here too because otherwise BR2_EXTERNAL_*
+	# is not evaluated in Config.in

Argh, that's mightily annoying... But I don't really see a better solution. The
reason is that .br2-external.in only gets included after all the other files have already been included. Normally Kconfig is indepdent of the order in which you declare things, but not when you use a variable in a source directive...

Quoted 5 lines

I don't see a good way around it. We could have yet another
.br2-external-variables.in that gets included *before* all the rest, but that's also pretty ugly...

Yann, any ideas?

Gaaahhh... I don't have a very impressive solution either... :-(

Building on my proposal above, what about expanding to:

    -m -> .mk file

    -k -> kconfig variables _only_, included very early, probably in
          Config.in, before line 105 (source "arch/Config.in")

    -t -> toolchains

    -M -> kconfig menu only  (but uppercase 'M' is not nice, since we
          already have lowercase 'm'... maybe switch them?)

If you have a better idea, be my guest and shout it out loud. ;-)

Quoted 7 lines

Assuming there is no better solution than this one, the patch looks pretty much
OK except for the whitespace. I believe it covers all cases. It would be nice if the file would also contain *something* if there are some externals but none of them have a toolchain-external/Config.in, but I don't think it's worth spending time on that.

For the final patch there should also be documentation of course.

Documentation, what's that? ;-)

Quoted 7 lines

Yes, the manual should explain that, when this file exist in a br2-external tree, it will be included in the toolchain choice, so it should only contain the boolean options that define the external toolchains.

Regards,
Yann E. MORIN.

Regards,
Vadim Kochan

#4 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Quoted 7 lines · Vadim Kochan

Hi Yann, Arnout, Thomas, All

Quoted 4 lines · Yann E. MORIN

Arnout, Vadim, All,

On 2019-04-17 23:26 +0200, Arnout Vandecappelle spake thusly:

Quoted 1 line · Vadim Kochan

[snip]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

Makefile endif · line 1047

Quoted 7 lines
Makefile +3 -0
@@ -1042,6 +1042,10 @@ endif
 $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
 	$(Q)support/scripts/br2-external -k -o "$(@)" $(BR2_EXTERNAL)
 
+.PHONY: $(BUILD_DIR)/.br2-external.in.toolchain
+$(BUILD_DIR)/.br2-external.in.toolchain: $(BUILD_DIR)
+	$(Q)support/scripts/br2-external -t -o "$(@)" $(BR2_EXTERNAL)

I don't like much that it's a separate target. However, it's consistent with
how .br2-external.in is called, so it's OK as it is.

Quoted 4 lines
Quoted 4 lines

If at all, I would have tried to make it a single target as well.

However, it would require a bit of reachitecting the file, and so maybe it is not worht the effort...

Ho ho, I just meant "a single target", not "a single invocation of the
br2-external script"... What I *actually* meant is that it would be more appropriate to have:

prepare-kconfig: outputmakefile
$(Q)support/scripts/br2-external -k -o "$(BUILD_DIR)/.br2-external.in"
$(BR2_EXTERNAL)
$(Q)support/scripts/br2-external -t -o
"$(BUILD_DIR)/.br2-external-toolchain.in" $(BR2_EXTERNAL)

Quoted 8 lines
Quoted 8 lines

But now, the 'kconfig' -k option is misleading, because it does no longer generate the whole kconfig stuff; it only generates parts of it. But I don't havea better idea either... :-(

Oh, what about removing the '-o' opiton, and requiring each type to expect the output file as arg :

Quoted 8 lines
Quoted 8 lines

- $(BUILD_DIR)/.br2-external.mk is generated with a call like:

        support/scripts/br2-external -m /path/to/.br2-external.mk $(BR2_EXTERNAL)

- $(BUILD_DIR)/.br2-external.in and $(BUILD_DIR)/.br2-external-toolchain.in are generated with a single call like so:

support/scripts/br2-external -k /path/to/.br2-external.in -t /path/to/.br2-external-toolchain.in $(BR2_EXTERNAL)

If you're generating things in a single call, you could just as well generate
all of them in the one call, i.e. including the .mk file. the br2-external script already gets called on every invocation of make, so if it generates all files in one shot, we don't need any commands anymore in prepare-kconfig.

However, I don't mind very much how it's done now with several invocations. It
makes the script rather elegant.

Quoted 36 lines
Quoted 25 lines
Quoted 12 lines
Quoted 12 lines
Makefile +0 -0
 # printvars prints all the variables currently defined in our
 # Makefiles. Alternatively, if a non-empty VARS variable is passed,
 # only the variables matching the make pattern passed in VARS are
support/scripts/br2-external +0 -0
index 00cb57d1ed..3ec332d93e 100755
@@ -16,10 +16,11 @@ main() {
     local OPT OPTARG
     local br2_ext ofile ofmt
 
Quoted 6 lines

support/scripts/br2-external main() { · line 23

Quoted 6 lines
support/scripts/br2-external +2 -1
-    while getopts :hkmo: OPT; do
+    while getopts :htkmo: OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
         o)  ofile="${OPTARG}";;
+        t)  ofmt="toolchain";;

Keep options in alphabetical order, except help which goes first.

Quoted 1 line

support/scripts/br2-external main() { · line 24

Quoted 1 line
support/scripts/br2-external +0 -0
         k)  ofmt="kconfig";;

And thus you'd have to be smart here, like so:

Quoted 10 lines
    do_kconfig=false
    do_mk=faile
    do_toolchain=false
    while getopts :ht:k:m: OPT; do
        case "${1}" in
        (m)     do_kconfig=true; kconfig_file="${OPTARG}";;
        (k)     do_mk=true; mk_file="${OPTARG}";;
        (t)     do_toolchain=true; toolchain_file="${OPTARG}";;
        esac
    done
Quoted 3 lines

What about to use '-o' (or better '-O' ?) arg for output folder and use default names for output files, but also add possibility to specify a custom file names per each output file (if it is needed actually) ?

Yes, if they're all done in one shot, it could be -O. No need for a -o in that
case, obviously.

Note that this is a purely internal script (otherwise it would reside in
utils/), so we can arbitrarily change the interface.

Quoted 23 lines

then
the invocation might look like:

for kconfig:

    $(BUILD_DIR)/.br2-external.in: $(BUILD_DIR)
    $(Q)support/scripts/br2-external -k -o "$(BUILD_DIR)" $(BR2_EXTERNAL)

for mk:

    $(shell support/scripts/br2-external \
	-m -o '$(BASE_DIR)' $(BR2_EXTERNAL))

in this case '-k' still makes sense and it will generate both toolchain and external kconfig files.

The following options might be used for specifying output file names:

-M - *.mk file

-E - main external kconfig

Quoted 7 lines

-T - toolchain file name

-C - common kconfig file name

(I really think this is not needed)

I think that it is ok that script aware about default output file names.

+1 to that.

Quoted 68 lines
Quoted 68 lines

This is missing the error case and help, but you get the idea. You could also check that do_toolchain and do_kconfig must be identical (both false or both true) and that it do_mk is incompatible with do_kconfig.

It would then be used as:

    if ${do_kconfig}; then gen_kconfig >"${kconfig_file}"; fi
    if ${do_mk}; then gen_mk >"${mk_file}"; fi
    if ${do_toolchain}; then gen_toolchain >"${toolchain_file}"; fi
Quoted 46 lines
Quoted 24 lines
support/scripts/br2-external +10 -1
         m)  ofmt="mk";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -30,7 +31,7 @@ main() {
     shift $((OPTIND-1))
 
     case "${ofmt}" in
-    mk|kconfig)
+    mk|kconfig|toolchain)
         ;;
     *)  error "no output format specified (-m/-k)\n";;
     esac
@@ -188,6 +189,37 @@ do_kconfig() {
     printf "endmenu # User-provided options\n"
 }
 
+# Generate the toolchain kconfig snippet for the br2-external tree.
+do_toolchain() {
+    local br2_name br2_ext
+
+    printf '#\n# Automatically generated file; DO NOT EDIT.\n#\n'
+    printf '\n'
+
+    if [ ${#BR2_EXT_NAMES[@]} -eq 0 ]; then
+        printf '# No br2-external tree defined.\n'

support/scripts/br2-external do_kconfig() { · line 210

Quoted 10 lines
support/scripts/br2-external +10 -0
+        return
+    fi
+
+    for br2_name in "${BR2_EXT_NAMES[@]}"; do
+        eval br2_desc="\"\${BR2_EXT_DESCS_${br2_name}}\""
+        eval br2_ext="\"\${BR2_EXT_PATHS_${br2_name}}\""
+
+        if [ ! -f "${br2_ext}/toolchain-external/Config.in" ]; then
+            continue
+	fi

There's some whitespace confusion here; this file uses spaces only.

support/scripts/br2-external do_kconfig() { · line 213

Quoted 3 lines
support/scripts/br2-external +3 -0
+
+        # we have to duplicate it here too because otherwise BR2_EXTERNAL_*
+	# is not evaluated in Config.in

Argh, that's mightily annoying... But I don't really see a better solution. The
reason is that .br2-external.in only gets included after all the other files have already been included. Normally Kconfig is indepdent of the order in which you declare things, but not when you use a variable in a source directive...

Quoted 5 lines

I don't see a good way around it. We could have yet another
.br2-external-variables.in that gets included *before* all the rest, but that's also pretty ugly...

Yann, any ideas?

Gaaahhh... I don't have a very impressive solution either... :-(

Building on my proposal above, what about expanding to:

    -m -> .mk file
Quoted 8 lines
Quoted 8 lines
    -k -> kconfig variables _only_, included very early, probably in
          Config.in, before line 105 (source "arch/Config.in")>>
    -t -> toolchains

    -M -> kconfig menu only  (but uppercase 'M' is not nice, since we
          already have lowercase 'm'... maybe switch them?)

If you have a better idea, be my guest and shout it out loud. ;-)

Still feels icky.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Assuming there is no better solution than this one, the patch looks pretty much
OK except for the whitespace. I believe it covers all cases. It would be nice if the file would also contain *something* if there are some externals but none of them have a toolchain-external/Config.in, but I don't think it's worth spending time on that.

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

For the final patch there should also be documentation of course.

Documentation, what's that? ;-)

Yes, the manual should explain that, when this file exist in a br2-external tree, it will be included in the toolchain choice, so it should only contain the boolean options that define the external toolchains.

Actually, you'll need both the choice part *and* the Config.in.options part.
Note that for those, there is no need to do something special to include them "at the right time", because normally they're only blind options so their location doesn't matter. But all this should be explained in the documentation.

Regards,
Arnout

#5 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Vadim, Arnout, All,

Here's my take on the topic (still WIP, but mostly there, needs commit log):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/br2-ext-toolchain

I'll be off until Monday, when I'll write proper commit log before submitting it.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Hi Yann,

Quoted 9 lines · Yann E. MORIN

Vadim, Arnout, All,

Here's my take on the topic (still WIP, but mostly there, needs commit log):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/br2-ext-toolchain

I'll be off until Monday, when I'll write proper commit log before submitting it.

A few comments:

- The rule for internal variables being prefixed by BR_ instead of BR2_ is written down, in some BR developer days report.

- Bikeshed: call the "dir" variable "outputdir" or something that makes it clear it's the output.

- For the removal of prepare-kconfig, maybe mention that it reverts 9429e7b698638399ecfd73aa37545594f253a074

- I think it's worth adding a flag that checks if there is any external toolchain at all, and to put a comment in br2-external.toolchains.in why it is empty.

Regards,
Arnout

#7 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Arnout, All,

On 2019-04-21 09:30 +0200, Arnout Vandecappelle spake thusly:

Quoted 10 lines

Hi Yann,

Quoted 7 lines · Yann E. MORIN

Vadim, Arnout, All,

Here's my take on the topic (still WIP, but mostly there, needs commit log):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/br2-ext-toolchain

Quoted 7 lines
Quoted 2 lines

I'll be off until Monday, when I'll write proper commit log before submitting it.

A few comments:

- The rule for internal variables being prefixed by BR_ instead of BR2_ is written down, in some BR developer days report.

I wouldn't qualify something that is hidden deep down an old dev-days report, a rule that is documented. ;-) But OK, I'll amend the commit log.

Quoted 2 lines

- Bikeshed: call the "dir" variable "outputdir" or something that makes it clear it's the output.

Are you sure? I'd like more feedback on the vriable before I rename it. Can we at least agree that this vairiable is a directory and as such warrants being named something with 'dir' in the name? If we do agree, then I suggest we plan on a meeting where all interested parties can voice their opinion and suggest alternate naming? ;-)

Quoted 2 lines

- For the removal of prepare-kconfig, maybe mention that it reverts 9429e7b698638399ecfd73aa37545594f253a074

I see reverts as a way to state "this was incorrect, so we undo it", and this is definitely not the case here; this is technically not a revert.

Quoted 3 lines

- I think it's worth adding a flag that checks if there is any external toolchain at all, and to put a comment in br2-external.toolchains.in why it is empty.

Can do too.

Thanks for the erarly feedback! :-)

But I have another pain-point to addres here: I'd like the br2-external trees to be able to provide other types of packages for which we have a choice. Besides toolchains, I can directly see libjpeg, where some vendors have an optimised library that benefts from hardware acceleration. So, rather than come up with something ad-hoc for toolchais, I'd prefer a generic solution, like so:

br2-external/provides/toolchain.in
br2-external/provides/libjpeg.in
br2-external/provides/openssl.in

And those would be included in the corresponding choices. My final series will include this solution (which is up for debate, I agree).

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#8 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

Quoted 4 lines · Yann E. MORIN
Quoted 2 lines

- For the removal of prepare-kconfig, maybe mention that it reverts 9429e7b698638399ecfd73aa37545594f253a074

I see reverts as a way to state "this was incorrect, so we undo it", and this is definitely not the case here; this is technically not a revert.

Refering to the original commit makes it easy to see that the only reason why
this was introduced in the first place was for this particular creation of the .br2-external.in file.

Regards,
Arnout

#9 lore raw

RFC 1/1 br2-external: Alow to include toolchain from external tree

In reply to Yann E. MORIN

Hi Yann, Arnout, All

Quoted 7 lines · Yann E. MORIN

Vadim, Arnout, All,

Here's my take on the topic (still WIP, but mostly there, needs commit log):

https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/br2-ext-toolchain

So, it looks like for each "special" package like openssl or jpeg it will be needed such additional support from buildroot ( I mean additional support of new "provides/xxx" from the buildroot core ). I am really think that Kconfig should be tuned for buildroot for such cases.

For example if to allow apply the last default value for the config option by kconfig parser, then it makes possible to provide easy replacements of existing default values (defined by buildroot) from the br2-external, like setting own "_PROVIDES" or other options.

Or another case like to able extend existin choice from br2-external with new option, for example:

	choice in buildroot:
	-------------------

	choice "external-toolchain"

	source "..."

	...
	endchoice

	choice in br2-external:
	----------------------

	choice "external-toolchain"
	source "custom-path.in"
	endchoice

in the above's case the "choice" from br2-external will be appended to the existing one from buildroot.

Yes, you pointed that it is better to have kconfig aligned with Linux kernel's one. But for me it looks like the kernel does not have such cases like buildroot.

I am sorry that I am still boring you with these kconfig things, just can't keep it in my head)

Regards,
Vadim Kochan