Skip to contentSkip to conversation

core/br2-external: fix reporting errors (branch yem/br2-ext-fixes)

PATCH New Yann E. MORIN 3 participants 10 messages May 22, 2020 – Jul 15, 2020

lore
Share & tools
lore
#0 lore raw

cover core/br2-external: fix reporting errors (branch yem/br2-ext-fixes)

Hello All!

Following a few reports on IRC, we discovered an issue with how our br2-external support was reporting errors. Fix that.

Regards,
Yann E. MORIN.

The following changes since commit a00db9f80819982348e64834f76048c2c8381f40

DEVELOPERS: remove python-pycrypto (2020-05-21 17:05:20 +0200)

are available in the git repository at:

git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to f7b374aa76093cdf59fd788d7c8cc39b86415eb6

core/br2-external: report better error messages (2020-05-22 23:39:05 +0200)

----------------------------------------------------------------
Yann E. MORIN (2):
core.br2-external: fix reporting errors core/br2-external: report better error messages

 support/scripts/br2-external | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

When a br2-external tree has an issue, e.g. a missing file, or does not have a name, or the name uses invalid cahrs, we report that condition by setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external, which outputs it on stdout, and checked by the Makefile.

Before d027cd75d09, stdout was explicitly redirected to the generated
.mk file, with exec >"${ofile}" as the Makefile and Kconfig
fragments were generated each with their own call to the script, and o the validation phase would emit the BR2_EXTERNAL_ERROR variable in the Makefile fragment.

But with d027cd75d09, both the Makefile and Kconfig fragmetns were now generated with a single call to the script, and as such the semantics of the scripts changed, and only each of the actual generators, do_mk and do_kconfig, ahd their out put redirected. Which left do_validate with the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

In turn, the stdout of the script would be interpreted by as part of the Makefile. But this does not end up very well when a br2-external tree indeed has an error:

Makefile:184: *** multiple target patterns. Stop.

So we must redirect the output of the validation step to the Makefile fragment.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no error, and thus the fragment is empty.

---
 support/scripts/br2-external | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
support/scripts/br2-external +4 -3
index 171526f8c8..fdea5aa251 100755
@@ -33,9 +33,8 @@ main() {
     # Trap any unexpected error to generate a meaningful error message
     trap "error 'unexpected error while generating ${ofile}\n'" ERR
 
-    do_validate ${@//:/ }
-
     mkdir -p "${outputdir}"
+    do_validate "${outputdir}" ${@//:/ }
     do_mk "${outputdir}"
     do_kconfig "${outputdir}"
 }
@@ -51,7 +50,9 @@ main() {
 # snippet means that there were no error.
 #
 do_validate() {
+    local outputdir="${1}"
     local br2_ext
+    shift
 
     if [ ${#} -eq 0 ]; then
         # No br2-external tree is valid
@@ -60,7 +61,7 @@ do_validate() {
 
     for br2_ext in "${@}"; do
         do_validate_one "${br2_ext}"
-    done
+    done >"${outputdir}/.br2-external.mk"
 }
 
 do_validate_one() {
Signature · 1 line
2.20.1
#2 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

Le 22/05/2020 ? 23:40, Yann E. MORIN a ?crit?:

Quoted 19 lines

When a br2-external tree has an issue, e.g. a missing file, or does not have a name, or the name uses invalid cahrs, we report that condition by setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external, which outputs it on stdout, and checked by the Makefile.

Before d027cd75d09, stdout was explicitly redirected to the generated
.mk file, with exec >"${ofile}" as the Makefile and Kconfig
fragments were generated each with their own call to the script, and o the validation phase would emit the BR2_EXTERNAL_ERROR variable in the Makefile fragment.

But with d027cd75d09, both the Makefile and Kconfig fragmetns were now generated with a single call to the script, and as such the semantics of the scripts changed, and only each of the actual generators, do_mk and do_kconfig, ahd their out put redirected. Which left do_validate with the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

Quoted 5 lines

In turn, the stdout of the script would be interpreted by as part of the Makefile. But this does not end up very well when a br2-external tree indeed has an error:

Makefile:184: *** multiple target patterns. Stop.

When external.desc exist but empty, I get another error (I'm currently on master)

$ echo "" > support/testing/tests/download/br2-external/git-hash/external.desc
$ make BR2_EXTERNAL=$PWD/support/testing/tests/download/br2-external/git-hash/
menuconfig

Config.in:22: can't open file "output/.br2-external.in.paths"

But Indeed, I get the same error when external.desc is missing.

Here is the error message displayed with the series applied:

case: missing external.desc

does not have an 'external.desc'. See
https://buildroot.org/manual.html#br2-external-converting.

case: empty external.desc

does not define the name.

While at it you can use MANUAL_URL in the error message when the external name is missing.

With that fixed(improved :p):

Also, it would be great to have some tests in the testsuite for such cases. It's easy to make a mistake while adding a br2_external tree.

Best regards,
Romain

Quoted 48 lines

So we must redirect the output of the validation step to the Makefile fragment.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no error, and thus the fragment is empty.

---
 support/scripts/br2-external | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
support/scripts/br2-external +4 -3
index 171526f8c8..fdea5aa251 100755
@@ -33,9 +33,8 @@ main() {
     # Trap any unexpected error to generate a meaningful error message
     trap "error 'unexpected error while generating ${ofile}\n'" ERR
 
-    do_validate ${@//:/ }
-
     mkdir -p "${outputdir}"
+    do_validate "${outputdir}" ${@//:/ }
     do_mk "${outputdir}"
     do_kconfig "${outputdir}"
 }
@@ -51,7 +50,9 @@ main() {
 # snippet means that there were no error.
 #
 do_validate() {
+    local outputdir="${1}"
     local br2_ext
+    shift
 
     if [ ${#} -eq 0 ]; then
         # No br2-external tree is valid
@@ -60,7 +61,7 @@ do_validate() {
 
     for br2_ext in "${@}"; do
         do_validate_one "${br2_ext}"
-    done
+    done >"${outputdir}/.br2-external.mk"
 }
 
 do_validate_one() {
#3 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

Romain, All,

On 2020-06-14 17:33 +0200, Romain Naour spake thusly:

Quoted 26 lines

Le 22/05/2020 ? 23:40, Yann E. MORIN a ?crit?:

Quoted 19 lines

When a br2-external tree has an issue, e.g. a missing file, or does not have a name, or the name uses invalid cahrs, we report that condition by setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external, which outputs it on stdout, and checked by the Makefile.

Before d027cd75d09, stdout was explicitly redirected to the generated
.mk file, with exec >"${ofile}" as the Makefile and Kconfig
fragments were generated each with their own call to the script, and o the validation phase would emit the BR2_EXTERNAL_ERROR variable in the Makefile fragment.

But with d027cd75d09, both the Makefile and Kconfig fragmetns were now generated with a single call to the script, and as such the semantics of the scripts changed, and only each of the actual generators, do_mk and do_kconfig, ahd their out put redirected. Which left do_validate with the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

Quoted 5 lines

In turn, the stdout of the script would be interpreted by as part of the Makefile. But this does not end up very well when a br2-external tree indeed has an error:

Makefile:184: *** multiple target patterns. Stop.

Quoted 7 lines

When external.desc exist but empty, I get another error (I'm currently on master)

$ echo "" > support/testing/tests/download/br2-external/git-hash/external.desc
$ make BR2_EXTERNAL=$PWD/support/testing/tests/download/br2-external/git-hash/
menuconfig

Config.in:22: can't open file "output/.br2-external.in.paths"

Ah yeah, this one is also fixed! ;-)

Quoted 10 lines

But Indeed, I get the same error when external.desc is missing.

Here is the error message displayed with the series applied:

case: missing external.desc

does not have an 'external.desc'. See
https://buildroot.org/manual.html#br2-external-converting.

Quoted 6 lines

case: empty external.desc

does not define the name.

While at it you can use MANUAL_URL in the error message when the external name is missing.

I disagree. The first test is for testing that an old br2-external, from before the multi-br2-external support, in which case we want to point the user to the manual for the conversion step.

However, when external.desc exists, this is not a conversion, and the user is already aware of the requirements for external.desc; they just made a mistake and should just refer to the manual.

Quoted 3 lines

With that fixed(improved :p):

Does that still stand with the patch as-is?

Quoted 2 lines

Also, it would be great to have some tests in the testsuite for such cases. It's easy to make a mistake while adding a br2_external tree.

As already shared on IRC, I hae a set of test-cases that I wrote back in the days I was working on multi-br2-external support:

https://git.buildroot.org/~ymorin/git/br2-external-tests/

Feel free to grab those and turn them into actual tests in our testing infra! ;-)

Regards,
Yann E. MORIN.

Quoted 53 lines

Best regards,
Romain

Quoted 48 lines

So we must redirect the output of the validation step to the Makefile fragment.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no error, and thus the fragment is empty.

---
 support/scripts/br2-external | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
support/scripts/br2-external +4 -3
index 171526f8c8..fdea5aa251 100755
@@ -33,9 +33,8 @@ main() {
     # Trap any unexpected error to generate a meaningful error message
     trap "error 'unexpected error while generating ${ofile}\n'" ERR
 
-    do_validate ${@//:/ }
-
     mkdir -p "${outputdir}"
+    do_validate "${outputdir}" ${@//:/ }
     do_mk "${outputdir}"
     do_kconfig "${outputdir}"
 }
@@ -51,7 +50,9 @@ main() {
 # snippet means that there were no error.
 #
 do_validate() {
+    local outputdir="${1}"
     local br2_ext
+    shift
 
     if [ ${#} -eq 0 ]; then
         # No br2-external tree is valid
@@ -60,7 +61,7 @@ do_validate() {
 
     for br2_ext in "${@}"; do
         do_validate_one "${br2_ext}"
-    done
+    done >"${outputdir}/.br2-external.mk"
 }
 
 do_validate_one() {
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

Yann,

Le 14/06/2020 ? 17:55, Yann E. MORIN a ?crit?:

Quoted 64 lines

Romain, All,

On 2020-06-14 17:33 +0200, Romain Naour spake thusly:

Quoted 26 lines

Le 22/05/2020 ? 23:40, Yann E. MORIN a ?crit?:

Quoted 19 lines

When a br2-external tree has an issue, e.g. a missing file, or does not have a name, or the name uses invalid cahrs, we report that condition by setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external, which outputs it on stdout, and checked by the Makefile.

Before d027cd75d09, stdout was explicitly redirected to the generated
.mk file, with exec >"${ofile}" as the Makefile and Kconfig
fragments were generated each with their own call to the script, and o the validation phase would emit the BR2_EXTERNAL_ERROR variable in the Makefile fragment.

But with d027cd75d09, both the Makefile and Kconfig fragmetns were now generated with a single call to the script, and as such the semantics of the scripts changed, and only each of the actual generators, do_mk and do_kconfig, ahd their out put redirected. Which left do_validate with the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

Quoted 5 lines

In turn, the stdout of the script would be interpreted by as part of the Makefile. But this does not end up very well when a br2-external tree indeed has an error:

Makefile:184: *** multiple target patterns. Stop.

Quoted 7 lines

When external.desc exist but empty, I get another error (I'm currently on master)

$ echo "" > support/testing/tests/download/br2-external/git-hash/external.desc
$ make BR2_EXTERNAL=$PWD/support/testing/tests/download/br2-external/git-hash/
menuconfig

Config.in:22: can't open file "output/.br2-external.in.paths"

Ah yeah, this one is also fixed! ;-)

Quoted 10 lines

But Indeed, I get the same error when external.desc is missing.

Here is the error message displayed with the series applied:

case: missing external.desc

does not have an 'external.desc'. See
https://buildroot.org/manual.html#br2-external-converting.

Quoted 6 lines

case: empty external.desc

does not define the name.

While at it you can use MANUAL_URL in the error message when the external name is missing.

I disagree. The first test is for testing that an old br2-external, from before the multi-br2-external support, in which case we want to point the user to the manual for the conversion step.

However, when external.desc exists, this is not a conversion, and the user is already aware of the requirements for external.desc; they just made a mistake and should just refer to the manual.

Quoted 5 lines
Quoted 3 lines

With that fixed(improved :p):

Does that still stand with the patch as-is?

Ok as-is.

Quoted 11 lines
Quoted 3 lines

Also, it would be great to have some tests in the testsuite for such cases. It's easy to make a mistake while adding a br2_external tree.

As already shared on IRC, I hae a set of test-cases that I wrote back in the days I was working on multi-br2-external support:

https://git.buildroot.org/~ymorin/git/br2-external-tests/

Feel free to grab those and turn them into actual tests in our testing infra! ;-)

Best regards,
Romain

Quoted 58 lines

Regards,
Yann E. MORIN.

Quoted 53 lines

Best regards,
Romain

Quoted 48 lines

So we must redirect the output of the validation step to the Makefile fragment.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no error, and thus the fragment is empty.

---
 support/scripts/br2-external | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
support/scripts/br2-external +4 -3
index 171526f8c8..fdea5aa251 100755
@@ -33,9 +33,8 @@ main() {
     # Trap any unexpected error to generate a meaningful error message
     trap "error 'unexpected error while generating ${ofile}\n'" ERR
 
-    do_validate ${@//:/ }
-
     mkdir -p "${outputdir}"
+    do_validate "${outputdir}" ${@//:/ }
     do_mk "${outputdir}"
     do_kconfig "${outputdir}"
 }
@@ -51,7 +50,9 @@ main() {
 # snippet means that there were no error.
 #
 do_validate() {
+    local outputdir="${1}"
     local br2_ext
+    shift
 
     if [ ${#} -eq 0 ]; then
         # No br2-external tree is valid
@@ -60,7 +61,7 @@ do_validate() {
 
     for br2_ext in "${@}"; do
         do_validate_one "${br2_ext}"
-    done
+    done >"${outputdir}/.br2-external.mk"
 }
 
 do_validate_one() {
author Applied
#5 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

Romain, All,

On 2020-06-14 18:47 +0200, Romain Naour spake thusly:

Quoted 2 lines

Le 14/06/2020 ? 17:55, Yann E. MORIN a ?crit?:

Quoted 1 line

On 2020-06-14 17:33 +0200, Romain Naour spake thusly:

[--SNIP--]

Quoted 5 lines
Quoted 4 lines
Quoted 3 lines

With that fixed(improved :p):

Does that still stand with the patch as-is?

Ok as-is.

Applied to master; thanks for the review.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#6 lore raw

PATCH 1/2 core.br2-external: fix reporting errors

In reply to Yann E. MORIN
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yann" == Yann E MORIN <[email protected]> writes:

When a br2-external tree has an issue, e.g. a missing file, or does not have a name, or the name uses invalid cahrs, we report that condition by setting the variable BR2_EXTERNAL_ERROR.

That variable is defined in the script support/scripts/br2-external, which outputs it on stdout, and checked by the Makefile.

Before d027cd75d09, stdout was explicitly redirected to the generated
.mk file, with exec >"${ofile}" as the Makefile and Kconfig
fragments were generated each with their own call to the script, and o the validation phase would emit the BR2_EXTERNAL_ERROR variable in the Makefile fragment.

But with d027cd75d09, both the Makefile and Kconfig fragmetns were now generated with a single call to the script, and as such the semantics of the scripts changed, and only each of the actual generators, do_mk and do_kconfig, ahd their out put redirected. Which left do_validate with the default stdout. Which would emit BR2_EXTERNAL_ERROR on stdout.

In turn, the stdout of the script would be interpreted by as part of the Makefile. But this does not end up very well when a br2-external tree indeed has an error:

Makefile:184: *** multiple target patterns. Stop.

Quoted 8 lines

So we must redirect the output of the validation step to the Makefile fragment.

Note that we don't need to append in do_mk, and we can do an overwrite
redirection: if we go so far as to call do_mk, it means there was no error, and thus the fragment is empty.

Committed to 2020.02.x and 2020.05.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard
#7 lore raw

PATCH 2/2 core/br2-external: report better error messages

In reply to Yann E. MORIN

The error is misleading: it reports that no name was provided, when in fact the external.desc file is missing.

---
 support/scripts/br2-external | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/br2-external +1 -1
index fdea5aa251..ededd2d900 100755
@@ -75,7 +75,7 @@ do_validate_one() {
         error "'%s': permission denied\n" "${br2_ext}"
     fi
     if [ ! -f "${br2_ext}/external.desc" ]; then
-        error "'%s': does not have a name (in 'external.desc'). See %s\n" \
+        error "'%s': does not have an 'external.desc'. See %s\n" \
             "${br2_ext}" "${MANUAL_URL}"
     fi
     br2_name="$(sed -r -e '/^name: +(.*)$/!d; s//\1/' "${br2_ext}/external.desc")"
Signature · 1 line
2.20.1
#8 lore raw

PATCH 2/2 core/br2-external: report better error messages

Le 22/05/2020 ? 23:40, Yann E. MORIN a ?crit?:

Quoted 2 lines

The error is misleading: it reports that no name was provided, when in fact the external.desc file is missing.

Best regards,
Romain

Quoted 20 lines
---
 support/scripts/br2-external | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/scripts/br2-external +1 -1
index fdea5aa251..ededd2d900 100755
@@ -75,7 +75,7 @@ do_validate_one() {
         error "'%s': permission denied\n" "${br2_ext}"
     fi
     if [ ! -f "${br2_ext}/external.desc" ]; then
-        error "'%s': does not have a name (in 'external.desc'). See %s\n" \
+        error "'%s': does not have an 'external.desc'. See %s\n" \
             "${br2_ext}" "${MANUAL_URL}"
     fi
     br2_name="$(sed -r -e '/^name: +(.*)$/!d; s//\1/' "${br2_ext}/external.desc")"