xen: check-endbr.sh fix and improvement
cover xen: check-endbr.sh fix and improvement
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.check-endbr-fix-v1
Hi.
Anthony PERARD (2):
xen: Fix check-endbr with mawk
xen: Introduce $AWK in check-endbr
xen/tools/check-endbr.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
Signature · 1 line
Anthony PERARD
PATCH 1/2 xen: Fix check-endbr with mawk
check-endbr.sh works well with gawk, but fails with mawk. The produced $ALL file is smaller, it is missing 0x$vma_lo on every line. On mawk, int(0x2A) just produce 0, instead of the expected value.
The use of hexadecimal-constant in awk is an optional part of the posix spec, and mawk doesn't seems to implemented.
There is a way to convert an hexadecimal to a number be putting it in a string, and awk as I understand is supposed to use strtod() to convert the string to a number when needed. The expression 'int("0x15") + 21' would produce the expected value in mawk but now gawk won't convert the string to a number unless we use the option "--non-decimal-data".
So let's convert the hexadecimal number before using it in the awk script. The shell as no issue with dealing with hexadecimal-constant so we'll simply use the expression "$(( 0x15 ))" to convert the value before using it in awk.
--- xen/tools/check-endbr.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
xen/tools/check-endbr.sh +1 -1
@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALL # Wait for $VALID to become complete wait
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Hi Anthony,
Quoted 21 lines · Anthony PERARD
check-endbr.shworks well with gawk, but fails with mawk. The produced $ALL file is smaller, it is missing 0x$vma_lo on every line. On mawk, int(0x2A) just produce 0, instead of the expected value.The use of hexadecimal-constant in awk is an optional part of the posix spec, and mawk doesn't seems to implemented.
There is a way to convert an hexadecimal to a number be putting it in a string, and awk as I understand is supposed to use
strtod()to convert the string to a number when needed. The expression 'int("0x15")+ 21' would produce the expected value inmawkbut nowgawkwon't convert the string to a number unless we use the option "--non-decimal-data".So let's convert the hexadecimal number before using it in the awk script. The shell as no issue with dealing with hexadecimal-constant so we'll simply use the expression "$((
0x15))" to convert the value before using it in awk.
Quoted 4 lines
Fixes 4d037425dc ("x86: Build check for embedded endbr64 instructions")Reported-by Luca Fancellu <[email protected]>Reported-by Mathieu Tarral <[email protected]>Signed-off-by Anthony PERARD <[email protected]>
Very nice solution
Cheers
Bertrand
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
In reply to Anthony PERARDQuoted 19 lines · Anthony PERARD
check-endbr.shworks well with gawk, but fails with mawk. The produced $ALL file is smaller, it is missing 0x$vma_lo on every line. On mawk, int(0x2A) just produce 0, instead of the expected value.The use of hexadecimal-constant in awk is an optional part of the posix spec, and mawk doesn't seems to implemented.
There is a way to convert an hexadecimal to a number be putting it in a string, and awk as I understand is supposed to use
strtod()to convert the string to a number when needed. The expression 'int("0x15")+ 21' would produce the expected value inmawkbut nowgawkwon't convert the string to a number unless we use the option "--non-decimal-data".So let's convert the hexadecimal number before using it in the awk script. The shell as no issue with dealing with hexadecimal-constant so we'll simply use the expression "$((
0x15))" to convert the value before using it in awk.
Quoted 4 lines
Fixes 4d037425dc ("x86: Build check for embedded endbr64 instructions")Reported-by Luca Fancellu <[email protected]>Reported-by Mathieu Tarral <[email protected]>Signed-off-by Anthony PERARD <[email protected]>
Thanks for doing this. You beat me to it.
On policy first, we have https://gitlab.com/xen-project/xen/-/issues/26
open for tracking this bug.
We should consider having
Resolves xen-project/xen#26
in our list of tags, so Gitlab can properly cross-reference this fix. (I wonder if Resolves: works...)
https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically
is the full list of patterns available, but I think we want to keep
Fixes: for it's current meaning.
I also want to wait for the patchew CI run to complete because we've got several build environments which have been a fertile source of shell related bugs.
~Andrew
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Quoted 29 lines · Andrew Cooper
Quoted 19 lines · Anthony PERARD
check-endbr.shworks well with gawk, but fails with mawk. The produced $ALL file is smaller, it is missing 0x$vma_lo on every line. On mawk, int(0x2A) just produce 0, instead of the expected value.The use of hexadecimal-constant in awk is an optional part of the posix spec, and mawk doesn't seems to implemented.
There is a way to convert an hexadecimal to a number be putting it in a string, and awk as I understand is supposed to use
strtod()to convert the string to a number when needed. The expression 'int("0x15")+ 21' would produce the expected value inmawkbut nowgawkwon't convert the string to a number unless we use the option "--non-decimal-data".So let's convert the hexadecimal number before using it in the awk script. The shell as no issue with dealing with hexadecimal-constant so we'll simply use the expression "$((
0x15))" to convert the value before using it in awk.Quoted 4 lines
Fixes 4d037425dc ("x86: Build check for embedded endbr64 instructions")Reported-by Luca Fancellu <[email protected]>Reported-by Mathieu Tarral <[email protected]>Signed-off-by Anthony PERARD <[email protected]>Thanks for doing this. You beat me to it.
On policy first, we have https://gitlab.com/xen-project/xen/-/issues/26
open for tracking this bug.
Quoted 6 lines
We should consider having
Resolves xen-project/xen#26
in our list of tags, so Gitlab can properly cross-reference this fix. (I wonder if Resolves: works...)
Yes it does. Gitlab successfully cross-referenced my dev branch ...
Quoted 7 lines
https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically
is the full list of patterns available, but I think we want to keep
Fixes: for it's current meaning.I also want to wait for the patchew CI run to complete
... pushed because patchew failed to pick the series up for some reason.
~Andrew
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Quoted 38 lines · Andrew Cooper
Quoted 29 lines · Andrew Cooper
Quoted 19 lines · Anthony PERARD
check-endbr.shworks well with gawk, but fails with mawk. The produced $ALL file is smaller, it is missing 0x$vma_lo on every line. On mawk, int(0x2A) just produce 0, instead of the expected value.The use of hexadecimal-constant in awk is an optional part of the posix spec, and mawk doesn't seems to implemented.
There is a way to convert an hexadecimal to a number be putting it in a string, and awk as I understand is supposed to use
strtod()to convert the string to a number when needed. The expression 'int("0x15")+ 21' would produce the expected value inmawkbut nowgawkwon't convert the string to a number unless we use the option "--non-decimal-data".So let's convert the hexadecimal number before using it in the awk script. The shell as no issue with dealing with hexadecimal-constant so we'll simply use the expression "$((
0x15))" to convert the value before using it in awk.Quoted 4 lines
Fixes 4d037425dc ("x86: Build check for embedded endbr64 instructions")Reported-by Luca Fancellu <[email protected]>Reported-by Mathieu Tarral <[email protected]>Signed-off-by Anthony PERARD <[email protected]>Thanks for doing this. You beat me to it.
On policy first, we have https://gitlab.com/xen-project/xen/-/issues/26
open for tracking this bug.Quoted 6 lines
We should consider having
Resolves xen-project/xen#26
in our list of tags, so Gitlab can properly cross-reference this fix. (I wonder if Resolves: works...)
Yes it does. Gitlab successfully cross-referenced my dev branch ...
Quoted 7 lines
Quoted 6 lines
https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically
is the full list of patterns available, but I think we want to keep
Fixes: for it's current meaning.I also want to wait for the patchew CI run to complete
... pushed because patchew failed to pick the series up for some reason.
This series is now fully acked/reviewed and ready, and passed CI (well - the bits of CI which aren't broken for other reasons).
Given the lack of objections, I'm going to use this patch alone as an experiment to see how Resolves: works through other bits of our workflow too. Unless someone objects very promptly.
~Andrew
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
In reply to Anthony PERARDxen/tools/check-endbr.sh then · line 84
Quoted 13 lines · Anthony PERARD
xen/tools/check-endbr.sh +1
-1@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALL # Wait for $VALID to become complete wait
I thought I'd found a cunning way to simply this, but alas.
warrants a comment, so I've added this hunk too:
xen/tools/check-endbr.sh +0 -0
@@ -64,6 +64,11 @@ ${OBJDUMP} -j .text $1 -d -w | grep ' endbr64
*$' | cut -f 1 -d ':' > $VALID &
# split the VMA in half so AWK's numeric addition is only working on 32 bit
# numbers, which don't lose precision.
#
+# 4) MAWK doesn't support plain hex constants (an optional part of the POSIX
+# spec), and GAWK and MAWK can't agree on how to work with hex constants +# in a string. Use the shell to convert $vma_lo to decimal before passing
+# to AWK.
+#
eval $(${OBJDUMP} -j .text $1 -h |
$AWK '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n", substr($4, 1, 8), substr($4, 9, 16)}')
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
In reply to Anthony PERARDxen/tools/check-endbr.sh then · line 81
Quoted 8 lines · Anthony PERARD
xen/tools/check-endbr.sh +1
-1@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALL
I'm afraid that's not portable to environments where sizeof(long) < 8. The shell isn't required to use wider than long for the evaluation.
Jan
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Quoted 11 lines · Jan Beulich
xen/tools/check-endbr.sh then · line 81
Quoted 8 lines · Anthony PERARD
xen/tools/check-endbr.sh +1
-1@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALLI'm afraid that's not portable to environments where sizeof(long) < 8. The shell isn't required to use wider than long for the evaluation.
Yuck. This works at the moment in 32 bit builds because:
++ objdump -j .text xen-syms -h
++ awk '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n", substr($4, 1, 8), substr($4, 9, 16)}'
+ eval vma_hi=ffff82d0 vma_lo=40200000
++ vma_hi=ffff82d0
++ vma_lo=40200000
so the top bit isn't set.
And going from an 8/8 split to a 9/7 split doesn't work either because that uses 4 bits and we've only got 2 to play with given .text's 1G alignment.
I know it's disgusting, but how about a BUILD_BUG_ON(XEN_VIRT_START & (1u << 31)) ?
~Andrew
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Quoted 22 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
xen/tools/check-endbr.sh then · line 81
Quoted 8 lines · Anthony PERARD
xen/tools/check-endbr.sh +1
-1@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALLI'm afraid that's not portable to environments where sizeof(long) < 8. The shell isn't required to use wider than long for the evaluation.
Yuck. This works at the moment in 32 bit builds because:
++ objdump -j .text xen-syms -h
++ awk '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n",substr($4, 1, 8),substr($4, 9, 16)}'
+ eval vma_hi=ffff82d0 vma_lo=40200000
++ vma_hi=ffff82d0
++ vma_lo=40200000
Quoted 5 lines
so the top bit isn't set.
And going from an 8/8 split to a 9/7 split doesn't work either because that uses 4 bits and we've only got 2 to play with given .text's 1G alignment.
Why does text alignment matter here? All we care about are offsets into the Xen image. An I guess we aren't really at risk of going beyond 256M in image size ...
Quoted 2 lines
I know it's disgusting, but how about a
BUILD_BUG_ON(XEN_VIRT_START & (1u << 31))?
In the worst case, why not. But that would be an odd restriction on changes to the memory layout (which we've done in the past).
Jan
Re: [XEN PATCH 1/2] xen: Fix check-endbr with mawk
Quoted 22 lines · Jan Beulich
Quoted 21 lines · Andrew Cooper
Quoted 11 lines · Jan Beulich
xen/tools/check-endbr.sh then · line 81
Quoted 8 lines · Anthony PERARD
xen/tools/check-endbr.sh +1
-1@@ -78,7 +78,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int(0x'$vma_lo') + $1}' > $ALL +fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALLI'm afraid that's not portable to environments where sizeof(long) < 8. The shell isn't required to use wider than long for the evaluation.
Yuck. This works at the moment in 32 bit builds because:
++ objdump -j .text xen-syms -h
++ awk '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n",substr($4, 1, 8),substr($4, 9, 16)}'
+ eval vma_hi=ffff82d0 vma_lo=40200000
++ vma_hi=ffff82d0
++ vma_lo=40200000
Quoted 8 lines
Quoted 5 lines
so the top bit isn't set.
And going from an 8/8 split to a 9/7 split doesn't work either because that uses 4 bits and we've only got 2 to play with given .text's 1G alignment.
Why does text alignment matter here? All we care about are offsets into the Xen image. An I guess we aren't really at risk of going beyond 256M in image size ...
Very good point, but it's not even the image size. It's only .text.
Furthermore, we can have the safety check for that in this script too.
Lemme see what I can do.
Quoted 4 lines
Quoted 2 lines
I know it's disgusting, but how about a
BUILD_BUG_ON(XEN_VIRT_START & (1u << 31))?In the worst case, why not. But that would be an odd restriction on changes to the memory layout (which we've done in the past).
I did say disgusting... I'm entirely happy that it doesn't appear to be needed.
~Andrew
PATCH 2/2 xen: Introduce $AWK in check-endbr
In reply to Anthony PERARD--- xen/tools/check-endbr.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
xen/tools/check-endbr.sh +3 -2
@@ -10,6 +10,7 @@ MSG_PFX="${0##*/} ${1##*/}" OBJCOPY="${OBJCOPY:-objcopy}" OBJDUMP="${OBJDUMP:-objdump}" ADDR2LINE="${ADDR2LINE:-addr2line}" +AWK="${AWK:-awk}" D=$(mktemp -d) trap "rm -rf $D" EXIT
@@ -64,7 +65,7 @@ ${OBJDUMP} -j .text $1 -d -w | grep ' endbr64 *$' | cut -f 1 -d ':' > $VALID & # numbers, which don't lose precision. # eval $(${OBJDUMP} -j .text $1 -h | - awk '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n", substr($4, 1, 8), substr($4, 9, 16)}') + $AWK '$2 == ".text" {printf "vma_hi=%s\nvma_lo=%s\n", substr($4, 1, 8), substr($4, 9, 16)}') ${OBJCOPY} -j .text $1 -O binary $TEXT_BIN
@@ -78,7 +79,7 @@ then else grep -aob -e "$(printf '\363\17\36\372')" -e "$(printf '\363\17\36\373')" \ -e "$(printf '\146\17\37\1')" $TEXT_BIN -fi | awk -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALL +fi | $AWK -F':' '{printf "%s%x\n", "'$vma_hi'", int('$((0x$vma_lo))') + $1}' > $ALL # Wait for $VALID to become complete wait
Signature · 1 line
Anthony PERARD
Re: [XEN PATCH 2/2] xen: Introduce $AWK in check-endbr
Quoted 1 line · Anthony PERARD
Signed-off-by Anthony PERARD <[email protected]>
[PATCH 3/2] README: State POSIX compatibility as a requirement for AWK
In reply to Anthony PERARDIn particular, we support FreeBSD and NetBSD build environments, and some Linux build environments use MAWK over GAWK anyway.
---
CC: George Dunlap <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Anthony PERARD <[email protected]>
--- README | 1 + 1 file changed, 1 insertion(+)
README +1 -0
@@ -48,6 +48,7 @@ provided by your OS distributor: - For ARM 64-bit: - GCC 5.1 or later - GNU Binutils 2.24 or later + * POSIX compatible awk * Development install of zlib (e.g., zlib-dev) * Development install of Python 2.6 or later (e.g., python-dev) * Development install of curses (e.g., libncurses-dev)
Signature · 1 line
2.11.0
Jan Beulich
Acked-byRe: [PATCH 3/2] README: State POSIX compatibility as a requirement for AWK
Quoted 4 lines · Andrew Cooper
In particular, we support FreeBSD and NetBSD build environments, and some Linux build environments use MAWK over GAWK anyway.
Signed-off-by Andrew Cooper <[email protected]>