Automatically generate syscalls.h
v5 cover Automatically generate syscalls.h
The problem we have at the moment is that all syscalls input files must be changed by hand each time we need to test a new syscall. The idea is to simplify this process, providing a script that is able to read from kernel sources and to generate all syscalls files we need.
This patch set adds a new command inside the syscalls folder and it refactors the previous regen.sh code.
The new way we have to generate the syscalls.h file is the following:
include/lapi/syscalls/generate_arch.sh path/to/kernel/source
include/lapi/syscalls/generate_syscalls.sh path/to/syscalls.h
Scripts are independent and they can be run separately. generate_syscalls.sh is the equivalent of regen.sh, but its code has been cleaned up.
---
Changes in v5:
- license SPDX inside scripts files
- quote shell variables inside scripts
- some generic code cleanup
- Link to v4: https://lore.kernel.org/r/[email protected]
Changes in v4:
- remove blacklist file
- update syscalls files
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- fix generate_arch.sh to work properly for all architectures
- rename some arch .in files and handle renaming inside
generate_syscalls.sh using the right gcc directives
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- refactor regen.sh
- split syscalls.h and syscalls .in files creation
- updated syscalls .in files
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Andrea Cervesato (3):
Refactor regen.sh script to generate syscalls Add script to generate arch(s) dependant syscalls Delete obsolete strip_syscall.awk file
configure.ac | 2 +-
include/lapi/syscalls/{aarch64.in => arm64.in} | 0
include/lapi/syscalls/generate_arch.sh | 211 +++++++++++++++++++++
include/lapi/syscalls/generate_syscalls.sh | 115 +++++++++++
.../lapi/syscalls/{loongarch.in => loongarch64.in} | 0
include/lapi/syscalls/{mips_n64.in => mips64.in} | 0
.../lapi/syscalls/{mips_n32.in => mips64n32.in} | 0
include/lapi/syscalls/{mips_o32.in => mipso32.in} | 0
include/lapi/syscalls/{hppa.in => parisc.in} | 0
include/lapi/syscalls/regen.sh | 129 -------------
include/lapi/syscalls/strip_syscall.awk | 19 --
.../lapi/syscalls/{order => supported-arch.txt} | 12 +-
12 files changed, 333 insertions(+), 155 deletions(-)---
base-commit: 34e6dd2d233abcebae435d34d793b6a49d08c190
change-id: 20240923-generate_syscalls-780dc2227bdb
Best regards,
Signature · 5 lines
Andrea Cervesato <[email protected]> -- Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v5 1/3 Refactor regen.sh script to generate syscalls
From: Andrea Cervesato <[email protected]>
Rename regen.sh into a more meaningful generate_syscalls.sh name, rename order into a more meaningful supported-syscalls.txt name and rewrite part of the regen.sh script code in order to execute it from anywhere in the filesystem, without need to be in its own folder. The new code is also more clear and concise, using native sh features which are simplifying the code.
---
configure.ac | 2 +-
include/lapi/syscalls/generate_syscalls.sh | 115 ++++++++++++++++++
include/lapi/syscalls/regen.sh | 129 ---------------------
.../lapi/syscalls/{order => supported-arch.txt} | 0
4 files changed, 116 insertions(+), 130 deletions(-)configure.ac +1 -1
@@ -386,7 +386,7 @@ else AC_SUBST([WITH_REALTIME_TESTSUITE],["no"]) fi -AC_CONFIG_COMMANDS([syscalls.h], [cd ${ac_top_srcdir}/include/lapi/syscalls; ./regen.sh; cd - >/dev/null]) +AC_CONFIG_COMMANDS([syscalls.h], [cd ${ac_top_srcdir}/include/lapi/syscalls; ./generate_syscalls.sh ../syscalls.h; cd - >/dev/null]) # custom functions # NOTE: don't create custom functions for simple checks, put them into this file
include/lapi/syscalls/generate_syscalls.sh new file +115 -0
115 lines not on this page · raw message
include/lapi/syscalls/regen.sh deleted +0 -129
129 lines not on this page · raw message
include/lapi/syscalls/supported-arch.txt renamed from include/lapi/syscalls/order +0 -0
Signature · 5 lines
2.43.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
Hi!
Quoted 6 lines
include/lapi/syscalls/generate_syscalls.sh new file +6
-0+( + echo + echo "/* Common stubs */" + while IFS= read -r arch; do + while IFS= read -r line; do + set -- ${line}
include/lapi/syscalls/generate_syscalls.sh line 115
Quoted 10 lines
include/lapi/syscalls/generate_syscalls.sh new file +10
-0+ syscall_nr="__NR_$1" + shift + + echo "# ifndef ${syscall_nr}" + echo "# define ${syscall_nr} __LTP__NR_INVALID_SYSCALL" + echo "# endif" + done <"${SCRIPT_DIR}/${arch}.in" + done <${SUPPORTED_ARCH} + echo "#endif" +) >>${SYSCALLS_FILE}
This will generate multiple fallback definitions, one per architecture for common syscall, there was a good reason why these files were parsed by awk with sort -u
include/lapi/syscalls/regen.sh line 112
Quoted 2 lines
include/lapi/syscalls/regen.sh deleted +0
-2-echo "#define __LTP__NR_INVALID_SYSCALL -1" >> "${output_pid}" -for nr in $(awk '{print $1}' "${srcdir}/"*.in | sort -u) ; do
This is the important part that IMHO should stay.
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
Hi Cyril,
Quoted 12 lines · Cyril Hrubis
Hi!
Quoted 11 lines
include/lapi/syscalls/generate_syscalls.sh new file +11
-0+( + echo + echo "/* Common stubs */" + while IFS= read -r arch; do + while IFS= read -r line; do + set -- ${line} + syscall_nr="__NR_$1" + shift + + echo "# ifndef ${syscall_nr}" + echo "# define ${syscall_nr} __LTP__NR_INVALID_SYSCALL"
Quoted 8 lines
include/lapi/syscalls/generate_syscalls.sh line 115
Quoted 5 lines
include/lapi/syscalls/generate_syscalls.sh new file +5
-0+ echo "# endif" + done <"${SCRIPT_DIR}/${arch}.in" + done <${SUPPORTED_ARCH} + echo "#endif" +) >>${SYSCALLS_FILE}This will generate multiple fallback definitions, one per architecture for common syscall, there was a good reason why these files were parsed by awk with sort -u
I wrote the code a few weeks ago, so I needed to read it again to understand the reason.
If you take a look at the drop_bad_entries() and generate_tables() you will see that sort command is used to remove duplicates and to sort the entire list per architecture.
Or am I missing something?
Quoted 4 lines
include/lapi/syscalls/regen.sh line 112
Quoted 2 lines
include/lapi/syscalls/regen.sh deleted +0
-2-echo "#define __LTP__NR_INVALID_SYSCALL -1" >> "${output_pid}" -for nr in $(awk '{print $1}' "${srcdir}/"*.in | sort -u) ; doThis is the important part that IMHO should stay.
Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
Hi!
Quoted 9 lines
Quoted 3 lines
This will generate multiple fallback definitions, one per architecture for common syscall, there was a good reason why these files were parsed by awk with sort -u
I wrote the code a few weeks ago, so I needed to read it again to understand the reason.
If you take a look at the
drop_bad_entries()andgenerate_tables()you will see thatsortcommand is used to remove duplicates and to sort the entire list per architecture.
No no, this is the code that generates the actuall LTP header and what you do here is:
for each architecture:
for each syscall in architecture:
generate fallback with invalid value
And what you need to do is:
read all syscall names for all architectures and do sort -u then generate fallback with invalid value
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
In reply to Andrea CervesatoHi Andrea,
include/lapi/syscalls/generate_syscalls.sh line 9
Quoted 11 lines
+++
b/include/lapi/syscalls/generate_syscalls.shinclude/lapi/syscalls/generate_syscalls.sh new file +9
-0@@ -0,0 +1,115 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Generate the syscalls.h file, merging all architectures syscalls input file +# which are in the current folder and defined inside supported-arch.txt file. + +SYSCALLS_FILE="$1" + +if [ -z "${SYSCALLS_FILE}" ]; then
nit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} => $SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use ${...} in whole script.
The same applies to include/lapi/syscalls/generate_arch.sh in the second commit.
Quoted 12 lines
include/lapi/syscalls/generate_syscalls.sh new file +12
-0+ echo "Please provide the syscalls.h directory:" + echo "" + echo "$0 path/of/syscalls.h" + echo "" + exit 1 +fi + +SCRIPT_DIR="$(realpath $(dirname "$0"))" +SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt" + +echo ' +/************************************************
include/lapi/syscalls/generate_syscalls.sh line 31
Quoted 10 lines
include/lapi/syscalls/generate_syscalls.sh new file +10
-0+* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE * +* change your arch specific .in file instead * +************************************************/ + +/* +* Here we stick all the ugly *fallback* logic for linux +* system call numbers (those __NR_ thingies). +* +* Licensed under the GPLv2 or later, see the COPYING file. +*/
nit: fixing space in /* */ comments (correct in the original regen.sh)
+ replacing old note about COPYING with SPDX in generated header:
+++ include/lapi/syscalls/generate_syscalls.sh @@ -17,18 +17,16 @@ fi
SCRIPT_DIR="$(realpath $(dirname "$0"))" SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt"
-echo '
+echo '// SPDX-License-Identifier: GPL-2.0-or-later
/************************************************
-* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE * -* change your arch specific .in file instead * -************************************************/
+ * GENERATED FILE: DO NOT EDIT/PATCH THIS FILE *
+ * change your arch specific .in file instead *
+ ************************************************/
/*
-* Here we stick all the ugly *fallback* logic for linux -* system call numbers (those __NR_ thingies).
-*
-* Licensed under the GPLv2 or later, see the COPYING file. -*/
+ * Here we stick all the ugly *fallback* logic for linux
+ * system call numbers (those __NR_ thingies).
+ */
#ifndef LAPI_SYSCALLS_H__
#define LAPI_SYSCALLS_H__
---
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
Hi Petr,
Quoted 17 lines · Petr Vorel
Hi Andrea,
include/lapi/syscalls/generate_syscalls.sh line 9
Quoted 11 lines
+++
b/include/lapi/syscalls/generate_syscalls.shinclude/lapi/syscalls/generate_syscalls.sh new file +9
-0@@ -0,0 +1,115 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Generate the syscalls.h file, merging all architectures syscalls input file +# which are in the current folder and defined inside supported-arch.txt file. + +SYSCALLS_FILE="$1" + +if [ -z "${SYSCALLS_FILE}" ]; thennit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} => $SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use ${...} in whole script.
Quoted 1 line
The same applies to
include/lapi/syscalls/generate_arch.shin the second commit.
This is a bit debatable. The reason why using ${VAR} over $VAR is the readability in the first place (i.e. when you have ${PATH}/bin instead $PATH/bin or when you use characters which could mix with PATH var name and it might create debug issues), but also the fact it provides ${VAR:-default} syntax which can always added afterwards.
I usually prefer this method for these reasons.
Quoted 18 lines
Quoted 18 lines
include/lapi/syscalls/generate_syscalls.sh new file +17
-0+ echo "Please provide the syscalls.h directory:" + echo "" + echo "$0 path/of/syscalls.h" + echo "" + exit 1 +fi + +SCRIPT_DIR="$(realpath $(dirname "$0"))" +SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt" + +echo ' +/************************************************ +* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE * +* change your arch specific .in file instead * +************************************************/ + +/*
Quoted 8 lines
include/lapi/syscalls/generate_syscalls.sh line 31
Quoted 5 lines
include/lapi/syscalls/generate_syscalls.sh new file +5
-0+* Here we stick all the ugly *fallback* logic for linux +* system call numbers (those __NR_ thingies). +* +* Licensed under the GPLv2 or later, see the COPYING file. +*/nit: fixing space in /* */ comments (correct in the original
regen.sh)
+ replacing old note about COPYING with SPDX in generated header:
Thanks you seen that. Now I know I can't fully trust auto-refactoring.
Quoted 30 lines
+++
include/lapi/syscalls/generate_syscalls.sh(quoted diff) +7
-10@@ -17,18 +17,16 @@ fi SCRIPT_DIR="$(realpath $(dirname "$0"))" SUPPORTED_ARCH="${SCRIPT_DIR}/supported-arch.txt" -echo ' +echo '// SPDX-License-Identifier: GPL-2.0-or-later /************************************************ -* GENERATED FILE: DO NOT EDIT/PATCH THIS FILE * -* change your arch specific .in file instead * -************************************************/ + * GENERATED FILE: DO NOT EDIT/PATCH THIS FILE * + * change your arch specific .in file instead * + ************************************************/ /* -* Here we stick all the ugly *fallback* logic for linux -* system call numbers (those __NR_ thingies). -* -* Licensed under the GPLv2 or later, see the COPYING file. -*/ + * Here we stick all the ugly *fallback* logic for linux + * system call numbers (those __NR_ thingies). + */ #ifndef LAPI_SYSCALLS_H__ #define LAPI_SYSCALLS_H__ ---
Quoted 2 lines
Kind regards,
Petr
Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 1/3] Refactor regen.sh script to generate syscalls
Quoted 22 lines
Hi Petr,
Quoted 17 lines · Petr Vorel
Hi Andrea,
include/lapi/syscalls/generate_syscalls.sh line 9
Quoted 11 lines
+++
b/include/lapi/syscalls/generate_syscalls.shinclude/lapi/syscalls/generate_syscalls.sh new file +9
-0@@ -0,0 +1,115 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Generate the syscalls.h file, merging all architectures syscalls input file +# which are in the current folder and defined inside supported-arch.txt file. + +SYSCALLS_FILE="$1" + +if [ -z "${SYSCALLS_FILE}" ]; thennit: I would still replace ${...} with $... (e.g. ${SYSCALLS_FILE} => $SYSCALLS_FILE) as I noted in v4 - readability. IMHO there is no need to use ${...} in whole script.
Quoted 1 line
The same applies to
include/lapi/syscalls/generate_arch.shin the second commit.
Quoted 7 lines
This is a bit debatable. The reason why using ${VAR} over $VAR is the readability in the first place (i.e. when you have
${PATH}/bininstead$PATH/binor when you use characters which could mix with PATH var name and it might create debug issues), but also the fact it provides ${VAR:-default} syntax which can always added afterwards.I usually prefer this method for these reasons.
Sure, it's matter of preference. At least LTP shell library tst_test.sh is happy with $VAR instead of ${VAR}. But as generate_arch.sh origin is in the separate project, and this is a separate shell script (not part of shell API) let's ignore my suggestion.
Kind regards,
Petr
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v5 2/3 Add script to generate arch(s) dependant syscalls
In reply to Andrea CervesatoFrom: Andrea Cervesato <[email protected]>
Add generate_arch.sh script which can be used to generate arch(s) dependant syscalls file. The way it works is pretty simple: for each architecture defined into supported-arch.txt, compile kernel headers, extract the list of syscalls and generate a .in file containing all of them, associated with their own syscall's number.
The way syscalls files are generated, passes through a C application which is automatically checking the availability of the syscalls in the user space environment.
---
include/lapi/syscalls/{aarch64.in => arm64.in} | 0
include/lapi/syscalls/generate_arch.sh | 211 +++++++++++++++++++++
.../lapi/syscalls/{loongarch.in => loongarch64.in} | 0
include/lapi/syscalls/{mips_n64.in => mips64.in} | 0
.../lapi/syscalls/{mips_n32.in => mips64n32.in} | 0
include/lapi/syscalls/{mips_o32.in => mipso32.in} | 0
include/lapi/syscalls/{hppa.in => parisc.in} | 0
include/lapi/syscalls/supported-arch.txt | 12 +-
8 files changed, 217 insertions(+), 6 deletions(-)include/lapi/syscalls/arm64.in renamed from include/lapi/syscalls/aarch64.in +0 -0
include/lapi/syscalls/generate_arch.sh new file +211 -0
211 lines not on this page · raw message
include/lapi/syscalls/loongarch64.in renamed from include/lapi/syscalls/loongarch.in +0 -0
include/lapi/syscalls/mips64.in renamed from include/lapi/syscalls/mips_n64.in +0 -0
include/lapi/syscalls/mips64n32.in renamed from include/lapi/syscalls/mips_n32.in +0 -0
include/lapi/syscalls/mipso32.in renamed from include/lapi/syscalls/mips_o32.in +0 -0
include/lapi/syscalls/parisc.in renamed from include/lapi/syscalls/hppa.in +0 -0
include/lapi/syscalls/supported-arch.txt +6 -6
@@ -1,13 +1,13 @@ -aarch64 arc +arm64 arm -hppa i386 ia64 -loongarch -mips_n32 -mips_n64 -mips_o32 +loongarch64 +mips64n32 +mips64 +mipso32 +parisc powerpc64 powerpc s390x
Signature · 5 lines
2.43.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls
Hi!
Uff what a monostrosity, one would have expected that this could be done easier...
Anyways I do not see anything wrong here, so:
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls
In reply to Andrea CervesatoHi Andrea,
Quoted 10 lines
From: Andrea Cervesato <[email protected]>
Add
generate_arch.shscript which can be used to generate arch(s) dependant syscalls file. The way it works is pretty simple: for each architecture defined into supported-arch.txt, compile kernel headers, extract the list of syscalls and generate a .in file containing all of them, associated with their own syscall's number.
The way syscalls files are generated, passes through a C application which is automatically checking the availability of the syscalls in the user space environment.
Thanks for this!
...
include/lapi/syscalls/generate_arch.sh line 6
Quoted 9 lines
include/lapi/syscalls/generate_arch.sh new file +6
-0@@ -0,0 +1,211 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# +# This is an adaptation of the update-tables.sh script, included in the +# syscalls-table project (https://github.com/hrw/syscalls-table) and released +# under the MIT license.
I was surprised you use shell instead of python, but I see the original author also use shell [1]. I wish he would have used python for this part, but good that you reuse what was available.
...
Quoted 11 lines
+++
b/include/lapi/syscalls/supported-arch.txtinclude/lapi/syscalls/generate_arch.sh new file +1
-4@@ -1,13 +1,13 @@ -aarch64 arc +arm64 arm -hppa i386 ia64 -loongarch -mips_n32
include/lapi/syscalls/generate_arch.sh line 13
Quoted 10 lines
include/lapi/syscalls/generate_arch.sh new file +5
-2-mips_n64 -mips_o32 +loongarch64 +mips64n32 +mips64 +mipso32 +parisc powerpc64 powerpc s390x
Interesting enough how many archs the original project supports [2], which makes sense for it as he products HTML table as reference [3].
Kind regards,
Petr
[1] https://github.com/hrw/syscalls-table/blob/master/scripts/update-tables.sh
[2] https://github.com/hrw/syscalls-table/blob/master/system_calls/architectures_in_kernel.py
[3] https://gpages.juszkiewicz.com.pl/syscalls-table/syscalls.html
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 2/3] Add script to generate arch(s) dependant syscalls
Hi,
On 10/30/24 16:15, Petr Vorel wrote
Quoted 13 lines
...
include/lapi/syscalls/generate_arch.sh line 6
Quoted 9 lines
include/lapi/syscalls/generate_arch.sh new file +6
-0@@ -0,0 +1,211 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0-or-later +# +# This is an adaptation of the update-tables.sh script, included in the +# syscalls-table project (https://github.com/hrw/syscalls-table) and released +# under the MIT license.I was surprised you use shell instead of python, but I see the original author also use shell [1]. I wish he would have used python for this part, but good that you reuse what was available.
One of the option was to use the syscalls-table project in order to generate tables, but we needed something out of the box, without complex scenarios using pip, which might be a bit unfamiliar for some people.
Andrea
Signature · 1 line
Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH v5 3/3 Delete obsolete strip_syscall.awk file
In reply to Andrea CervesatoFrom: Andrea Cervesato <[email protected]>
--- include/lapi/syscalls/strip_syscall.awk | 19 ------------------- 1 file changed, 19 deletions(-)
include/lapi/syscalls/strip_syscall.awk deleted +0 -19
@@ -1,19 +0,0 @@ -#!/usr/bin/awk -f -# -# Dumb script that can be used to strip all of the syscall information from -# the arch-respective unistd*.h. -# -# Examples: -# -# 1. Grab the i386 32-bit syscalls from unistd_32.h and put them in i386.in -# strip_syscall.awk arch/x86/include/asm/unistd_32.h > i386.in -# - -/^#define[[:space:]]+__NR_[0-9a-z]+/ { - - sub (/#define[[:space:]]+__NR_/, "", $0); - sub (/[[:space:]]*(\/\*.*)/, "", $0); - sub (/[[:space:]]+/, " ", $0); - - print -}
Signature · 5 lines
2.43.0 -- Mailing list info: https://lists.linux.it/listinfo/ltp
Re: [LTP] [PATCH v5 3/3] Delete obsolete strip_syscall.awk file
Hi!
Signature · 5 lines
Cyril Hrubis [email protected] -- Mailing list info: https://lists.linux.it/listinfo/ltp