scripts/nsdeps: use $MODORDER to obtain correct modules.order path
PATCH 1/4 scripts/nsdeps: use $MODORDER to obtain correct modules.order path
The nsdeps script calls generate_deps() for every module in the modules.order file. It prepends $objtree to obtain the full path of the top-level modules.order file. This produces incorrect results when calling nsdeps for an external module, as only the ns dependencies for in-tree modules listed in $objtree/modules.order are generated rather than the ns dependencies for the external module. To fix this, just use the MODORDER variable provided by kbuild - it uses the correct path for the relevant modules.order file (either in-tree or the one produced by the external module build).
---
So, not being too familiar with kbuild, I am not sure if MODORDER was the appropriate kbuild variable to use, but I could not find anything else that gives us the modules.order path. Masahiro, please let me know if this is appropriate usage.
scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/nsdeps +1 -1
@@ -52,7 +52,7 @@ generate_deps() { done } -for f in `cat $objtree/modules.order`; do +for f in `cat $MODORDER`; do generate_deps $f done
Signature · 1 line
2.16.4
PATCH 2/4 scripts/nsdeps: don't prepend $srctree if *.mod already contains full paths
When building in-tree modules, the *.mod file contains relative paths. When building external modules, the resulting *.mod file contains absolute paths. Allow for the nsdeps script to account for both types of paths and only prepend $srctree in the case of relative paths. Otherwise, the script will append $srctree to the path regardless and it will error out with file not found errors if the path was already absolute to begin with.
---
The sed regex is getting more ugly. It's not my strong point :/ If anyone has a better regex to prepend $srctree for every relative path encountered while ignoring absolute paths, I'm all ears.
scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/nsdeps +1 -1
@@ -33,7 +33,7 @@ generate_deps() { if [ ! -f "$ns_deps_file" ]; then return; fi local mod_source_files=`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." generate_deps_for_ns $ns $mod_source_files
Signature · 1 line
2.16.4
Re: [PATCH 2/4] scripts/nsdeps: don't prepend $srctree if *.mod already contains full paths
Quoted 4 lines · Jessica Yu
When building in-tree modules, the *.mod file contains relative paths. When building external modules, the resulting *.mod file contains absolute paths.
Not necessarily true.Kbuild does not impose any restriction about absolute/relative path. M= can be a relative path.
Quoted 11 lines
Allow for the nsdeps script to account for both types of paths and only prepend $srctree in the case of relative paths. Otherwise, the script will append $srctree to the path regardless and it will error out with file not found errors if the path was already absolute to begin with.
Signed-off-by Jessica Yu <[email protected]>---
The sed regex is getting more ugly. It's not my strong point :/ If anyone has a better regex to prepend $srctree for every relative path encountered while ignoring absolute paths, I'm all ears.
It is not the problem of sed regex ugliness.
You can prefix $srctree/
unlesss building external modules.
if [ "$KBUILD_EXTMOD" ]; then
src_prefix=
else
src_prefix=$srctree/
fi
Then,
sed "s|[^ ]* *|${src_prefix}&|g"`
Caution: not tested at all
Quoted 17 lines
scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +1
-2@@ -33,7 +33,7 @@ generate_deps() { if [ ! -f "$ns_deps_file" ]; then return; fi local mod_source_files=`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." generate_deps_for_ns $ns $mod_source_files --
Quoted 2 lines
2.16.4
--
Best Regards
Masahiro Yamada
PATCH 3/4 nsdeps: remove stale .ns_deps files before generating new ones
In reply to Jessica YuWhen adding or removing namespaces, calling make does not necessarily remove existing stale .ns_deps files. That is, one could remove a namespace, call make, and while modpost writes the correct, new .ns_deps files, stale ones are not removed from the source tree, thus producing incorrect results when running make nsdeps, i.e., inserting MODULE_IMPORT_NS() statements for namespaces that have been removed. Clean up old .ns_deps files before generating new ones with modpost.
--- Makefile | 2 ++ 1 file changed, 2 insertions(+)
Makefile +2 -0
@@ -1685,6 +1685,8 @@ tags TAGS cscope gtags: FORCE PHONY += nsdeps nsdeps: modules + @find $(if $(KBUILD_EXTMOD), $(KBUILD_EXTMOD), .) $(RCS_FIND_IGNORE) \ + -name '*.ns_deps' -type f -print | xargs rm -f $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost nsdeps $(Q)$(CONFIG_SHELL) $(srctree)/scripts/$@
Signature · 1 line
2.16.4
Re: [PATCH 3/4] nsdeps: remove stale .ns_deps files before generating new ones
Quoted 8 lines · Jessica Yu
When adding or removing namespaces, calling
makedoes not necessarily remove existing stale .ns_deps files. That is, one could remove a namespace, call make, and while modpost writes the correct, new .ns_deps files, stale ones are not removed from the source tree, thus producing incorrect results when runningmake nsdeps, i.e., insertingMODULE_IMPORT_NS()statements for namespaces that have been removed. Clean up old .ns_deps files before generating new ones with modpost.
Hmm, correct.
But, removing *.ns_deps every time is not an elegant solution.
I want to try a different approach.
Quoted 20 lines
Signed-off-by Jessica Yu <[email protected]>--- Makefile | 2 ++ 1 file changed, 2 insertions(+)Makefile +2
-1@@ -1685,6 +1685,8 @@ tags TAGS cscope gtags: FORCE PHONY += nsdeps nsdeps: modules + @find $(if $(KBUILD_EXTMOD), $(KBUILD_EXTMOD), .) $(RCS_FIND_IGNORE) \ + -name '*.ns_deps' -type f -print | xargs rm -f $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost nsdeps $(Q)$(CONFIG_SHELL) $(srctree)/scripts/$@ --
Quoted 2 lines
2.16.4
--
Best Regards
Masahiro Yamada
PATCH 4/4 scripts/nsdeps: make sure to pass all module source files to spatch
In reply to Jessica YuThe nsdeps script passes a list of the module source files to generate_deps_for_ns() as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.
This causes problems (namely, the MODULE_IMPORT_NS() statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing the MODULE_LICENSE() statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.
--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/nsdeps +1 -1
@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmp
Signature · 1 line
2.16.4
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
Quoted 26 lines · Jessica Yu
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0
scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmp
I think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?
I think the following is needed too:
scripts/nsdeps +2 -2
@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns $ns $mod_source_files
Without this, a module that consists of two files
will be expanded to:
local mod_source_files=source1.c source2.c
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_files
Quoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.c
Yes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the generate_deps_for_ns call, (I just added a test namespace MODULE):
Adding namespace MODULE to module fs/nfs/nfs.ko.
+ generate_deps_for_ns MODULE /tmp/ppyu/linux/fs/nfs/client.c /tmp/ppyu/linux/fs/nfs/dir.c /tmp/ppyu/linux/fs/nfs/file.c /tmp/ppyu/linux/fs/nfs/getroot.c /tmp/ppyu/linux/fs/nfs/inode.c /tmp/ppyu/linux/fs/nfs/super.c /tmp/ppyu/linux/fs/nfs/io.c /tmp/ppyu/linux/fs/nfs/direct.c /tmp/ppyu/linux/fs/nfs/pagelist.c /tmp/ppyu/linux/fs/nfs/read.c /tmp/ppyu/linux/fs/nfs/symlink.c /tmp/ppyu/linux/fs/nfs/unlink.c /tmp/ppyu/linux/fs/nfs/write.c /tmp/ppyu/linux/fs/nfs/namespace.c /tmp/ppyu/linux/fs/nfs/mount_clnt.c /tmp/ppyu/linux/fs/nfs/nfstrace.c /tmp/ppyu/linux/fs/nfs/export.c /tmp/ppyu/linux/fs/nfs/sysfs.c /tmp/ppyu/linux/fs/nfs/sysctl.c /tmp/ppyu/linux/fs/nfs/fscache.c /tmp/ppyu/linux/fs/nfs/fscache-index.c
+ /usr/bin/spatch --very-quiet --in-place --sp-file /tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci -D ns=MODULE /tmp/ppyu/linux/fs/nfs/client.c
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to generate_deps_for_ns.
But we need to include your change anyway, to make the script more robust. It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
Quoted 78 lines · Jessica Yu
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_filesQuoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.cYes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the
generate_deps_for_nscall, (I just added a test namespace MODULE):Adding namespace MODULE to module
fs/nfs/nfs.ko.
+generate_deps_for_nsMODULE/tmp/ppyu/linux/fs/nfs/client.c/tmp/ppyu/linux/fs/nfs/dir.c/tmp/ppyu/linux/fs/nfs/file.c/tmp/ppyu/linux/fs/nfs/getroot.c/tmp/ppyu/linux/fs/nfs/inode.c/tmp/ppyu/linux/fs/nfs/super.c/tmp/ppyu/linux/fs/nfs/io.c/tmp/ppyu/linux/fs/nfs/direct.c/tmp/ppyu/linux/fs/nfs/pagelist.c/tmp/ppyu/linux/fs/nfs/read.c/tmp/ppyu/linux/fs/nfs/symlink.c/tmp/ppyu/linux/fs/nfs/unlink.c/tmp/ppyu/linux/fs/nfs/write.c/tmp/ppyu/linux/fs/nfs/namespace.c/tmp/ppyu/linux/fs/nfs/mount_clnt.c/tmp/ppyu/linux/fs/nfs/nfstrace.c/tmp/ppyu/linux/fs/nfs/export.c/tmp/ppyu/linux/fs/nfs/sysfs.c/tmp/ppyu/linux/fs/nfs/sysctl.c/tmp/ppyu/linux/fs/nfs/fscache.c/tmp/ppyu/linux/fs/nfs/fscache-index.c
+/usr/bin/spatch--very-quiet --in-place --sp-file/tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci-D ns=MODULE/tmp/ppyu/linux/fs/nfs/client.c
Quoted 6 lines
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to
generate_deps_for_ns.But we need to include your change anyway, to make the script more robust.
Hmm.
With this patch only, I see "bad variable name" error.
masahiro@grover:~/ref/linux$ make -j8 nsdeps DESCEND objtool CALL scripts/atomic/check-atomics.sh CALL scripts/checksyscalls.sh CHK include/generated/compile.h Building modules, stage 2. MODPOST 20 modules WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but does not import it. Building modules, stage 2. MODPOST 20 modules ./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name make: *** [Makefile;1689: nsdeps] Error 2
Quoted 5 lines
It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Anyway.
Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?
If you touch the mod_source_files line,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.
How should we organize the patch order?
--
Best Regards
Masahiro Yamada
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
+++ Masahiro Yamada [31/10/19 21:27 +0900]:
Quoted 91 lines · Jessica Yu
Quoted 78 lines
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_filesQuoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.cYes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the
generate_deps_for_nscall, (I just added a test namespace MODULE):Adding namespace MODULE to module
fs/nfs/nfs.ko.
+generate_deps_for_nsMODULE/tmp/ppyu/linux/fs/nfs/client.c/tmp/ppyu/linux/fs/nfs/dir.c/tmp/ppyu/linux/fs/nfs/file.c/tmp/ppyu/linux/fs/nfs/getroot.c/tmp/ppyu/linux/fs/nfs/inode.c/tmp/ppyu/linux/fs/nfs/super.c/tmp/ppyu/linux/fs/nfs/io.c/tmp/ppyu/linux/fs/nfs/direct.c/tmp/ppyu/linux/fs/nfs/pagelist.c/tmp/ppyu/linux/fs/nfs/read.c/tmp/ppyu/linux/fs/nfs/symlink.c/tmp/ppyu/linux/fs/nfs/unlink.c/tmp/ppyu/linux/fs/nfs/write.c/tmp/ppyu/linux/fs/nfs/namespace.c/tmp/ppyu/linux/fs/nfs/mount_clnt.c/tmp/ppyu/linux/fs/nfs/nfstrace.c/tmp/ppyu/linux/fs/nfs/export.c/tmp/ppyu/linux/fs/nfs/sysfs.c/tmp/ppyu/linux/fs/nfs/sysctl.c/tmp/ppyu/linux/fs/nfs/fscache.c/tmp/ppyu/linux/fs/nfs/fscache-index.c
+/usr/bin/spatch--very-quiet --in-place --sp-file/tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci-D ns=MODULE/tmp/ppyu/linux/fs/nfs/client.cQuoted 6 lines
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to
generate_deps_for_ns.But we need to include your change anyway, to make the script more robust.
Hmm.
With this patch only, I see "bad variable name" error.
Quoted 13 lines
masahiro@grover:~/ref/linux$ make -j8 nsdeps DESCEND objtool CALL scripts/atomic/check-atomics.sh CALL scripts/checksyscalls.sh CHK include/generated/compile.h Building modules, stage 2. MODPOST 20 modules WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but does not import it. Building modules, stage 2. MODPOST 20 modules ./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name make: *** [Makefile;1689: nsdeps] Error 2
Hm, I was having trouble reproducing this until I changed the shell to dash, /bin/sh is a symlink to bash on my system, that might explain slightly different behavior. In any case, we should add quotes in both places.
Quoted 9 lines
Quoted 5 lines
It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Anyway.
Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?
I am hoping for fixes, we should try to get all the small bugs out of nsdeps by 5.4 if we can..
Quoted 6 lines
If you touch the
mod_source_filesline,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.How should we organize the patch order?
Would you like to fold these changes into your nsdeps improvements patchset? Since it's a pretty trivial change.
Thanks!
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
Quoted 130 lines · Jessica Yu
+++ Masahiro Yamada [31/10/19 21:27 +0900]:
Quoted 91 lines · Jessica Yu
Quoted 78 lines
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_filesQuoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.cYes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the
generate_deps_for_nscall, (I just added a test namespace MODULE):Adding namespace MODULE to module
fs/nfs/nfs.ko.
+generate_deps_for_nsMODULE/tmp/ppyu/linux/fs/nfs/client.c/tmp/ppyu/linux/fs/nfs/dir.c/tmp/ppyu/linux/fs/nfs/file.c/tmp/ppyu/linux/fs/nfs/getroot.c/tmp/ppyu/linux/fs/nfs/inode.c/tmp/ppyu/linux/fs/nfs/super.c/tmp/ppyu/linux/fs/nfs/io.c/tmp/ppyu/linux/fs/nfs/direct.c/tmp/ppyu/linux/fs/nfs/pagelist.c/tmp/ppyu/linux/fs/nfs/read.c/tmp/ppyu/linux/fs/nfs/symlink.c/tmp/ppyu/linux/fs/nfs/unlink.c/tmp/ppyu/linux/fs/nfs/write.c/tmp/ppyu/linux/fs/nfs/namespace.c/tmp/ppyu/linux/fs/nfs/mount_clnt.c/tmp/ppyu/linux/fs/nfs/nfstrace.c/tmp/ppyu/linux/fs/nfs/export.c/tmp/ppyu/linux/fs/nfs/sysfs.c/tmp/ppyu/linux/fs/nfs/sysctl.c/tmp/ppyu/linux/fs/nfs/fscache.c/tmp/ppyu/linux/fs/nfs/fscache-index.c
+/usr/bin/spatch--very-quiet --in-place --sp-file/tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci-D ns=MODULE/tmp/ppyu/linux/fs/nfs/client.cQuoted 6 lines
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to
generate_deps_for_ns.But we need to include your change anyway, to make the script more robust.
Hmm.
With this patch only, I see "bad variable name" error.Quoted 13 lines
masahiro@grover:~/ref/linux$ make -j8 nsdeps DESCEND objtool CALL scripts/atomic/check-atomics.sh CALL scripts/checksyscalls.sh CHK include/generated/compile.h Building modules, stage 2. MODPOST 20 modules WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but does not import it. Building modules, stage 2. MODPOST 20 modules ./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name make: *** [Makefile;1689: nsdeps] Error 2Hm, I was having trouble reproducing this until I changed the shell to dash,
/bin/shis a symlink to bash on my system, that might explain slightly different behavior. In any case, we should add quotes in both places.Quoted 9 lines
Quoted 5 lines
It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Anyway.
Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?
I am hoping for fixes, we should try to get all the small bugs out of nsdeps by 5.4 if we can..
Quoted 5 lines
If you touch the
mod_source_filesline,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.
Quoted 8 lines
Quoted 1 line
How should we organize the patch order?
Would you like to fold these changes into your nsdeps improvements patchset? Since it's a pretty trivial change.
Thanks!
My change set is quite big, so I
am planning to send it for v5.5-rc1.
If you want to fix nsdeps for composite modules earlier, please apply this patch to your tree
and send another pull request.
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
+++ Masahiro Yamada [01/11/19 14:56 +0900]:
Quoted 136 lines · Jessica Yu
Quoted 135 lines
+++ Masahiro Yamada [31/10/19 21:27 +0900]:
Quoted 91 lines · Jessica Yu
Quoted 78 lines
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_filesQuoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.cYes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the
generate_deps_for_nscall, (I just added a test namespace MODULE):Adding namespace MODULE to module
fs/nfs/nfs.ko.
+generate_deps_for_nsMODULE/tmp/ppyu/linux/fs/nfs/client.c/tmp/ppyu/linux/fs/nfs/dir.c/tmp/ppyu/linux/fs/nfs/file.c/tmp/ppyu/linux/fs/nfs/getroot.c/tmp/ppyu/linux/fs/nfs/inode.c/tmp/ppyu/linux/fs/nfs/super.c/tmp/ppyu/linux/fs/nfs/io.c/tmp/ppyu/linux/fs/nfs/direct.c/tmp/ppyu/linux/fs/nfs/pagelist.c/tmp/ppyu/linux/fs/nfs/read.c/tmp/ppyu/linux/fs/nfs/symlink.c/tmp/ppyu/linux/fs/nfs/unlink.c/tmp/ppyu/linux/fs/nfs/write.c/tmp/ppyu/linux/fs/nfs/namespace.c/tmp/ppyu/linux/fs/nfs/mount_clnt.c/tmp/ppyu/linux/fs/nfs/nfstrace.c/tmp/ppyu/linux/fs/nfs/export.c/tmp/ppyu/linux/fs/nfs/sysfs.c/tmp/ppyu/linux/fs/nfs/sysctl.c/tmp/ppyu/linux/fs/nfs/fscache.c/tmp/ppyu/linux/fs/nfs/fscache-index.c
+/usr/bin/spatch--very-quiet --in-place --sp-file/tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci-D ns=MODULE/tmp/ppyu/linux/fs/nfs/client.cQuoted 6 lines
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to
generate_deps_for_ns.But we need to include your change anyway, to make the script more robust.
Hmm.
With this patch only, I see "bad variable name" error.Quoted 13 lines
masahiro@grover:~/ref/linux$ make -j8 nsdeps DESCEND objtool CALL scripts/atomic/check-atomics.sh CALL scripts/checksyscalls.sh CHK include/generated/compile.h Building modules, stage 2. MODPOST 20 modules WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but does not import it. Building modules, stage 2. MODPOST 20 modules ./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name make: *** [Makefile;1689: nsdeps] Error 2Hm, I was having trouble reproducing this until I changed the shell to dash,
/bin/shis a symlink to bash on my system, that might explain slightly different behavior. In any case, we should add quotes in both places.Quoted 9 lines
Quoted 5 lines
It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Anyway.
Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?
I am hoping for fixes, we should try to get all the small bugs out of nsdeps by 5.4 if we can..
Quoted 6 lines
If you touch the
mod_source_filesline,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.How should we organize the patch order?
Would you like to fold these changes into your nsdeps improvements patchset? Since it's a pretty trivial change.
Quoted 6 lines
Quoted 3 lines
Thanks!
My change set is quite big, so I
am planning to send it for v5.5-rc1.
OK, sounds good.
Quoted 3 lines
If you want to fix nsdeps for composite modules earlier, please apply this patch to your tree
and send another pull request.
I plan to send this in for -rc7, since it's a minor fix.
For the conflict, if these patches are not pushed anywhere yet (at least I did not see them on the kbuild for-next branch yet), perhaps you could rebase on top of -rc7 and put your patchset on top?
Thanks,
Jessica
Re: [PATCH 4/4] scripts/nsdeps: make sure to pass all module source files to spatch
Quoted 145 lines · Jessica Yu
+++ Masahiro Yamada [01/11/19 14:56 +0900]:
Quoted 136 lines · Jessica Yu
Quoted 135 lines
+++ Masahiro Yamada [31/10/19 21:27 +0900]:
Quoted 91 lines · Jessica Yu
Quoted 78 lines
+++ Masahiro Yamada [29/10/19 21:57 +0900]:
Quoted 62 lines · Jessica Yu
Quoted 26 lines
The nsdeps script passes a list of the module source files to
generate_deps_for_ns()as a space delimited string named $mod_source_files, which then passes it to spatch. But since $mod_source_files is not encased in quotes, each source file in that string is treated as a separate shell function argument (as $2, $3, $4, etc.). However, the spatch invocation only refers to $2, so only the first file out of $mod_source_files is processed by spatch.This causes problems (namely, the
MODULE_IMPORT_NS()statement doesn't get inserted) when a module is composed of many source files and the "main" module file containing theMODULE_LICENSE()statement is not the first file listed in $mod_source_files. Fix this by encasing $mod_source_files in quotes so that the entirety of the string is treated as a single argument and can be referred to as $2.Signed-off-by Jessica Yu <[email protected]>--- scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +0
-0scripts/nsdeps generate_deps() { · line 42
Quoted 8 lines
scripts/nsdeps +1
-1@@ -36,7 +36,7 @@ generate_deps() { | sed -E "s%(^|\s)([^/][^ ]*)%\1$srctree/\2%g"` for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)." - generate_deps_for_ns $ns $mod_source_files + generate_deps_for_ns $ns "$mod_source_files" # sort the imports for source_file in $mod_source_files; do sed '/MODULE_IMPORT_NS/Q' $source_file > ${source_file}.tmpI think this change is correct, but
did you succeed in nsdeps for composite modules
with this patch only?I think the following is needed too:
scripts/nsdeps +2
-2@@ -31,9 +31,9 @@ generate_deps() { local mod_file=`echo $@ | sed -e 's/\.ko/\.mod/'` local ns_deps_file=`echo $@ | sed -e 's/\.ko/\.ns_deps/'` if [ ! -f "$ns_deps_file" ]; then return; fi - local mod_source_files=`cat $mod_file | sed -n 1p \ + local mod_source_files="`cat $mod_file | sed -n 1p \ | sed -e 's/\.o/\.c/g' \ - | sed "s|[^ ]* *|${srctree}/&|g"` + | sed "s|[^ ]* *|${srctree}/&|g"`" for ns in `cat $ns_deps_file`; do echo "Adding namespace $ns to module $mod_name (if needed)."
generate_deps_for_ns$ns $mod_source_filesQuoted 4 lines
Without this, a module that consists of two files
will be expanded to:local mod_source_files=source1.c
source2.cYes, I was able to have nsdeps work for composite modules with just my patch. Without this patch applied, the script produces the following expansion of the
generate_deps_for_nscall, (I just added a test namespace MODULE):Adding namespace MODULE to module
fs/nfs/nfs.ko.
+generate_deps_for_nsMODULE/tmp/ppyu/linux/fs/nfs/client.c/tmp/ppyu/linux/fs/nfs/dir.c/tmp/ppyu/linux/fs/nfs/file.c/tmp/ppyu/linux/fs/nfs/getroot.c/tmp/ppyu/linux/fs/nfs/inode.c/tmp/ppyu/linux/fs/nfs/super.c/tmp/ppyu/linux/fs/nfs/io.c/tmp/ppyu/linux/fs/nfs/direct.c/tmp/ppyu/linux/fs/nfs/pagelist.c/tmp/ppyu/linux/fs/nfs/read.c/tmp/ppyu/linux/fs/nfs/symlink.c/tmp/ppyu/linux/fs/nfs/unlink.c/tmp/ppyu/linux/fs/nfs/write.c/tmp/ppyu/linux/fs/nfs/namespace.c/tmp/ppyu/linux/fs/nfs/mount_clnt.c/tmp/ppyu/linux/fs/nfs/nfstrace.c/tmp/ppyu/linux/fs/nfs/export.c/tmp/ppyu/linux/fs/nfs/sysfs.c/tmp/ppyu/linux/fs/nfs/sysctl.c/tmp/ppyu/linux/fs/nfs/fscache.c/tmp/ppyu/linux/fs/nfs/fscache-index.c
+/usr/bin/spatch--very-quiet --in-place --sp-file/tmp/ppyu/linux/scripts/coccinelle/misc/add_namespace.cocci-D ns=MODULE/tmp/ppyu/linux/fs/nfs/client.cQuoted 6 lines
So only the first file got included in the spatch invocation. But the spatch call gets fixed with all the files when quotes are added in the call to
generate_deps_for_ns.But we need to include your change anyway, to make the script more robust.
Hmm.
With this patch only, I see "bad variable name" error.Quoted 13 lines
masahiro@grover:~/ref/linux$ make -j8 nsdeps DESCEND objtool CALL scripts/atomic/check-atomics.sh CALL scripts/checksyscalls.sh CHK include/generated/compile.h Building modules, stage 2. MODPOST 20 modules WARNING: module nfs uses symbol foo from namespace USB_STORAGE, but does not import it. Building modules, stage 2. MODPOST 20 modules ./scripts/nsdeps: 34: local: ./fs/nfs/dir.c: bad variable name make: *** [Makefile;1689: nsdeps] Error 2Hm, I was having trouble reproducing this until I changed the shell to dash,
/bin/shis a symlink to bash on my system, that might explain slightly different behavior. In any case, we should add quotes in both places.Quoted 9 lines
Quoted 5 lines
It would probably prevent more shell script related bugs in the future (Like [1]). I can respin this patch only while the other ones are superceded by your patchset.
Anyway.
Is this patch aiming for v5.4 (i.e. fixes) or v5.5-rc1 ?
I am hoping for fixes, we should try to get all the small bugs out of nsdeps by 5.4 if we can..
Quoted 6 lines
If you touch the
mod_source_filesline,
we will have a conflict because
https://patchwork.kernel.org/patch/11217839/
is touching the same line.How should we organize the patch order?
Would you like to fold these changes into your nsdeps improvements patchset? Since it's a pretty trivial change.
Quoted 6 lines
Quoted 3 lines
Thanks!
My change set is quite big, so I
am planning to send it for v5.5-rc1.
Quoted 7 lines
OK, sounds good.
Quoted 3 lines
If you want to fix nsdeps for composite modules earlier, please apply this patch to your tree
and send another pull request.I plan to send this in for -rc7, since it's a minor fix.
OK.
Quoted 3 lines
For the conflict, if these patches are not pushed anywhere yet (at least I did not see them on the kbuild for-next branch yet), perhaps you could rebase on top of -rc7 and put your patchset on top?
No problem. I will do so.
Thanks.
Signature · 2 lines
Best Regards Masahiro Yamada
Quoted 11 lines · Jessica Yu
The nsdeps script calls
generate_deps()for every module in the modules.order file. It prepends $objtree to obtain the full path of the top-level modules.order file. This produces incorrect results when calling nsdeps for an external module, as only the ns dependencies for in-tree modules listed in $objtree/modules.order are generated rather than the ns dependencies for the external module. To fix this, just use the MODORDER variable provided by kbuild - it uses the correct path for the relevant modules.order file (either in-tree or the one produced by the external module build).
Quoted 7 lines
Signed-off-by Jessica Yu <[email protected]>---
So, not being too familiar with kbuild, I am not sure if MODORDER was the appropriate kbuild variable to use, but I could not find anything else that gives us the modules.order path. Masahiro, please let me know if this is appropriate usage.
Right, MODORDER allows you to get access to
the right modules.order depending on what you are building although this patch alone is not useful since nsdeps does not work with external modules.
Quoted 17 lines
scripts/nsdeps | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/nsdeps +1
-2@@ -52,7 +52,7 @@ generate_deps() { done } -for f in `cat $objtree/modules.order`; do +for f in `cat $MODORDER`; do generate_deps $f done --
Quoted 2 lines
2.16.4
--
Best Regards
Masahiro Yamada