Skip to contentSkip to conversation

kbuild: clang-tidy

PATCH Superseded Masahiro Yamada 4 participants 15 messages Aug 12, 2020 – Aug 20, 2020

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover kbuild: clang-tidy

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I built clang-tidy from the latest source.
I had no idea how to make it work...

[1] https://patchwork.kernel.org/patch/11687833/

Masahiro Yamada (2):
gen_compile_commands: parse only the first line of .*.cmd files
gen_compile_commands: wire up build rule to Makefile

Nathan Huckleberry (1):
Makefile: Add clang-tidy and static analyzer support to makefile

 MAINTAINERS                                 |   1 +
 Makefile                                    |  45 +++++-
 scripts/clang-tools/gen_compile_commands.py | 117 +++++++++++++++
 scripts/clang-tools/run-clang-tools.py      |  74 ++++++++++
 scripts/gen_compile_commands.py             | 151 --------------------
 5 files changed, 233 insertions(+), 155 deletions(-)
 create mode 100755 scripts/clang-tools/gen_compile_commands.py
 create mode 100755 scripts/clang-tools/run-clang-tools.py
 delete mode 100755 scripts/gen_compile_commands.py
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/3 gen_compile_commands: parse only the first line of .*.cmd files

After the allmodconfig build, this script takes about 5 sec on my machine. Most of the run-time is consumed for needless regex matching.

We know the format of .*.cmd file; the first line is the build command. There is no need to parse the remaining.

With this optimization, now it runs in about 1 sec with the allmodconfig build.

---

 scripts/gen_compile_commands.py | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
scripts/gen_compile_commands.py +3 -5
index c458696ef3a7..19c7338740e7 100755
@@ -125,11 +125,9 @@ def main():
             filepath = os.path.join(dirpath, filename)
 
             with open(filepath, 'rt') as f:
-                for line in f:
-                    result = line_matcher.match(line)
-                    if not result:
-                        continue
-
+                line = f.readline()
+                result = line_matcher.match(line)
+                if result:
                     try:
                         entry = process_line(directory, dirpath,
                                              result.group(1), result.group(2))
Signature · 1 line
2.25.1
reviewer Reviewed-by Tested-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] gen_compile_commands: parse only the first line of .*.cmd files

Quoted 11 lines · Masahiro Yamada

After the allmodconfig build, this script takes about 5 sec on my machine. Most of the run-time is consumed for needless regex matching.

We know the format of .*.cmd file; the first line is the build command. There is no need to parse the remaining.

With this optimization, now it runs in about 1 sec with the allmodconfig build.

Before your patch, my x86_64 allmodconfig:

$ /usr/bin/time -v ./scripts/gen_compile_commands.py
...
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.35
...
After:
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.10

So it's a nice speedup.

small nit below:

Quoted 10 lines

---

 scripts/gen_compile_commands.py | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
scripts/gen_compile_commands.py +0 -0
index c458696ef3a7..19c7338740e7 100755

scripts/gen_compile_commands.py def main(): · line 129

Quoted 10 lines
scripts/gen_compile_commands.py +2 -5
@@ -125,11 +125,9 @@ def main():
             filepath = os.path.join(dirpath, filename)
 
             with open(filepath, 'rt') as f:
-                for line in f:
-                    result = line_matcher.match(line)
-                    if not result:
-                        continue
-
+                line = f.readline()
+                result = line_matcher.match(line)

If line is not referenced beyond the following statement, consider combining the two statement into one:
result = line_matcher.match(f.readline())

Quoted 7 lines
scripts/gen_compile_commands.py +1 -1
+                if result:
                     try:
                         entry = process_line(directory, dirpath,
                                              result.group(1), result.group(2))
--

2.25.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
#3 lore raw

PATCH 2/3 gen_compile_commands: wire up build rule to Makefile

In reply to Masahiro Yamada

Currently, you need to explicitly run scripts/gen_compile_commands.py to create compile_commands.json. It traverses the object tree (you need to pass the -d option to deal with a separate output tree), and parses all the .*.cmd file found.

If you rebuild the kernel over again without 'make clean', stale .*.cmd files from older builds will create invalid entries in compile_commands.json.

This commit wires up the compile_commands.json rule to the top Makefile, and makes it parse .*.cmd files only from the current build to avoid stale entries.

It is possible to extract only relevant .*.cmd files by checking $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order. The objects or archives linked to vmlinux are listed in $(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are listed in modules.order.

You can create compile_commands.json from Make:

  $ make -j$(nproc) CC=clang compile_commands.json

Of course, you can build vmlinux, modules, and compile_commands.json all together in a single command:

  $ make -j$(nproc) CC=clang all compile_commands.json

It works also for M= builds. In this case, compile_commands.json is created in the top directory of the external module.

I hope this will be overall improvements, but it has a drawback; the coverage of the compile_commands.json is reduced because only the objects linked to vmlinux or modules are handled. For example, the following C files are not included in compile_commands.json:

- Decompressor source files (arch/*/boot/compressed/)
- VDSO source files
- C files used to generate intermediates (e.g. kernel/bounds.c)
- standalone host programs

Here is a note for out-of-tree builds. 'make compile_commands.json' works with O= option, but please notice compile_commands.json is created in the object tree instead of the source tree.

Some people may want to have compile_commands.json in the source tree because Clang Tools searches for it through all parent paths of the first input source file.

However, you cannot do it for O= builds. Kbuild should never generate any build artifact in the source tree when O= is given because the source tree might be read-only. Any write attempt to the source tree is monitored and the violation may be reported. See the commit log of 8ef14c2c41d9.

So, the only possible way it to create compile_commands.json in the object tree, then specify '-p <build-path>' when you use clang-check, clang-tidy, etc.

---

 Makefile                        |  29 ++++++-
 scripts/gen_compile_commands.py | 146 +++++++++++++-------------------
 2 files changed, 82 insertions(+), 93 deletions(-)
Makefile +25 -4
index 6844b848bfec..4d65affb6917 100644
@@ -634,7 +634,7 @@ endif
 # in addition to whatever we do anyway.
 # Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all modules nsdeps,$(MAKECMDGOALS)),)
+ifneq ($(filter all modules nsdeps %compile_commands.json,$(MAKECMDGOALS)),)
   KBUILD_MODULES := 1
 endif
 
@@ -1459,7 +1459,8 @@ endif # CONFIG_MODULES
 
 # Directories & files removed with 'make clean'
 CLEAN_FILES += include/ksym vmlinux.symvers \
-	       modules.builtin modules.builtin.modinfo modules.nsdeps
+	       modules.builtin modules.builtin.modinfo modules.nsdeps \
+	       compile_commands.json
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -1693,9 +1694,12 @@ KBUILD_MODULES := 1
 
 build-dirs := $(KBUILD_EXTMOD)
 PHONY += modules
-modules: descend
+modules: $(MODORDER)
 	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
 
+$(MODORDER): descend
+	@:
+
 PHONY += modules_install
 modules_install: _emodinst_ _emodinst_post
 
@@ -1709,8 +1713,12 @@ PHONY += _emodinst_post
 _emodinst_post: _emodinst_
 	$(call cmd,depmod)
 
+compile_commands.json: $(extmod-prefix)compile_commands.json
+PHONY += compile_commands.json
+
 clean-dirs := $(KBUILD_EXTMOD)
-clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps
+clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps \
+	$(KBUILD_EXTMOD)/compile_commands.json
 
 PHONY += help
 help:
@@ -1823,6 +1831,19 @@ nsdeps: export KBUILD_NSDEPS=1
 nsdeps: modules
 	$(Q)$(CONFIG_SHELL) $(srctree)/scripts/nsdeps
 
+# Clang Tooling
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_compile_commands = GEN     $@
+      cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs))
+
+$(extmod-prefix)compile_commands.json: scripts/gen_compile_commands.py \
+	$(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \
+	$(if $(CONFIG_MODULES), $(MODORDER)) FORCE
+	$(call if_changed,gen_compile_commands)
+
+targets += $(extmod-prefix)compile_commands.json
+
 # Scripts to check various things for consistency
 # ---------------------------------------------------------------------------
 
scripts/gen_compile_commands.py +57 -89

197 lines not on this page · raw message

Signature · 1 line
2.25.1
#4 lore raw

Re: [PATCH 2/3] gen_compile_commands: wire up build rule to Makefile

Quoted 9 lines · Masahiro Yamada

Currently, you need to explicitly run scripts/gen_compile_commands.py to create compile_commands.json. It traverses the object tree (you need to pass the -d option to deal with a separate output tree), and parses all the .*.cmd file found.

If you rebuild the kernel over again without 'make clean', stale .*.cmd files from older builds will create invalid entries in compile_commands.json.

Definitely a problem; happy to see compile_commands.json added to make clean target, too.

Quoted 28 lines

This commit wires up the compile_commands.json rule to the top Makefile, and makes it parse .*.cmd files only from the current build to avoid stale entries.

It is possible to extract only relevant .*.cmd files by checking $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order. The objects or archives linked to vmlinux are listed in $(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are listed in modules.order.

You can create compile_commands.json from Make:

  $ make -j$(nproc) CC=clang compile_commands.json

Of course, you can build vmlinux, modules, and compile_commands.json all together in a single command:

  $ make -j$(nproc) CC=clang all compile_commands.json

It works also for M= builds. In this case, compile_commands.json is created in the top directory of the external module.

I hope this will be overall improvements, but it has a drawback; the coverage of the compile_commands.json is reduced because only the objects linked to vmlinux or modules are handled. For example, the following C files are not included in compile_commands.json:

Quoted 4 lines

- Decompressor source files (arch/*/boot/compressed/)
- VDSO source files
- C files used to generate intermediates (e.g. kernel/bounds.c)
- standalone host programs

Oof, for an x86_64 defconfig, the difference in line count of compile_commands.json
before: 12826
after: 12351

That's a loss of 475 (3.7% of 12826) coverage. Is there something more we can do to preserve this functionality, while avoiding stale .cmd files?

Is it that those aren't specified by $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) ?

Quoted 64 lines

Here is a note for out-of-tree builds. 'make compile_commands.json' works with O= option, but please notice compile_commands.json is created in the object tree instead of the source tree.

Some people may want to have compile_commands.json in the source tree because Clang Tools searches for it through all parent paths of the first input source file.

However, you cannot do it for O= builds. Kbuild should never generate any build artifact in the source tree when O= is given because the source tree might be read-only. Any write attempt to the source tree is monitored and the violation may be reported. See the commit log of 8ef14c2c41d9.

So, the only possible way it to create compile_commands.json in the object tree, then specify '-p <build-path>' when you use clang-check, clang-tidy, etc.

---

 Makefile                        |  29 ++++++-
 scripts/gen_compile_commands.py | 146 +++++++++++++-------------------
 2 files changed, 82 insertions(+), 93 deletions(-)
Makefile +7 -3
index 6844b848bfec..4d65affb6917 100644
@@ -634,7 +634,7 @@ endif
 # in addition to whatever we do anyway.
 # Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all modules nsdeps,$(MAKECMDGOALS)),)
+ifneq ($(filter all modules nsdeps %compile_commands.json,$(MAKECMDGOALS)),)
   KBUILD_MODULES := 1
 endif
 
@@ -1459,7 +1459,8 @@ endif # CONFIG_MODULES
 
 # Directories & files removed with 'make clean'
 CLEAN_FILES += include/ksym vmlinux.symvers \
-              modules.builtin modules.builtin.modinfo modules.nsdeps
+              modules.builtin modules.builtin.modinfo modules.nsdeps \
+              compile_commands.json
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
@@ -1693,9 +1694,12 @@ KBUILD_MODULES := 1
 
 build-dirs := $(KBUILD_EXTMOD)
 PHONY += modules
-modules: descend
+modules: $(MODORDER)
        $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
 
+$(MODORDER): descend
+       @:
+
 PHONY += modules_install
 modules_install: _emodinst_ _emodinst_post
 

Makefile PHONY += _emodinst_post · line 1721

Quoted 10 lines
Makefile +5 -1
@@ -1709,8 +1713,12 @@ PHONY += _emodinst_post
 _emodinst_post: _emodinst_
        $(call cmd,depmod)
 
+compile_commands.json: $(extmod-prefix)compile_commands.json
+PHONY += compile_commands.json
+
 clean-dirs := $(KBUILD_EXTMOD)
-clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps
+clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps \
+       $(KBUILD_EXTMOD)/compile_commands.json

So the clean target doesn't make use of CLEAN_FILES? It looks like there's some duplication there? Oh, this is dependent on !KBUILD_EXTMOD, and is a new clean target. Do I understand that correctly?

Quoted 57 lines
Makefile +13 -0
 PHONY += help
 help:
@@ -1823,6 +1831,19 @@ nsdeps: export KBUILD_NSDEPS=1
 nsdeps: modules
        $(Q)$(CONFIG_SHELL) $(srctree)/scripts/nsdeps
 
+# Clang Tooling
+# ---------------------------------------------------------------------------
+
+quiet_cmd_gen_compile_commands = GEN     $@
+      cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs))
+
+$(extmod-prefix)compile_commands.json: scripts/gen_compile_commands.py \
+       $(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \
+       $(if $(CONFIG_MODULES), $(MODORDER)) FORCE
+       $(call if_changed,gen_compile_commands)
+
+targets += $(extmod-prefix)compile_commands.json
+
 # Scripts to check various things for consistency
 # ---------------------------------------------------------------------------
 
scripts/gen_compile_commands.py +2 -15
index 19c7338740e7..d2ff0d982521 100755
@@ -9,80 +9,49 @@
 
 import argparse
 import json
-import logging
 import os
 import re
-
-_DEFAULT_OUTPUT = 'compile_commands.json'
-_DEFAULT_LOG_LEVEL = 'WARNING'
-
-_FILENAME_PATTERN = r'^\..*\.cmd$'
-_LINE_PATTERN = r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$'
-_VALID_LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
-
-# A kernel build generally has over 2000 entries in its compile_commands.json
-# database. If this code finds 300 or fewer, then warn the user that they might
-# not have all the .cmd files, and they might need to compile the kernel.
-_LOW_COUNT_THRESHOLD = 300
+import subprocess
 
 
 def parse_arguments():
     """Sets up and parses command-line arguments.
 
     Returns:
-        log_level: A logging level to filter log output.
-        directory: The directory to search for .cmd files.
+        ar: Command used for parsing .a archives
         output: Where to write the compile-commands JSON file.

scripts/gen_compile_commands.py line 29

Quoted 10 lines
scripts/gen_compile_commands.py +3 -3
+        files: Files to parse
     """
     usage = 'Creates a compile_commands.json database from kernel .cmd files'
     parser = argparse.ArgumentParser(description=usage)
 
-    directory_help = ('Path to the kernel source directory to search '
-                      '(defaults to the working directory)')
-    parser.add_argument('-d', '--directory', type=str, help=directory_help)
+    ar_help = 'command used for parsing .a archives'
+    parser.add_argument('-a', '--ar', type=str, default='ar', help=ar_help)

Might be nice to warn if run with no arguments? In case someone does:

$ ./scripts/clang-tools/gen_compile_commands.py
Quoted 96 lines
scripts/gen_compile_commands.py +20 -48
-    output_help = ('The location to write compile_commands.json (defaults to '
-                   'compile_commands.json in the search directory)')
-    parser.add_argument('-o', '--output', type=str, help=output_help)
+    output_help = 'output file for the compilation database'
+    parser.add_argument('-o', '--output', type=str,
+                        default='compile_commands.json', help=output_help)
 
-    log_level_help = ('The level of log messages to produce (one of ' +
-                      ', '.join(_VALID_LOG_LEVELS) + '; defaults to ' +
-                      _DEFAULT_LOG_LEVEL + ')')
-    parser.add_argument(
-        '--log_level', type=str, default=_DEFAULT_LOG_LEVEL,
-        help=log_level_help)
+    files_help='files to parse (should be *.o, *.a, or modules.order)'
+    parser.add_argument('files', type=str, nargs='*', help=files_help)
 
     args = parser.parse_args()
 
-    log_level = args.log_level
-    if log_level not in _VALID_LOG_LEVELS:
-        raise ValueError('%s is not a valid log level' % log_level)
-
-    directory = args.directory or os.getcwd()
-    output = args.output or os.path.join(directory, _DEFAULT_OUTPUT)
-    directory = os.path.abspath(directory)
-
-    return log_level, directory, output
+    return args.ar, args.output, args.files
 
 
-def process_line(root_directory, file_directory, command_prefix, relative_path):
+def process_line(root_directory, command_prefix, file_path):
     """Extracts information from a .cmd line and creates an entry from it.
 
     Args:
         root_directory: The directory that was searched for .cmd files. Usually
             used directly in the "directory" entry in compile_commands.json.
-        file_directory: The path to the directory the .cmd file was found in.
         command_prefix: The extracted command line, up to the last element.
-        relative_path: The .c file from the end of the extracted command.
-            Usually relative to root_directory, but sometimes relative to
-            file_directory and sometimes neither.
+        file_path: The .c file from the end of the extracted command.
+            It can be either relative or absolute.
 
     Returns:
         An entry to append to compile_commands.
-
-    Raises:
-        ValueError: Could not find the extracted file based on relative_path and
-            root_directory or file_directory.
     """
     # The .cmd files are intended to be included directly by Make, so they
     # escape the pound sign '#', either as '\#' or '$(pound)' (depending on the
@@ -90,60 +59,59 @@ def process_line(root_directory, file_directory, command_prefix, relative_path):
     # by Make, so this code replaces the escaped version with '#'.
     prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#')
 
-    cur_dir = root_directory
-    expected_path = os.path.join(cur_dir, relative_path)
-    if not os.path.exists(expected_path):
-        # Try using file_directory instead. Some of the tools have a different
-        # style of .cmd file than the kernel.
-        cur_dir = file_directory
-        expected_path = os.path.join(cur_dir, relative_path)
-        if not os.path.exists(expected_path):
-            raise ValueError('File %s not in %s or %s' %
-                             (relative_path, root_directory, file_directory))
     return {
-        'directory': cur_dir,
-        'file': relative_path,
-        'command': prefix + relative_path,
+        'directory': root_directory,
+        'file': file_path,
+        'command': prefix + file_path,
     }
 
 
 def main():
-    """Walks through the directory and finds and parses .cmd files."""
-    log_level, directory, output = parse_arguments()
-
-    level = getattr(logging, log_level)
-    logging.basicConfig(format='%(levelname)s: %(message)s', level=level)
-
-    filename_matcher = re.compile(_FILENAME_PATTERN)
-    line_matcher = re.compile(_LINE_PATTERN)
+    """Find and parse .cmd files for vmlinux and modules"""
+    ar, output, files = parse_arguments()
+
+    line_matcher = re.compile(r'^cmd_[^ ]*\.o := (.* )([^ ]*\.c)$')
+
+    # Collect objects compiled for vmlinux or modules
+    objects = []
+    for file in files:

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 87

Quoted 10 lines
scripts/gen_compile_commands.py +10 -0
+        if file.endswith('.o'):
+            # Some objects (head-y) are linked to vmlinux directly
+            objects.append(file)
+        elif file.endswith('.a'):
+            # Most of built-in objects are linked via built-in.a or lib.a.
+            # Use 'ar -t' to get the list of the contained objects.
+            objects += subprocess.check_output([ar, '-t', file]).decode().split()
+        elif file.endswith('modules.order'):
+           # modules.order lists all the modules.
+            with open(file) as f:

file is another builtin (or at least was in Python2), perhaps filename?

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 95

Quoted 8 lines
scripts/gen_compile_commands.py +8 -0
+                for line in f:
+                    ko = line.rstrip()
+                    base, ext = os.path.splitext(ko)
+                    if ext != '.ko':
+                        sys.exit('{}: mobule path must end with .ko'.format(ko))
+                    mod = base + '.mod'
+                   # The first line of *.mod lists the objects that
+                   # compose the module.

This comment and the one above it uses tabs for indentation vs spaces for the rest of the file. I use
https://github.com/nickdesaulniers/dotfiles/blob/a90865a9ea48bbefa0082f7508607fdeb361e801/.vimrc#L37-L43
to help me catch these.

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 99

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+                    with open(mod) as mod_f:
+                        objects += mod_f.readline().split()
+        else:
+            sys.exit('{}: unknown file type'.format(file))

Consider breaking up this one long function into multiple, perhaps the above could just return objects?

Quoted 12 lines
scripts/gen_compile_commands.py +0 -10
     compile_commands = []
-    for dirpath, _, filenames in os.walk(directory):
-        for filename in filenames:
-            if not filename_matcher.match(filename):
-                continue
-            filepath = os.path.join(dirpath, filename)
-
-            with open(filepath, 'rt') as f:
-                line = f.readline()
-                result = line_matcher.match(line)
-                if result:

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 104

Quoted 10 lines
scripts/gen_compile_commands.py +3 -7
-                    try:
-                        entry = process_line(directory, dirpath,
-                                             result.group(1), result.group(2))
-                        compile_commands.append(entry)
-                    except ValueError as err:
-                        logging.info('Could not add line from %s: %s',
-                                     filepath, err)
+    cwd = os.getcwd()
+    for object in objects:
+        dir, notdir = os.path.split(object)

object is a builtin Class in python. I'm not sure if it's quite considered a keyword, but maybe a different identifier would be nicer, like object_file or something?

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 108

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+        cmd_file = os.path.join(dir, '.' + notdir + '.cmd')
+        with open(cmd_file, 'rt') as f:
+            line = f.readline()
+            result = line_matcher.match(line)

^ combine statements.

Quoted 16 lines
scripts/gen_compile_commands.py +3 -7
+            if result:
+                entry = process_line(cwd, result.group(1), result.group(2))
+                compile_commands.append(entry)
 
     with open(output, 'wt') as f:
         json.dump(compile_commands, f, indent=2, sort_keys=True)
 
-    count = len(compile_commands)
-    if count < _LOW_COUNT_THRESHOLD:
-        logging.warning(
-            'Found %s entries. Have you compiled the kernel?', count)
-
-
 if __name__ == '__main__':
     main()
--
Quoted 2 lines

2.25.1

Thank you for your assistance and help enabling these tools.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#5 lore raw

Re: [PATCH 2/3] gen_compile_commands: wire up build rule to Makefile

Quoted 43 lines · Nick Desaulniers' via Clang Built Linux
Quoted 9 lines · Masahiro Yamada

Currently, you need to explicitly run scripts/gen_compile_commands.py to create compile_commands.json. It traverses the object tree (you need to pass the -d option to deal with a separate output tree), and parses all the .*.cmd file found.

If you rebuild the kernel over again without 'make clean', stale .*.cmd files from older builds will create invalid entries in compile_commands.json.

Definitely a problem; happy to see compile_commands.json added to make clean target, too.

Quoted 28 lines

This commit wires up the compile_commands.json rule to the top Makefile, and makes it parse .*.cmd files only from the current build to avoid stale entries.

It is possible to extract only relevant .*.cmd files by checking $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order. The objects or archives linked to vmlinux are listed in $(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are listed in modules.order.

You can create compile_commands.json from Make:

  $ make -j$(nproc) CC=clang compile_commands.json

Of course, you can build vmlinux, modules, and compile_commands.json all together in a single command:

  $ make -j$(nproc) CC=clang all compile_commands.json

It works also for M= builds. In this case, compile_commands.json is created in the top directory of the external module.

I hope this will be overall improvements, but it has a drawback; the coverage of the compile_commands.json is reduced because only the objects linked to vmlinux or modules are handled. For example, the following C files are not included in compile_commands.json:

Quoted 8 lines
Quoted 4 lines

- Decompressor source files (arch/*/boot/compressed/)
- VDSO source files
- C files used to generate intermediates (e.g. kernel/bounds.c)
- standalone host programs

Oof, for an x86_64 defconfig, the difference in line count of compile_commands.json
before: 12826

I think some lines of 'before'
are not so important.

Files suffixed with *.mod.c
are generated sources for modules.
There is no point to check them by Clang tools.

Some entries appear twice:

For example, 'before' contains two entries of "file": "lib/cmdline.c"
Which entry is used by 'clang-tidy lib/cmdline.c', the first one, the second one, or both?

Having said that, there is still a loss of more than 3%, yes.

Quoted 5 lines

after: 12351

That's a loss of 475 (3.7% of 12826) coverage. Is there something more we can do to preserve this functionality, while avoiding stale .cmd files?

I have no idea how to do this correctly.

Quoted 2 lines

Is it that those aren't specified by $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) ?

These variables contain only objects and archives
linked to vmlinux.

For example, VDSO is built as a prerequisite of
another object that wraps it.

See line 61 of arch/arm64/kernel/vdso/Makefile:
$(obj)/vdso.o : $(obj)/vdso.so

I do not know how to get the full list of active objects, some of which are built on demand
in the dependency chain.

Idea 1)
Merge this series, and accept the loss.

Idea 2)
Add Makefile targets,
and also keep the previous work-flow.

When you run it from Make,
only objects for vmlinux and modules are handled.

When you need the full coverage, including non-kernel-space sources, run scripts manually:

$ scripts/clang-tools/gen_compile_commands.py
$ scripts/clang-tools/run-clang-tools.py clang-tidy

Idea 3)
Give up supporting it from Makefile.
Instead, improve gen_scripts_commands.py
as a standalone program.

Maybe we can check whether the compiler is Clang or not. We can run '<compiler> --version' and drop the entry if it is GCC.

Usually, the compiler is the first word of
the "command" field in compile_commands.json,
but there are exceptions because
people may do CC="ccache clang".

If there are still stale entries causing troubles, you need to run 'make clean', and rebuild the tree.

We were trying to have separate scripts,
gen_compile_commands.py and run-clang-tools.py,
and to add Makefile targets to run them in a row.

I think unifying the two scripts
might be handier.

Add two options, -t, -a,
to scripts/gen_compile_commands.py

If they are given,
scripts/gen_compilile_commands.py
generates compile_commands.json,
and immediately runs clang-tidy against it.

-t, --tidy
Run 'clang-tidy -checks=-*,linuxkernel-*' after generating
compilation database
-a, --analyzer
Run 'clang-tidy -checks=-*,clang-analyzer-*' after generating
compilation database

Both -a and -t are given,
it runs
'clang-tidy -checks=-*,linuxkernel-*,clang-analyzer-*'

This works more efficiently
if you want to check everything.

'make clang-tidy clang-analyzer'
will invoke clang-tidy twice for each file,
which is not very efficient.

Quoted 9 lines

Makefile PHONY += _emodinst_post · line 1721

Quoted 4 lines

clean-dirs := $(KBUILD_EXTMOD)

Makefile +2 -1
-clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps
+clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps \
+       $(KBUILD_EXTMOD)/compile_commands.json

So the clean target doesn't make use of CLEAN_FILES? It looks like there's some duplication there? Oh, this is dependent on !KBUILD_EXTMOD, and is a new clean target. Do I understand that correctly?

Correct.

We can move CLEAN_FILES to a common part
so external module builds can use it.

Quoted 4 lines
Quoted 4 lines

"""
usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage)

Quoted 8 lines

scripts/gen_compile_commands.py line 29

Quoted 5 lines
scripts/gen_compile_commands.py +2 -3
-    directory_help = ('Path to the kernel source directory to search '
-                      '(defaults to the working directory)')
-    parser.add_argument('-d', '--directory', type=str, help=directory_help)
+    ar_help = 'command used for parsing .a archives'
+    parser.add_argument('-a', '--ar', type=str, default='ar', help=ar_help)

Might be nice to warn if run with no arguments? In case someone does:

$ ./scripts/clang-tools/gen_compile_commands.py

Right.

nargs='+' seems to work.

Quoted 25 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 87

Quoted 13 lines
scripts/gen_compile_commands.py +13 -0
+    # Collect objects compiled for vmlinux or modules
+    objects = []
+    for file in files:
+        if file.endswith('.o'):
+            # Some objects (head-y) are linked to vmlinux directly
+            objects.append(file)
+        elif file.endswith('.a'):
+            # Most of built-in objects are linked via built-in.a or lib.a.
+            # Use 'ar -t' to get the list of the contained objects.
+            objects += subprocess.check_output([ar, '-t', file]).decode().split()
+        elif file.endswith('modules.order'):
+           # modules.order lists all the modules.
+            with open(file) as f:

file is another builtin (or at least was in Python2), perhaps filename?

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 95

Quoted 8 lines
scripts/gen_compile_commands.py +8 -0
+                for line in f:
+                    ko = line.rstrip()
+                    base, ext = os.path.splitext(ko)
+                    if ext != '.ko':
+                        sys.exit('{}: mobule path must end with .ko'.format(ko))
+                    mod = base + '.mod'
+                   # The first line of *.mod lists the objects that
+                   # compose the module.
Quoted 4 lines

This comment and the one above it uses tabs for indentation vs spaces for the rest of the file. I use
https://github.com/nickdesaulniers/dotfiles/blob/a90865a9ea48bbefa0082f7508607fdeb361e801/.vimrc#L37-L43
to help me catch these.

Awesome. Copied to mine.

Quoted 7 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 99

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+                    with open(mod) as mod_f:
+                        objects += mod_f.readline().split()
+        else:
+            sys.exit('{}: unknown file type'.format(file))

Consider breaking up this one long function into multiple, perhaps the above could just return objects?

I thought that returning a big list causes needless memory-copy. If we do not need to be worried too much,
I can make it a helper function.

Quoted 24 lines
Quoted 13 lines

     compile_commands = []
scripts/gen_compile_commands.py +0 -10
-    for dirpath, _, filenames in os.walk(directory):
-        for filename in filenames:
-            if not filename_matcher.match(filename):
-                continue
-            filepath = os.path.join(dirpath, filename)
-
-            with open(filepath, 'rt') as f:
-                line = f.readline()
-                result = line_matcher.match(line)
-                if result:

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 104

Quoted 10 lines
scripts/gen_compile_commands.py +3 -7
-                    try:
-                        entry = process_line(directory, dirpath,
-                                             result.group(1), result.group(2))
-                        compile_commands.append(entry)
-                    except ValueError as err:
-                        logging.info('Could not add line from %s: %s',
-                                     filepath, err)
+    cwd = os.getcwd()
+    for object in objects:
+        dir, notdir = os.path.split(object)
Quoted 3 lines

object is a builtin Class in python. I'm not sure if it's quite considered a keyword, but maybe a different identifier would be nicer, like object_file or something?

Not a keyword, but 'object' is a class, yes.
Not sure about 'file'.

$ python
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import keyword
keyword.iskeyword("import")

True

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("if")

True

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("file")

False

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("object")

False

Quoted 1 line
Quoted 1 line
Quoted 1 line

object

<class 'object'>

Quoted 1 line
Quoted 1 line
Quoted 1 line

file

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'file' is not defined

Quoted 1 line
Quoted 1 line
Quoted 1 line

If this is a problem, I can rename it.

Quoted 6 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 108

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+        cmd_file = os.path.join(dir, '.' + notdir + '.cmd')
+        with open(cmd_file, 'rt') as f:
+            line = f.readline()
+            result = line_matcher.match(line)

^ combine statements.

OK.

Quoted 21 lines
Quoted 16 lines
scripts/gen_compile_commands.py +3 -7
+            if result:
+                entry = process_line(cwd, result.group(1), result.group(2))
+                compile_commands.append(entry)
 
     with open(output, 'wt') as f:
         json.dump(compile_commands, f, indent=2, sort_keys=True)
 
-    count = len(compile_commands)
-    if count < _LOW_COUNT_THRESHOLD:
-        logging.warning(
-            'Found %s entries. Have you compiled the kernel?', count)
-
-
 if __name__ == '__main__':
     main()
--
Quoted 2 lines

2.25.1

Thank you for your assistance and help enabling these tools.

Quoted 8 lines

--
Thanks,
~Nick Desaulniers

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/CAKwvOdkL%3D667%2Bcw_Rxq_5zaOKeTTptsMaxkkSXBic9QxozOWVg%40mail.gmail.com.

--
Best Regards
Masahiro Yamada

#6 lore raw

Re: [PATCH 2/3] gen_compile_commands: wire up build rule to Makefile

Hi Nick,

Quoted 114 lines · Masahiro Yamada
Quoted 43 lines · Nick Desaulniers' via Clang Built Linux
Quoted 9 lines · Masahiro Yamada

Currently, you need to explicitly run scripts/gen_compile_commands.py to create compile_commands.json. It traverses the object tree (you need to pass the -d option to deal with a separate output tree), and parses all the .*.cmd file found.

If you rebuild the kernel over again without 'make clean', stale .*.cmd files from older builds will create invalid entries in compile_commands.json.

Definitely a problem; happy to see compile_commands.json added to make clean target, too.

Quoted 28 lines

This commit wires up the compile_commands.json rule to the top Makefile, and makes it parse .*.cmd files only from the current build to avoid stale entries.

It is possible to extract only relevant .*.cmd files by checking $(KBUILD_VMLINUX_OBJS), $(KBUILD_VMLINUX_LIBS), and modules.order. The objects or archives linked to vmlinux are listed in $(KBUILD_VMLINUX_OBJS) or $(KBUILD_VMLINUX_LIBS). All the modules are listed in modules.order.

You can create compile_commands.json from Make:

  $ make -j$(nproc) CC=clang compile_commands.json

Of course, you can build vmlinux, modules, and compile_commands.json all together in a single command:

  $ make -j$(nproc) CC=clang all compile_commands.json

It works also for M= builds. In this case, compile_commands.json is created in the top directory of the external module.

I hope this will be overall improvements, but it has a drawback; the coverage of the compile_commands.json is reduced because only the objects linked to vmlinux or modules are handled. For example, the following C files are not included in compile_commands.json:

Quoted 8 lines
Quoted 4 lines

- Decompressor source files (arch/*/boot/compressed/)
- VDSO source files
- C files used to generate intermediates (e.g. kernel/bounds.c)
- standalone host programs

Oof, for an x86_64 defconfig, the difference in line count of compile_commands.json
before: 12826

I think some lines of 'before'
are not so important.

Files suffixed with *.mod.c
are generated sources for modules.
There is no point to check them by Clang tools.

Some entries appear twice:

For example, 'before' contains two entries of "file": "lib/cmdline.c"
Which entry is used by 'clang-tidy lib/cmdline.c', the first one, the second one, or both?

Having said that, there is still a loss of more than 3%, yes.

Quoted 5 lines

after: 12351

That's a loss of 475 (3.7% of 12826) coverage. Is there something more we can do to preserve this functionality, while avoiding stale .cmd files?

I have no idea how to do this correctly.

Quoted 2 lines

Is it that those aren't specified by $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) ?

These variables contain only objects and archives
linked to vmlinux.

For example, VDSO is built as a prerequisite of
another object that wraps it.

See line 61 of arch/arm64/kernel/vdso/Makefile:
$(obj)/vdso.o : $(obj)/vdso.so

I do not know how to get the full list of active objects, some of which are built on demand
in the dependency chain.

Idea 1)
Merge this series, and accept the loss.

Idea 2)
Add Makefile targets,
and also keep the previous work-flow.

Quoted 8 lines

When you run it from Make,
only objects for vmlinux and modules are handled.

When you need the full coverage, including non-kernel-space sources, run scripts manually:

$ scripts/clang-tools/gen_compile_commands.py
$ scripts/clang-tools/run-clang-tools.py clang-tidy

Do you have any idea to cope with
the 3% loss problem?

If it is a problem, maybe I should try Idea 2).

Thanks.

Quoted 246 lines

Idea 3)
Give up supporting it from Makefile.
Instead, improve gen_scripts_commands.py
as a standalone program.

Maybe we can check whether the compiler is Clang or not. We can run '<compiler> --version' and drop the entry if it is GCC.

Usually, the compiler is the first word of
the "command" field in compile_commands.json,
but there are exceptions because
people may do CC="ccache clang".

If there are still stale entries causing troubles, you need to run 'make clean', and rebuild the tree.

We were trying to have separate scripts,
gen_compile_commands.py and run-clang-tools.py,
and to add Makefile targets to run them in a row.

I think unifying the two scripts
might be handier.

Add two options, -t, -a,
to scripts/gen_compile_commands.py

If they are given,
scripts/gen_compilile_commands.py
generates compile_commands.json,
and immediately runs clang-tidy against it.

-t, --tidy
Run 'clang-tidy -checks=-*,linuxkernel-*' after generating
compilation database
-a, --analyzer
Run 'clang-tidy -checks=-*,clang-analyzer-*' after generating
compilation database

Both -a and -t are given,
it runs
'clang-tidy -checks=-*,linuxkernel-*,clang-analyzer-*'

This works more efficiently
if you want to check everything.

'make clang-tidy clang-analyzer'
will invoke clang-tidy twice for each file,
which is not very efficient.

Quoted 9 lines

Makefile PHONY += _emodinst_post · line 1721

Quoted 4 lines

clean-dirs := $(KBUILD_EXTMOD)

Makefile +2 -1
-clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps
+clean: rm-files := $(KBUILD_EXTMOD)/Module.symvers $(KBUILD_EXTMOD)/modules.nsdeps \
+       $(KBUILD_EXTMOD)/compile_commands.json

So the clean target doesn't make use of CLEAN_FILES? It looks like there's some duplication there? Oh, this is dependent on !KBUILD_EXTMOD, and is a new clean target. Do I understand that correctly?

Correct.

We can move CLEAN_FILES to a common part
so external module builds can use it.

Quoted 4 lines
Quoted 4 lines

"""
usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage)

Quoted 8 lines

scripts/gen_compile_commands.py line 29

Quoted 5 lines
scripts/gen_compile_commands.py +2 -3
-    directory_help = ('Path to the kernel source directory to search '
-                      '(defaults to the working directory)')
-    parser.add_argument('-d', '--directory', type=str, help=directory_help)
+    ar_help = 'command used for parsing .a archives'
+    parser.add_argument('-a', '--ar', type=str, default='ar', help=ar_help)

Might be nice to warn if run with no arguments? In case someone does:

$ ./scripts/clang-tools/gen_compile_commands.py

Right.

nargs='+' seems to work.

Quoted 25 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 87

Quoted 13 lines
scripts/gen_compile_commands.py +13 -0
+    # Collect objects compiled for vmlinux or modules
+    objects = []
+    for file in files:
+        if file.endswith('.o'):
+            # Some objects (head-y) are linked to vmlinux directly
+            objects.append(file)
+        elif file.endswith('.a'):
+            # Most of built-in objects are linked via built-in.a or lib.a.
+            # Use 'ar -t' to get the list of the contained objects.
+            objects += subprocess.check_output([ar, '-t', file]).decode().split()
+        elif file.endswith('modules.order'):
+           # modules.order lists all the modules.
+            with open(file) as f:

file is another builtin (or at least was in Python2), perhaps filename?

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 95

Quoted 8 lines
scripts/gen_compile_commands.py +8 -0
+                for line in f:
+                    ko = line.rstrip()
+                    base, ext = os.path.splitext(ko)
+                    if ext != '.ko':
+                        sys.exit('{}: mobule path must end with .ko'.format(ko))
+                    mod = base + '.mod'
+                   # The first line of *.mod lists the objects that
+                   # compose the module.
Quoted 4 lines

This comment and the one above it uses tabs for indentation vs spaces for the rest of the file. I use
https://github.com/nickdesaulniers/dotfiles/blob/a90865a9ea48bbefa0082f7508607fdeb361e801/.vimrc#L37-L43
to help me catch these.

Awesome. Copied to mine.

Quoted 7 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 99

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+                    with open(mod) as mod_f:
+                        objects += mod_f.readline().split()
+        else:
+            sys.exit('{}: unknown file type'.format(file))

Consider breaking up this one long function into multiple, perhaps the above could just return objects?

I thought that returning a big list causes needless memory-copy. If we do not need to be worried too much,
I can make it a helper function.

Quoted 24 lines
Quoted 13 lines

     compile_commands = []
scripts/gen_compile_commands.py +0 -10
-    for dirpath, _, filenames in os.walk(directory):
-        for filename in filenames:
-            if not filename_matcher.match(filename):
-                continue
-            filepath = os.path.join(dirpath, filename)
-
-            with open(filepath, 'rt') as f:
-                line = f.readline()
-                result = line_matcher.match(line)
-                if result:

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 104

Quoted 10 lines
scripts/gen_compile_commands.py +3 -7
-                    try:
-                        entry = process_line(directory, dirpath,
-                                             result.group(1), result.group(2))
-                        compile_commands.append(entry)
-                    except ValueError as err:
-                        logging.info('Could not add line from %s: %s',
-                                     filepath, err)
+    cwd = os.getcwd()
+    for object in objects:
+        dir, notdir = os.path.split(object)
Quoted 3 lines

object is a builtin Class in python. I'm not sure if it's quite considered a keyword, but maybe a different identifier would be nicer, like object_file or something?

Not a keyword, but 'object' is a class, yes.
Not sure about 'file'.

$ python
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

import keyword
keyword.iskeyword("import")

True

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("if")

True

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("file")

False

Quoted 1 line
Quoted 1 line
Quoted 1 line

keyword.iskeyword("object")

False

Quoted 1 line
Quoted 1 line
Quoted 1 line

object

<class 'object'>

Quoted 1 line
Quoted 1 line
Quoted 1 line

file

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'file' is not defined

Quoted 1 line
Quoted 1 line
Quoted 1 line

If this is a problem, I can rename it.

Quoted 6 lines

scripts/gen_compile_commands.py def process_line(root_directory, file_directory, command_prefix, relative_path): · line 108

Quoted 4 lines
scripts/gen_compile_commands.py +4 -0
+        cmd_file = os.path.join(dir, '.' + notdir + '.cmd')
+        with open(cmd_file, 'rt') as f:
+            line = f.readline()
+            result = line_matcher.match(line)

^ combine statements.

OK.

Quoted 21 lines
Quoted 16 lines
scripts/gen_compile_commands.py +3 -7
+            if result:
+                entry = process_line(cwd, result.group(1), result.group(2))
+                compile_commands.append(entry)
 
     with open(output, 'wt') as f:
         json.dump(compile_commands, f, indent=2, sort_keys=True)
 
-    count = len(compile_commands)
-    if count < _LOW_COUNT_THRESHOLD:
-        logging.warning(
-            'Found %s entries. Have you compiled the kernel?', count)
-
-
 if __name__ == '__main__':
     main()
--
Quoted 2 lines

2.25.1

Thank you for your assistance and help enabling these tools.

Quoted 8 lines

--
Thanks,
~Nick Desaulniers

--
You received this message because you are subscribed to the Google Groups "Clang Built Linux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/clang-built-linux/CAKwvOdkL%3D667%2Bcw_Rxq_5zaOKeTTptsMaxkkSXBic9QxozOWVg%40mail.gmail.com.

Quoted 3 lines

--
Best Regards
Masahiro Yamada

--
Best Regards
Masahiro Yamada

reviewer Reviewed-by
#7 lore raw

Re: [PATCH 2/3] gen_compile_commands: wire up build rule to Makefile

Quoted 5 lines · Masahiro Yamada

Do you have any idea to cope with
the 3% loss problem?

If it is a problem, maybe I should try Idea 2).

I think it will be ok.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#8 lore raw

PATCH 3/3 Makefile: Add clang-tidy and static analyzer support to makefile

In reply to Masahiro Yamada

From: Nathan Huckleberry <[email protected]>

This patch adds clang-tidy and the clang static-analyzer as make targets. The goal of this patch is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++.

The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease.

===Clang-tidy===

Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST
(https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings.

Here is an example clang-tidy check to report functions that have calls to local_irq_disable without calls to local_irq_enable and vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis. (https://reviews.llvm.org/D65828)

===Clang static analyzer===

The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel.

The clang static analyzer is well documented and designed to be extensible.
(https://clang-analyzer.llvm.org/checker_dev_manual.html)
(https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf)

The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase.

---

 MAINTAINERS                                   |  1 +
 Makefile                                      | 20 ++++-
 .../{ => clang-tools}/gen_compile_commands.py |  0
 scripts/clang-tools/run-clang-tools.py        | 74 +++++++++++++++++++
 4 files changed, 93 insertions(+), 2 deletions(-)
 rename scripts/{ => clang-tools}/gen_compile_commands.py (100%)
 create mode 100755 scripts/clang-tools/run-clang-tools.py
MAINTAINERS +1 -0
index f77df02e4121..7ffddfecc711 100644
@@ -4246,6 +4246,7 @@ W:	https://clangbuiltlinux.github.io/
 B:	https://github.com/ClangBuiltLinux/linux/issues
 C:	irc://chat.freenode.net/clangbuiltlinux
 F:	Documentation/kbuild/llvm.rst
+F:	scripts/clang-tools/
 K:	\b(?i:clang|llvm)\b
 
 CLEANCACHE API
Makefile +18 -2
index 4d65affb6917..95cdbffa29e5 100644
@@ -634,7 +634,7 @@ endif
 # in addition to whatever we do anyway.
 # Just "make" or "make all" shall build modules as well
 
-ifneq ($(filter all modules nsdeps %compile_commands.json,$(MAKECMDGOALS)),)
+ifneq ($(filter all modules nsdeps %compile_commands.json clang-%,$(MAKECMDGOALS)),)
   KBUILD_MODULES := 1
 endif
 
@@ -1572,6 +1572,8 @@ help:
 	@echo  '  export_report   - List the usages of all exported symbols'
 	@echo  '  headerdep       - Detect inclusion cycles in headers'
 	@echo  '  coccicheck      - Check with Coccinelle'
+	@echo  '  clang-analyzer  - Check with clang static analyzer'
+	@echo  '  clang-tidy      - Check with clang-tidy'
 	@echo  ''
 	@echo  'Tools:'
 	@echo  '  nsdeps          - Generate missing symbol namespace dependencies'
@@ -1837,13 +1839,27 @@ nsdeps: modules
 quiet_cmd_gen_compile_commands = GEN     $@
       cmd_gen_compile_commands = $(PYTHON3) $< -a $(AR) -o $@ $(filter-out $<, $(real-prereqs))
 
-$(extmod-prefix)compile_commands.json: scripts/gen_compile_commands.py \
+$(extmod-prefix)compile_commands.json: scripts/clang-tools/gen_compile_commands.py \
 	$(if $(KBUILD_EXTMOD),,$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS)) \
 	$(if $(CONFIG_MODULES), $(MODORDER)) FORCE
 	$(call if_changed,gen_compile_commands)
 
 targets += $(extmod-prefix)compile_commands.json
 
+PHONY += clang-tidy clang-analyzer
+
+ifdef CONFIG_CC_IS_CLANG
+quiet_cmd_clang_tools = CHECK   $<
+      cmd_clang_tools = $(PYTHON3) $(srctree)/scripts/clang-tools/run-clang-tools.py $@ $<
+
+clang-tidy clang-analyzer: $(extmod-prefix)compile_commands.json
+	$(call cmd,clang_tools)
+else
+clang-tidy clang-analyzer:
+	@echo "$@ requires CC=clang" >&2
+	@false
+endif
+
 # Scripts to check various things for consistency
 # ---------------------------------------------------------------------------
 
scripts/clang-tools/gen_compile_commands.py renamed from scripts/gen_compile_commands.py +0 -0
similarity index 100%
rename from scripts/gen_compile_commands.py
rename to scripts/clang-tools/gen_compile_commands.py
scripts/clang-tools/run-clang-tools.py new file +74 -0

74 lines not on this page · raw message

Signature · 1 line
2.25.1
#9 lore raw
In reply to Masahiro Yamada
Quoted 25 lines · Masahiro Yamada

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json
Quoted 7 lines

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I built clang-tidy from the latest source.
I had no idea how to make it work...

How are you building clang-tidy? The clang static-analyzer may not have been built.
I believe the static analyzer is built as a part of clang, not as a part of clang-tools-extra.

I use this command to build.
cmake -DCMAKE_BUILD_TYPE="release" -DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;RISCV" -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;llvm-as" -DLLVM_ENABLE_LLD=1 -G "Ninja" ../llvm

Adding clang to the list of -DLLVM_ENABLE_PROJECTS will build the static analyzer.
-DCLANG_ENABLE_STATIC_ANALYZER=1 might also work, but I haven't tested it.

I tested the patchset and both clang-tidy and clang-analyzer work for me.

Quoted 25 lines

[1] https://patchwork.kernel.org/patch/11687833/

Masahiro Yamada (2):
gen_compile_commands: parse only the first line of .*.cmd files
gen_compile_commands: wire up build rule to Makefile

Nathan Huckleberry (1):
Makefile: Add clang-tidy and static analyzer support to makefile

 MAINTAINERS                                 |   1 +
 Makefile                                    |  45 +++++-
 scripts/clang-tools/gen_compile_commands.py | 117 +++++++++++++++
 scripts/clang-tools/run-clang-tools.py      |  74 ++++++++++
 scripts/gen_compile_commands.py             | 151 --------------------
 5 files changed, 233 insertions(+), 155 deletions(-)
 create mode 100755 scripts/clang-tools/gen_compile_commands.py
 create mode 100755 scripts/clang-tools/run-clang-tools.py
 delete mode 100755 scripts/gen_compile_commands.py

--
2.25.1

#10 lore raw
Quoted 27 lines · Nathan Huckleberry
Quoted 25 lines · Masahiro Yamada

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json
Quoted 4 lines
Quoted 4 lines

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I wasn't able to reproduce Masahiro's reported failure, but seeing as he has GEN for compile_commands.json and I have CHK, I wonder if that's from a run when the series was still under development?

I can reproduce if I run:

$ clang-tidy '-checks='
so maybe was string quoting problem?
Quoted 15 lines
Quoted 3 lines

I built clang-tidy from the latest source.
I had no idea how to make it work...

How are you building clang-tidy? The clang static-analyzer may not have been built.
I believe the static analyzer is built as a part of clang, not as a part of clang-tools-extra.

I use this command to build.
cmake -DCMAKE_BUILD_TYPE="release"

(quoted diff) +0 -3
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;RISCV"
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;llvm-as"
-DLLVM_ENABLE_LLD=1 -G "Ninja" ../llvm
Quoted 5 lines

Adding clang to the list of -DLLVM_ENABLE_PROJECTS will build the static analyzer.
-DCLANG_ENABLE_STATIC_ANALYZER=1 might also work, but I haven't tested it.

I tested the patchset and both clang-tidy and clang-analyzer work for me.

If you rename clang-tidy in your build dir, and ensure you don't have a clang-tidy in your $PATH (which clang-tidy), maybe there's more we can do to politely inform the user they're missing a dependency to execute the make target? Not sure if we could could test that clang-tidy supports the clang-analyzer-* checks. Isn't there an invocation that prints the supported checks? clang-tidy '-checks=*' --list-checks is in my shell history. Maybe grepping that and informing the user how to fix the problem might solve a "papercut?"

If I remove clang-tidy with this series applied, I get (the failure is obvious to me, but...):

```
$ make LLVM=1 -j71 clang-tidy
...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./scripts/clang-tools/run-clang-tools.py", line 54, in run_analysis
    p = subprocess.run(["clang-tidy", "-p", args.path, checks, entry["file"]],
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
    main()
  File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
    pool.map(run_analysis, datastore)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
make: *** [Makefile:1861: clang-tidy] Error 1
```
$ clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111

And I'm not sure you can even build clang or clang-tidy but not the analyzer.

Quoted 26 lines
Quoted 26 lines

[1] https://patchwork.kernel.org/patch/11687833/

Masahiro Yamada (2):
gen_compile_commands: parse only the first line of .*.cmd files
gen_compile_commands: wire up build rule to Makefile

Nathan Huckleberry (1):
Makefile: Add clang-tidy and static analyzer support to makefile

 MAINTAINERS                                 |   1 +
 Makefile                                    |  45 +++++-
 scripts/clang-tools/gen_compile_commands.py | 117 +++++++++++++++
 scripts/clang-tools/run-clang-tools.py      |  74 ++++++++++
 scripts/gen_compile_commands.py             | 151 --------------------
 5 files changed, 233 insertions(+), 155 deletions(-)
 create mode 100755 scripts/clang-tools/gen_compile_commands.py
 create mode 100755 scripts/clang-tools/run-clang-tools.py
 delete mode 100755 scripts/gen_compile_commands.py

--
2.25.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
#11 lore raw
Quoted 3 lines · Nick Desaulniers

I wasn't able to reproduce Masahiro's reported failure, but seeing as he has GEN for compile_commands.json and I have CHK, I wonder if

Nevermind, I misread the output from the build.

Signature · 2 lines
Thanks,
~Nick Desaulniers
#12 lore raw
In reply to Nick Desaulniers
Quoted 111 lines · Nick Desaulniers' via Clang Built Linux
Quoted 27 lines · Nathan Huckleberry
Quoted 25 lines · Masahiro Yamada

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json
Quoted 4 lines
Quoted 4 lines

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I wasn't able to reproduce Masahiro's reported failure, but seeing as he has GEN for compile_commands.json and I have CHK, I wonder if that's from a run when the series was still under development?

I can reproduce if I run:

$ clang-tidy '-checks='
so maybe was string quoting problem?
Quoted 15 lines
Quoted 3 lines

I built clang-tidy from the latest source.
I had no idea how to make it work...

How are you building clang-tidy? The clang static-analyzer may not have been built.
I believe the static analyzer is built as a part of clang, not as a part of clang-tools-extra.

I use this command to build.
cmake -DCMAKE_BUILD_TYPE="release"

(quoted diff) +0 -3
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;RISCV"
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;llvm-as"
-DLLVM_ENABLE_LLD=1 -G "Ninja" ../llvm
Quoted 5 lines

Adding clang to the list of -DLLVM_ENABLE_PROJECTS will build the static analyzer.
-DCLANG_ENABLE_STATIC_ANALYZER=1 might also work, but I haven't tested it.

I tested the patchset and both clang-tidy and clang-analyzer work for me.

If you rename clang-tidy in your build dir, and ensure you don't have a clang-tidy in your $PATH (which clang-tidy), maybe there's more we can do to politely inform the user they're missing a dependency to execute the make target? Not sure if we could could test that clang-tidy supports the clang-analyzer-* checks. Isn't there an invocation that prints the supported checks? clang-tidy '-checks=*' --list-checks is in my shell history. Maybe grepping that and informing the user how to fix the problem might solve a "papercut?"

If I remove clang-tidy with this series applied, I get (the failure is obvious to me, but...):

```
$ make LLVM=1 -j71 clang-tidy
...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./scripts/clang-tools/run-clang-tools.py", line 54, in run_analysis
    p = subprocess.run(["clang-tidy", "-p", args.path, checks, entry["file"]],
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
    main()
  File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
    pool.map(run_analysis, datastore)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
make: *** [Makefile:1861: clang-tidy] Error 1
```
$ clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111
Quoted 1 line

And I'm not sure you can even build clang or clang-tidy but not the analyzer.

I think that is the point of '-DCLANG_ENABLE_STATIC_ANALYZER=OFF'. clang-tools-extra/clang-tidy/CMakeLists.txt has some checks for CLANG_ENABLE_STATIC_ANALYZER to link in certain libraries related to the analyzer.

For the record, tc-build adds that cmake define:

https://github.com/ClangBuiltLinux/tc-build/blob/071eeefd2e201d3f24468cc06ed6a5860161437d/build-llvm.py#L610-L613

$ ../build-llvm.py --build-stage1-only --projects "clang;clang-tools-extra" --targets X86
...
$ ../build/llvm/stage1/bin/clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
0

If I remove that define and rebuild:

$ ../build-llvm.py --build-stage1-only --projects "clang;clang-tools-extra" --targets X86
...
$ ../build/llvm/stage1/bin/clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111

I suppose if this series depends on it, we can remove that from the base defines and either add a flag to enable/disable it depending on people's preferences.

Cheers,
Nathan

#13 lore raw
Quoted 135 lines · Nathan Chancellor
Quoted 111 lines · Nick Desaulniers' via Clang Built Linux
Quoted 27 lines · Nathan Huckleberry
Quoted 25 lines · Masahiro Yamada

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json
Quoted 4 lines
Quoted 4 lines

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I wasn't able to reproduce Masahiro's reported failure, but seeing as he has GEN for compile_commands.json and I have CHK, I wonder if that's from a run when the series was still under development?

I can reproduce if I run:

$ clang-tidy '-checks='
so maybe was string quoting problem?
Quoted 15 lines
Quoted 3 lines

I built clang-tidy from the latest source.
I had no idea how to make it work...

How are you building clang-tidy? The clang static-analyzer may not have been built.
I believe the static analyzer is built as a part of clang, not as a part of clang-tools-extra.

I use this command to build.
cmake -DCMAKE_BUILD_TYPE="release"

(quoted diff) +0 -3
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;RISCV"
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;llvm-as"
-DLLVM_ENABLE_LLD=1 -G "Ninja" ../llvm
Quoted 5 lines

Adding clang to the list of -DLLVM_ENABLE_PROJECTS will build the static analyzer.
-DCLANG_ENABLE_STATIC_ANALYZER=1 might also work, but I haven't tested it.

I tested the patchset and both clang-tidy and clang-analyzer work for me.

If you rename clang-tidy in your build dir, and ensure you don't have a clang-tidy in your $PATH (which clang-tidy), maybe there's more we can do to politely inform the user they're missing a dependency to execute the make target? Not sure if we could could test that clang-tidy supports the clang-analyzer-* checks. Isn't there an invocation that prints the supported checks? clang-tidy '-checks=*' --list-checks is in my shell history. Maybe grepping that and informing the user how to fix the problem might solve a "papercut?"

If I remove clang-tidy with this series applied, I get (the failure is obvious to me, but...):

```
$ make LLVM=1 -j71 clang-tidy
...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./scripts/clang-tools/run-clang-tools.py", line 54, in run_analysis
    p = subprocess.run(["clang-tidy", "-p", args.path, checks, entry["file"]],
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
    main()
  File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
    pool.map(run_analysis, datastore)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
make: *** [Makefile:1861: clang-tidy] Error 1
```
$ clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111
Quoted 1 line

And I'm not sure you can even build clang or clang-tidy but not the analyzer.

I think that is the point of '-DCLANG_ENABLE_STATIC_ANALYZER=OFF'. clang-tools-extra/clang-tidy/CMakeLists.txt has some checks for CLANG_ENABLE_STATIC_ANALYZER to link in certain libraries related to the analyzer.

For the record, tc-build adds that cmake define:

https://github.com/ClangBuiltLinux/tc-build/blob/071eeefd2e201d3f24468cc06ed6a5860161437d/build-llvm.py#L610-L613

$ ../build-llvm.py --build-stage1-only --projects "clang;clang-tools-extra" --targets X86
...
$ ../build/llvm/stage1/bin/clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
0

If I remove that define and rebuild:

$ ../build-llvm.py --build-stage1-only --projects "clang;clang-tools-extra" --targets X86
...
Quoted 6 lines
$ ../build/llvm/stage1/bin/clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111

I suppose if this series depends on it, we can remove that from the base defines and either add a flag to enable/disable it depending on people's preferences.

Ah, right.

I used tc-build to build clang, clang-tools-extra.

I will remove 'CLANG_ENABLE_STATIC_ANALYZER': 'OFF',
and rebuild clang-tools-extra.
Thanks.

Signature · 2 lines
Best Regards
Masahiro Yamada
#14 lore raw
In reply to Nick Desaulniers
Quoted 112 lines · Nick Desaulniers
Quoted 27 lines · Nathan Huckleberry
Quoted 25 lines · Masahiro Yamada

I improved gen_compile_commands.py in the first two patches, then rebased Nathan's v7 [1] on top of them.
To save time, I modified the Makefile part.
No change for run-clang-tools.py

I am not sure if the new directory, scripts/clang-tools/, is worth creating only for 2 files, but I do not have a strong opinion about it.

"make clang-tidy" should work in-tree build, out-of-tree build (O=), and external module build (M=). Tests and reviews are appreciated.

"make clang-tidy" worked for me.

masahiro@oscar:~/workspace/linux-kbuild$ make -j24 CC=clang clang-tidy
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  GEN     compile_commands.json
  CHECK   compile_commands.json
Quoted 4 lines
Quoted 4 lines

But "make clang-analyzer" just sprinkled the following error:

Error: no checks enabled.
USAGE: clang-tidy [options] <source0> [... <sourceN>]

I wasn't able to reproduce Masahiro's reported failure, but seeing as he has GEN for compile_commands.json and I have CHK, I wonder if that's from a run when the series was still under development?

I can reproduce if I run:

$ clang-tidy '-checks='
so maybe was string quoting problem?
Quoted 15 lines
Quoted 3 lines

I built clang-tidy from the latest source.
I had no idea how to make it work...

How are you building clang-tidy? The clang static-analyzer may not have been built.
I believe the static analyzer is built as a part of clang, not as a part of clang-tools-extra.

I use this command to build.
cmake -DCMAKE_BUILD_TYPE="release"

(quoted diff) +0 -3
-DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM;RISCV"
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;llvm-as"
-DLLVM_ENABLE_LLD=1 -G "Ninja" ../llvm
Quoted 5 lines

Adding clang to the list of -DLLVM_ENABLE_PROJECTS will build the static analyzer.
-DCLANG_ENABLE_STATIC_ANALYZER=1 might also work, but I haven't tested it.

I tested the patchset and both clang-tidy and clang-analyzer work for me.

If you rename clang-tidy in your build dir, and ensure you don't have a clang-tidy in your $PATH (which clang-tidy), maybe there's more we can do to politely inform the user they're missing a dependency to execute the make target? Not sure if we could could test that clang-tidy supports the clang-analyzer-* checks. Isn't there an invocation that prints the supported checks? clang-tidy '-checks=*' --list-checks is in my shell history. Maybe grepping that and informing the user how to fix the problem might solve a "papercut?"

If I remove clang-tidy with this series applied, I get (the failure is obvious to me, but...):

```
$ make LLVM=1 -j71 clang-tidy
...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./scripts/clang-tools/run-clang-tools.py", line 54, in run_analysis
    p = subprocess.run(["clang-tidy", "-p", args.path, checks, entry["file"]],
  File "/usr/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./scripts/clang-tools/run-clang-tools.py", line 74, in <module>
    main()
  File "./scripts/clang-tools/run-clang-tools.py", line 70, in main
    pool.map(run_analysis, datastore)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'clang-tidy'
make: *** [Makefile:1861: clang-tidy] Error 1
```
$ clang-tidy '-checks=*' --list-checks | grep clang-analyzer | wc -l
111
Quoted 1 line

And I'm not sure you can even build clang or clang-tidy but not the analyzer.

I rebuilt clang-tools-extra with -DCLANG_ENABLE_STATIC_ANALYZER on, and now clang-analyzer worked for me. :)

Perhaps, we could do some checks for clang-tidy.
I am OK with leaving it as a follow-up work.

Signature · 2 lines
Best Regards
Masahiro Yamada