gen_compile_commands: Add support for separate kbuild output directory
PATCH 1/1 gen_compile_commands: Add support for separate kbuild output directory
Add support for builds that use an output directory different than the kernel source tree (e.g. make O=/tmp/kernel-obj). This also introduces support for .cmd files that use absolute paths.
Previously, gen_compile_commands.py only supported builds where the kernel source tree and the output directory were the same:
$ make defconfig $ make -j32 $ ./scripts/gen_compile_commands.py
gen_compile_commands.py had flags to support out of tree use, but the generated compile_commands.json file still assumed that the source tree and kbuild output directory were the same.
Now, the following cases are supported as well:
- Absolute output path:
$ mkdir /tmp/kernel-obj $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -k /tmp/kernel-obj
- Relative output path:
$ mkdir kernel-obj $ make O=kernel-obj/ defconfig $ make O=kernel-obj/ -j32 $ ./scripts/gen_compile_commands.py -k kernel-obj
The new argument, -k, is introduced in a way that makes the script backward compatible with how -d was previously used.
--- scripts/gen_compile_commands.py | 112 ++++++++++++++++++++++---------- 1 file changed, 77 insertions(+), 35 deletions(-)
gen_compile_commands.py +77 -35
@@ -31,16 +31,24 @@ def parse_arguments(): Returns: log_level: A logging level to filter log output. - directory: The directory to search for .cmd files. + source_directory: The directory of the kernel source tree. + kbuild_output_directory: The directory to search for .cmd files. output: Where to write the compile-commands JSON file. """ usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage) - directory_help = ('Path to the kernel source directory to search ' + directory_help = ('Path to the kernel source directory ' '(defaults to the working directory)') parser.add_argument('-d', '--directory', type=str, help=directory_help) + kbuild_output_directory_help = ( + 'Path to the kernel output directory to search for .cmd files ' + '(defaults to match --directory)') + parser.add_argument( + '-k', '--kbuild-output-directory', type=str, + help=kbuild_output_directory_help) + output_help = ('The location to write compile_commands.json (defaults to ' 'compile_commands.json in the search directory)') parser.add_argument('-o', '--output', type=str, help=output_help)
@@ -58,58 +66,91 @@ def parse_arguments(): if log_level not in _VALID_LOG_LEVELS: raise ValueError('%s is not a valid log level' % log_level) - directory = args.directory or os.getcwd() - output = args.output or os.path.join(directory, _DEFAULT_OUTPUT) - directory = os.path.abspath(directory) + source_directory = args.directory or os.getcwd() + kbuild_output_directory = args.kbuild_output_directory or source_directory + output = args.output or os.path.join(source_directory, _DEFAULT_OUTPUT) + source_directory = os.path.abspath(source_directory) + kbuild_output_directory = os.path.abspath(kbuild_output_directory) - return log_level, directory, output + return log_level, source_directory, kbuild_output_directory, output -def process_line(root_directory, file_directory, command_prefix, relative_path): +def process_line(src_dir, kbuild_out_dir, file_dir, cmd_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. Args: - root_directory: The directory that was searched for .cmd files. Usually + src_dir: The directory of the kernel source tree. + kbuild_out_dir: The directory that was searched for .cmd files. Usually used directly in the "directory" entry in compile_commands.json. - file_directory: The path to the directory the .cmd file was found in. - command_prefix: The extracted command line, up to the last element. - relative_path: The .c file from the end of the extracted command. - Usually relative to root_directory, but sometimes relative to - file_directory and sometimes neither. + file_dir: The path to the directory the .cmd file was found in. + cmd_prefix: The extracted command line, up to the last element. + file_path: The .c file from the end of the extracted command. + Usually relative to kbuild_out_dir, but sometimes relative to + src_dir and sometimes neither. Returns: An entry to append to compile_commands. Raises: - ValueError: Could not find the extracted file based on relative_path and - root_directory or file_directory. + ValueError: Could not find the extracted file. """ # The .cmd files are intended to be included directly by Make, so they # escape the pound sign '#', either as '\#' or '$(pound)' (depending on the - # kernel version). The compile_commands.json file is not interepreted + # kernel version). The compile_commands.json file is not interpreted # by Make, so this code replaces the escaped version with '#'. - prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - - cur_dir = root_directory - expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - # Try using file_directory instead. Some of the tools have a different - # style of .cmd file than the kernel. - cur_dir = file_directory - expected_path = os.path.join(cur_dir, relative_path) + prefix = cmd_prefix.replace('\#', '#').replace('$(pound)', '#') + + # Compile commands are usually run in the top level of the kbuild output + # directory + working_dir = kbuild_out_dir + + if os.path.isabs(file_path): + # This is the most common case when the kbuild output directory is + # outside the kernel source tree (e.g. cd src/linux; + # make O=/tmp/kernel-obj). In this case, the command is run in + # kbuild_out_dir, and file_path is an absolute path to the file being + # compiled. + if not os.path.exists(file_path): + raise ValueError('File %s does not exist' % file_path) + else: + # Otherwise, try to locate the file using its relative path. + # + # First, check for the file relative to kbuild_out_dir. This is the most + # common case when output directory is the same as the kernel source + # directory, or if the output directory is specified using a relative + # path (e.g. make, or make O=kernel-obj/) + expected_path = os.path.join(kbuild_out_dir, file_path) + if not os.path.exists(expected_path): - raise ValueError('File %s not in %s or %s' % - (relative_path, root_directory, file_directory)) + # Try using file_dir instead. Some of the tools have a different + # style of .cmd file than the kernel. In this case, the command is + # run in a subdirectory of the kernel source tree. The subdirectory + # will match the directory the cmd file was found in (e.g. + # /tmp/kernel-obj/tools/objtool/.weak.o.cmd contains a command + # that's run in src/linux/tools/objtool/) + + # Translate file_dir to a relative path, and use the relative path + # to locate where in the kernel source tree the command may have + # been executed. + relative_file_dir = os.path.relpath(file_dir, kbuild_out_dir) + working_dir = os.path.join(src_dir, relative_file_dir) + expected_path = os.path.join(working_dir, file_path) + + if not os.path.exists(expected_path): + # At this point, failures are often from tools/objtool/ + # and tools/lib/subcmd/ + raise ValueError('File %s not in %s or %s' % + (file_path, kbuild_out_dir, file_dir)) return { - 'directory': cur_dir, - 'file': relative_path, - 'command': prefix + relative_path, + 'directory': working_dir, + 'file': file_path, + 'command': prefix + file_path, } def main(): """Walks through the directory and finds and parses .cmd files.""" - log_level, directory, output = parse_arguments() + log_level, src_dir, kbuild_out_dir, out_file = parse_arguments() level = getattr(logging, log_level) logging.basicConfig(format='%(levelname)s: %(message)s', level=level)
@@ -118,7 +159,7 @@ def main(): line_matcher = re.compile(_LINE_PATTERN) compile_commands = [] - for dirpath, _, filenames in os.walk(directory): + for dirpath, _, filenames in os.walk(kbuild_out_dir): for filename in filenames: if not filename_matcher.match(filename): continue
@@ -131,14 +172,15 @@ def main(): continue try: - entry = process_line(directory, dirpath, - result.group(1), result.group(2)) + entry = process_line(src_dir, kbuild_out_dir, + dirpath, result.group(1), + result.group(2)) compile_commands.append(entry) except ValueError as err: logging.info('Could not add line from %s: %s', filepath, err) - with open(output, 'wt') as f: + with open(out_file, 'wt') as f: json.dump(compile_commands, f, indent=2, sort_keys=True) count = len(compile_commands)
Signature · 1 line
2.28.0.163.g6104cc2f0b6-goog
Adding the kbuild mailing list on CC, which owns this directory.
Quoted 56 lines · Peter Kalauskas
Add support for builds that use an output directory different than the kernel source tree (e.g. make O=/tmp/kernel-obj). This also introduces support for .cmd files that use absolute paths.
Previously,
gen_compile_commands.pyonly supported builds where the kernel source tree and the output directory were the same:$ make defconfig $ make -j32 $ ./scripts/gen_compile_commands.py
gen_compile_commands.pyhad flags to support out of tree use, but the generated compile_commands.json file still assumed that the source tree and kbuild output directory were the same.Now, the following cases are supported as well:
- Absolute output path:
$ mkdir /tmp/kernel-obj $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -k /tmp/kernel-obj- Relative output path:
$ mkdir kernel-obj $ make O=kernel-obj/ defconfig $ make O=kernel-obj/ -j32 $ ./scripts/gen_compile_commands.py -k kernel-objThe new argument, -k, is introduced in a way that makes the script backward compatible with how -d was previously used.
Signed-off-by Peter Kalauskas <[email protected]>---
scripts/gen_compile_commands.py| 112 ++++++++++++++++++++++----------1 file changed, 77 insertions(+), 35 deletions(-)gen_compile_commands.py +3
-2@@ -31,16 +31,24 @@ def parse_arguments(): Returns: log_level: A logging level to filter log output. - directory: The directory to search for .cmd files. + source_directory: The directory of the kernel source tree. + kbuild_output_directory: The directory to search for .cmd files. output: Where to write the compile-commands JSON file. """ usage = 'Creates a compile_commands.json database from kernel .cmd files' parser = argparse.ArgumentParser(description=usage) - directory_help = ('Path to the kernel source directory to search ' + directory_help = ('Path to the kernel source directory ' '(defaults to the working directory)') parser.add_argument('-d', '--directory', type=str, help=directory_help)
gen_compile_commands.py def parse_arguments(): · line 53
Quoted 10 lines
gen_compile_commands.py +7
-0+ kbuild_output_directory_help = ( + 'Path to the kernel output directory to search for .cmd files ' + '(defaults to match --directory)') + parser.add_argument( + '-k', '--kbuild-output-directory', type=str, + help=kbuild_output_directory_help) + output_help = ('The location to write compile_commands.json (defaults to ' 'compile_commands.json in the search directory)')
Now that there are two directory options, I think the text here should match the name of the option more closely. How about "defaults to compile_commands.json in --kbuild-output-directory"?
gen_compile_commands.py def parse_arguments(): · line 71
Quoted 11 lines
gen_compile_commands.py +3
-3parser.add_argument('-o', '--output', type=str, help=output_help)@@ -58,58 +66,91 @@ def parse_arguments(): if log_level not in _VALID_LOG_LEVELS: raise ValueError('%s is not a valid log level' % log_level) - directory = args.directory or os.getcwd() - output = args.output or os.path.join(directory, _DEFAULT_OUTPUT) - directory = os.path.abspath(directory) + source_directory = args.directory or os.getcwd() + kbuild_output_directory = args.kbuild_output_directory or source_directory + output = args.output or os.path.join(source_directory, _DEFAULT_OUTPUT)
I'm not as familiar with out-of-tree options for compile_commands.json; should the default be in the source tree or should it be in the object tree? I assumed in my previous comment that the right place by default was the object tree, but this line suggests that the default will be the source tree if the two differ.
Quoted 43 lines
gen_compile_commands.py +13
-14+ source_directory = os.path.abspath(source_directory) + kbuild_output_directory = os.path.abspath(kbuild_output_directory) - return log_level, directory, output + return log_level, source_directory, kbuild_output_directory, output -def process_line(root_directory, file_directory, command_prefix, relative_path): +def process_line(src_dir, kbuild_out_dir, file_dir, cmd_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. Args: - root_directory: The directory that was searched for .cmd files. Usually + src_dir: The directory of the kernel source tree. + kbuild_out_dir: The directory that was searched for .cmd files. Usually used directly in the "directory" entry in compile_commands.json. - file_directory: The path to the directory the .cmd file was found in. - command_prefix: The extracted command line, up to the last element. - relative_path: The .c file from the end of the extracted command. - Usually relative to root_directory, but sometimes relative to - file_directory and sometimes neither. + file_dir: The path to the directory the .cmd file was found in. + cmd_prefix: The extracted command line, up to the last element. + file_path: The .c file from the end of the extracted command. + Usually relative to kbuild_out_dir, but sometimes relative to + src_dir and sometimes neither. Returns: An entry to append to compile_commands. Raises: - ValueError: Could not find the extracted file based on relative_path and - root_directory or file_directory. + ValueError: Could not find the extracted file. """ # The .cmd files are intended to be included directly by Make, so they # escape the pound sign '#', either as '\#' or '$(pound)' (depending on the - # kernel version). The compile_commands.json file is not interepreted + # kernel version). The compile_commands.json file is not interpreted # by Make, so this code replaces the escaped version with '#'. - prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - - cur_dir = root_directory
gen_compile_commands.py def parse_arguments(): · line 104
Quoted 10 lines
gen_compile_commands.py +4
-6- expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - # Try using file_directory instead. Some of the tools have a different - # style of .cmd file than the kernel. - cur_dir = file_directory - expected_path = os.path.join(cur_dir, relative_path) + prefix = cmd_prefix.replace('\#', '#').replace('$(pound)', '#') + + # Compile commands are usually run in the top level of the kbuild output + # directory
nit: for consistency with the rest of the comments in this file, please add punctuation to the end of this sentence and subsequent ones.
Quoted 89 lines
gen_compile_commands.py +41
-5+ working_dir = kbuild_out_dir + + if os.path.isabs(file_path): + # This is the most common case when the kbuild output directory is + # outside the kernel source tree (e.g. cd src/linux; + # make O=/tmp/kernel-obj). In this case, the command is run in + # kbuild_out_dir, and file_path is an absolute path to the file being + # compiled. + if not os.path.exists(file_path): + raise ValueError('File %s does not exist' % file_path) + else: + # Otherwise, try to locate the file using its relative path. + # + # First, check for the file relative to kbuild_out_dir. This is the most + # common case when output directory is the same as the kernel source + # directory, or if the output directory is specified using a relative + # path (e.g. make, or make O=kernel-obj/) + expected_path = os.path.join(kbuild_out_dir, file_path) + if not os.path.exists(expected_path): - raise ValueError('File %s not in %s or %s' % - (relative_path, root_directory, file_directory)) + # Try using file_dir instead. Some of the tools have a different + # style of .cmd file than the kernel. In this case, the command is + # run in a subdirectory of the kernel source tree. The subdirectory + # will match the directory the cmd file was found in (e.g. + # /tmp/kernel-obj/tools/objtool/.weak.o.cmd contains a command + # that's run in src/linux/tools/objtool/) + + # Translate file_dir to a relative path, and use the relative path + # to locate where in the kernel source tree the command may have + # been executed. + relative_file_dir = os.path.relpath(file_dir, kbuild_out_dir) + working_dir = os.path.join(src_dir, relative_file_dir) + expected_path = os.path.join(working_dir, file_path) + + if not os.path.exists(expected_path): + # At this point, failures are often from tools/objtool/ + # and tools/lib/subcmd/ + raise ValueError('File %s not in %s or %s' % + (file_path, kbuild_out_dir, file_dir)) return { - 'directory': cur_dir, - 'file': relative_path, - 'command': prefix + relative_path, + 'directory': working_dir, + 'file': file_path, + 'command': prefix + file_path, }def
main():
"""Walks through the directory and finds and parses .cmd files."""gen_compile_commands.py +6
-5- log_level, directory, output = parse_arguments() + log_level, src_dir, kbuild_out_dir, out_file = parse_arguments() level = getattr(logging, log_level) logging.basicConfig(format='%(levelname)s: %(message)s', level=level)@@ -118,7 +159,7 @@ def main(): line_matcher = re.compile(_LINE_PATTERN) compile_commands = [] - for dirpath, _, filenames in os.walk(directory): + for dirpath, _, filenames in os.walk(kbuild_out_dir): for filename in filenames: if not filename_matcher.match(filename): continue@@ -131,14 +172,15 @@ def main(): continue try: - entry = process_line(directory, dirpath, - result.group(1), result.group(2)) + entry = process_line(src_dir, kbuild_out_dir, + dirpath, result.group(1), + result.group(2)) compile_commands.append(entry) except ValueError as err: logging.info('Could not add line from %s: %s', filepath, err) - with open(output, 'wt') as f: + with open(out_file, 'wt') as f: json.dump(compile_commands, f, indent=2, sort_keys=True) count = len(compile_commands)Signature · 1 line
2.28.0.163.g6104cc2f0b6-goog
(+cc clang-built-linux)
Quoted 7 lines · Peter Kalauskas
Add support for builds that use an output directory different than the kernel source tree (e.g. make O=/tmp/kernel-obj). This also introduces support for .cmd files that use absolute paths.
Previously,
gen_compile_commands.pyonly supported builds where the kernel source tree and the output directory were the same:
Quoted 7 lines
$ make defconfig $ make -j32 $ ./scripts/gen_compile_commands.py
gen_compile_commands.pyhad flags to support out of tree use, but the generated compile_commands.json file still assumed that the source tree and kbuild output directory were the same.
Thanks Tom for CCing Kbuild ML.
Otherwise, I would not have noticed this patch.
I think the current code _mostly_ supports the out of tree use. The key is to specify '-p <build-dir>'
when you use compile_commands.json from
clang-check, clang-tidy, etc.
Quoted 8 lines
Now, the following cases are supported as well:
- Absolute output path:
$ mkdir /tmp/kernel-obj $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -k /tmp/kernel-obj
Quoted 5 lines
- Relative output path:
$ mkdir kernel-obj $ make O=kernel-obj/ defconfig $ make O=kernel-obj/ -j32 $ ./scripts/gen_compile_commands.py -k kernel-obj
In the current script, I would do like follows:
- Absolute output path:
$ export CC=clang $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -d /tmp/kernel-obj $ clang-tidy '--checks=linuxkernel*' -p /tmp/kernel-obj kernel/fork.c
- Relative output path:
$ export CC=clang $ make O=kernel-obj defconfig $ make O=kernel-obj -j32 $ ./scripts/gen_compile_commands.py -d kernel-obj $ clang-tidy '--checks=linuxkernel*' -p kernel-obj kernel/fork.c
With your patch and the -k option use, compile_commands.json is always created in the source tree whether O= is given or not.
Then, you no longer need to pass '-p <build-path>' from clang tools.
However, I like the workflow to create any build artifact in the output tree for O= use-case, and keep the source tree completely clean.
This is because the source tree might be read-only. Perhaps, it is located under /usr/src/,
or the source code might be provided by DVD-ROM, etc.
In my understanding, the flaw of the -d option is, it cannot handle objtool.
'--log_level DEBUG' emits error log for objtool.
masahiro@oscar:~/ref/linux$ ./scripts/gen_compile_commands.py -d /tmp/kernel-obj --log_level DEBUG INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.elf.o.cmd: File elf.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.builtin-orc.o.cmd: File builtin-orc.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.builtin-check.o.cmd: File builtin-check.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.orc_dump.o.cmd: File orc_dump.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.libstring.o.cmd: File ../lib/string.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.libctype.o.cmd: File ../lib/ctype.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.parse-options.o.cmd: File parse-options.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.objtool.o.cmd: File objtool.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.fixdep.o.cmd: File fixdep.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.pager.o.cmd: File pager.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.str_error_r.o.cmd: File ../lib/str_error_r.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.special.o.cmd: File special.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.orc_gen.o.cmd: File orc_gen.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.help.o.cmd: File help.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.librbtree.o.cmd: File ../lib/rbtree.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.run-command.o.cmd: File run-command.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.exec-cmd.o.cmd: File exec-cmd.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.weak.o.cmd: File weak.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.subcmd-config.o.cmd: File subcmd-config.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.check.o.cmd: File check.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.sigchain.o.cmd: File sigchain.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/arch/x86/.decode.o.cmd: File arch/x86/decode.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool/arch/x86
Your patch solves it, but I wonder if it is worth lots of code addition. objtool is the only unfortunate case because it did not join Kbuild.
Quoted 47 lines
The new argument, -k, is introduced in a way that makes the script backward compatible with how -d was previously used.
Signed-off-by Peter Kalauskas <[email protected]>---
gen_compile_commands.py +10
-16-def process_line(root_directory, file_directory, command_prefix, relative_path): +def process_line(src_dir, kbuild_out_dir, file_dir, cmd_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. Args: - root_directory: The directory that was searched for .cmd files. Usually + src_dir: The directory of the kernel source tree. + kbuild_out_dir: The directory that was searched for .cmd files. Usually used directly in the "directory" entry in compile_commands.json. - file_directory: The path to the directory the .cmd file was found in. - command_prefix: The extracted command line, up to the last element. - relative_path: The .c file from the end of the extracted command. - Usually relative to root_directory, but sometimes relative to - file_directory and sometimes neither. + file_dir: The path to the directory the .cmd file was found in. + cmd_prefix: The extracted command line, up to the last element. + file_path: The .c file from the end of the extracted command. + Usually relative to kbuild_out_dir, but sometimes relative to + src_dir and sometimes neither. Returns: An entry to append to compile_commands. Raises: - ValueError: Could not find the extracted file based on relative_path and - root_directory or file_directory. + ValueError: Could not find the extracted file. """ # The .cmd files are intended to be included directly by Make, so they # escape the pound sign '#', either as '\#' or '$(pound)' (depending on the - # kernel version). The compile_commands.json file is not interepreted + # kernel version). The compile_commands.json file is not interpreted # by Make, so this code replaces the escaped version with '#'. - prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - - cur_dir = root_directory - expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - # Try using file_directory instead. Some of the tools have a different
gen_compile_commands.py def parse_arguments(): · line 107
Quoted 10 lines
gen_compile_commands.py +7
-3- # style of .cmd file than the kernel. - cur_dir = file_directory - expected_path = os.path.join(cur_dir, relative_path) + prefix = cmd_prefix.replace('\#', '#').replace('$(pound)', '#') + + # Compile commands are usually run in the top level of the kbuild output + # directory + working_dir = kbuild_out_dir + + if os.path.isabs(file_path):
I might be misreading the code, but
is this if-else switch needed?
os.path.join() returns the last parameter as-is
if it is already absolute, right?
This is my quick experiment:
masahiro@oscar:~$ python3 Python 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
import os
os.path.join("/foo/bar/", "a/b/c")
'/foo/bar/a/b/c'
Quoted 1 line
Quoted 1 line
Quoted 1 line
os.path.join("/foo/bar/", "/a/b/c")
'/a/b/c'
So, the current code:
expected_path = os.path.join(cur_dir, relative_path)
works whether 'relative_path' is relative or not.
I was also thinking this variable name was misleading since 'relative_path' may be actually absolute.
--
Best Regards
Masahiro Yamada
Quoted 24 lines · Masahiro Yamada
(+cc clang-built-linux)
Quoted 7 lines · Peter Kalauskas
Add support for builds that use an output directory different than the kernel source tree (e.g. make O=/tmp/kernel-obj). This also introduces support for .cmd files that use absolute paths.
Previously,
gen_compile_commands.pyonly supported builds where the kernel source tree and the output directory were the same:Quoted 7 lines
$ make defconfig $ make -j32 $ ./scripts/gen_compile_commands.py
gen_compile_commands.pyhad flags to support out of tree use, but the generated compile_commands.json file still assumed that the source tree and kbuild output directory were the same.Thanks Tom for CCing
KbuildML.
Otherwise, I would not have noticed this patch.
Quoted 5 lines
I think the current code _mostly_ supports the out of tree use. The key is to specify '-p <build-dir>'
when you use compile_commands.json from
clang-check, clang-tidy, etc.
Thanks, I wasn't aware of the -p option for clang-tidy. When I wrote this, I was trying to use the compile_commands.json with YCM for vim. I see now that I can easily change the build dir for that use-case as well by specifying:
:let g:ycm_clangd_args=['--compile-commands-dir=kernel-obj']
Quoted 136 lines
Quoted 12 lines
Now, the following cases are supported as well:
- Absolute output path:
$ mkdir /tmp/kernel-obj $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -k /tmp/kernel-objQuoted 5 lines
- Relative output path:
$ mkdir kernel-obj $ make O=kernel-obj/ defconfig $ make O=kernel-obj/ -j32 $ ./scripts/gen_compile_commands.py -k kernel-objIn the current script, I would do like follows:
- Absolute output path:
$ export CC=clang $ make O=/tmp/kernel-obj defconfig $ make O=/tmp/kernel-obj -j32 $ ./scripts/gen_compile_commands.py -d /tmp/kernel-obj $ clang-tidy '--checks=linuxkernel*' -p /tmp/kernel-obj kernel/fork.c- Relative output path:
$ export CC=clang $ make O=kernel-obj defconfig $ make O=kernel-obj -j32 $ ./scripts/gen_compile_commands.py -d kernel-obj $ clang-tidy '--checks=linuxkernel*' -p kernel-obj kernel/fork.cWith your patch and the -k option use, compile_commands.json is always created in the source tree whether O= is given or not.
Then, you no longer need to pass '-p <build-path>' from clang tools.However, I like the workflow to create any build artifact in the output tree for O= use-case, and keep the source tree completely clean.
This is because the source tree might be read-only. Perhaps, it is located under/usr/src/,
or the source code might be provided by DVD-ROM, etc.In my understanding, the flaw of the -d option is, it cannot handle objtool.
'--log_level DEBUG' emits error log for objtool.
masahiro@oscar:~/ref/linux$ ./scripts/gen_compile_commands.py -d /tmp/kernel-obj --log_level DEBUG INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.elf.o.cmd: File elf.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.builtin-orc.o.cmd: File builtin-orc.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.builtin-check.o.cmd: File builtin-check.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.orc_dump.o.cmd: File orc_dump.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.libstring.o.cmd: File ../lib/string.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.libctype.o.cmd: File ../lib/ctype.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.parse-options.o.cmd: File parse-options.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.objtool.o.cmd: File objtool.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.fixdep.o.cmd: File fixdep.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.pager.o.cmd: File pager.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.str_error_r.o.cmd: File ../lib/str_error_r.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.special.o.cmd: File special.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.orc_gen.o.cmd: File orc_gen.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.help.o.cmd: File help.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.librbtree.o.cmd: File ../lib/rbtree.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.run-command.o.cmd: File run-command.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.exec-cmd.o.cmd: File exec-cmd.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.weak.o.cmd: File weak.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.subcmd-config.o.cmd: File subcmd-config.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.check.o.cmd: File check.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/.sigchain.o.cmd: File sigchain.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool INFO: Could not add line from /tmp/kernel-obj/tools/objtool/arch/x86/.decode.o.cmd: File arch/x86/decode.c not in /tmp/kernel-obj or /tmp/kernel-obj/tools/objtool/arch/x86
Quoted 3 lines
Your patch solves it, but I wonder if it is worth lots of code addition. objtool is the only unfortunate case because it did not join
Kbuild.
I'd also prefer not to complicate things more than necessary, so maybe it's best to abandon this patch and leave the script as is. Probably the only thing needed is to clean up some of the variable names (e.g. as you mentioned, relative_path can sometimes be absolute) and maybe add more documentation on suggested usage.
Quoted 84 lines
Quoted 50 lines
The new argument, -k, is introduced in a way that makes the script backward compatible with how -d was previously used.
Signed-off-by Peter Kalauskas <[email protected]>---
gen_compile_commands.py +10
-16-def process_line(root_directory, file_directory, command_prefix, relative_path): +def process_line(src_dir, kbuild_out_dir, file_dir, cmd_prefix, file_path): """Extracts information from a .cmd line and creates an entry from it. Args: - root_directory: The directory that was searched for .cmd files. Usually + src_dir: The directory of the kernel source tree. + kbuild_out_dir: The directory that was searched for .cmd files. Usually used directly in the "directory" entry in compile_commands.json. - file_directory: The path to the directory the .cmd file was found in. - command_prefix: The extracted command line, up to the last element. - relative_path: The .c file from the end of the extracted command. - Usually relative to root_directory, but sometimes relative to - file_directory and sometimes neither. + file_dir: The path to the directory the .cmd file was found in. + cmd_prefix: The extracted command line, up to the last element. + file_path: The .c file from the end of the extracted command. + Usually relative to kbuild_out_dir, but sometimes relative to + src_dir and sometimes neither. Returns: An entry to append to compile_commands. Raises: - ValueError: Could not find the extracted file based on relative_path and - root_directory or file_directory. + ValueError: Could not find the extracted file. """ # The .cmd files are intended to be included directly by Make, so they # escape the pound sign '#', either as '\#' or '$(pound)' (depending on the - # kernel version). The compile_commands.json file is not interepreted + # kernel version). The compile_commands.json file is not interpreted # by Make, so this code replaces the escaped version with '#'. - prefix = command_prefix.replace('\#', '#').replace('$(pound)', '#') - - cur_dir = root_directory - expected_path = os.path.join(cur_dir, relative_path) - if not os.path.exists(expected_path): - # Try using file_directory instead. Some of the tools have a differentgen_compile_commands.py def parse_arguments(): · line 107
Quoted 10 lines
gen_compile_commands.py +7
-3- # style of .cmd file than the kernel. - cur_dir = file_directory - expected_path = os.path.join(cur_dir, relative_path) + prefix = cmd_prefix.replace('\#', '#').replace('$(pound)', '#') + + # Compile commands are usually run in the top level of the kbuild output + # directory + working_dir = kbuild_out_dir + + if os.path.isabs(file_path):I might be misreading the code, but
is this if-else switch needed?
os.path.join()returns the last parameter as-is
if it is already absolute, right?This is my quick experiment:
masahiro@oscar:~$ python3 Python 3.8.2 (default, Jul 16 2020, 14:00:26) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
import os
os.path.join("/foo/bar/", "a/b/c")'/foo/bar/a/b/c'
Quoted 1 line
Quoted 1 line
Quoted 1 line
os.path.join("/foo/bar/", "/a/b/c")'/a/b/c'
Quoted 9 lines
So, the current code:
expected_path=os.path.join(cur_dir, relative_path)works whether '
relative_path' is relative or not.I was also thinking this variable name was misleading since '
relative_path' may be actually absolute.
You are correct about os.path.join. That if-else is unnecessary in hindsight.
Thanks again for reviewing!
Quoted 8 lines
--
Best RegardsMasahiro Yamada