Add clang-tidy and static analyzer support to makefile
Subsystems (12)
RFC PATCH 1/2 Add clang-tidy and static analyzer support to makefile
---
These patches add clang-tidy and the clang static-analyzer as make targets. The goal of these patches is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++.
The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease.
===Clang-tidy===
Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST
(https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings.
Here is an example clang-tidy check to report functions that have calls to local_irq_disable without calls to local_irq_enable and vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis.
The full patch can be found here (https://reviews.llvm.org/D65828)
```
void IrqUnbalancedCheck::registerMatchers(MatchFinder *Finder) {
// finds calls to "arch_local_irq_disable" in a function body
auto disable =
forEachDescendant(
callExpr(
hasDeclaration(
namedDecl(
hasName("arch_local_irq_disable")))).bind("disable"));
// finds calls to "arch_local_irq_enable" in a function body
auto enable =
forEachDescendant(
callExpr(
hasDeclaration(
namedDecl(
hasName("arch_local_irq_enable")))).bind("enable"));
// Looks for function body that has the following property:
// ((disable && !enable) || (enable && !disable))
auto matcher = functionDecl(
anyOf(allOf(disable, unless(enable)), allOf(enable, unless(disable))));
Finder->addMatcher(matcher.bind("func"), this);
}
std::string annotation = "ignore_irq_balancing";
void IrqUnbalancedCheck::check(const MatchFinder::MatchResult &Result) {
const auto *MatchedDecl = Result.Nodes.getNodeAs<FunctionDecl>("func");
const auto *DisableCall = Result.Nodes.getNodeAs<CallExpr>("disable");
const auto *EnableCall = Result.Nodes.getNodeAs<CallExpr>("enable");
// If the function has __attribute((annotate("ignore_irq_balancing"))
for (const auto *Attr : MatchedDecl->attrs()) {
if (Attr->getKind() == clang::attr::Annotate) {
if(dyn_cast<AnnotateAttr>(Attr)->getAnnotation().str() == annotation) {
return;
}
}
}
if(EnableCall) {
diag(MatchedDecl->getLocation(), "call to 'enable_local_irq' without 'disable_local_irq' in %0 ")
<< MatchedDecl;
diag(EnableCall->getBeginLoc(), "call to 'enable_local_irq'", DiagnosticIDs::Note)
<< MatchedDecl;
}
if(DisableCall) {
diag(MatchedDecl->getLocation(), "call to 'disable_local_irq' without 'enable_local_irq' in %0 ")
<< MatchedDecl;
diag(DisableCall->getBeginLoc(), "call to 'disable_local_irq'", DiagnosticIDs::Note)
<< MatchedDecl;
}
}
```===Clang static analyzer===
The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel.
The clang static analyzer is well documented and designed to be extensible.
(https://clang-analyzer.llvm.org/checker_dev_manual.html)
(https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf)
Why add clang-tidy and the clang static analyzer when other static analyzers are already in the kernel?
The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase.
Makefile | 3 ++
scripts/clang-tools/Makefile.clang-tools | 35 ++++++++++++++
.../{ => clang-tools}/gen_compile_commands.py | 0
scripts/clang-tools/parse_compile_commands.py | 47 +++++++++++++++++++
4 files changed, 85 insertions(+)
create mode 100644 scripts/clang-tools/Makefile.clang-tools
rename scripts/{ => clang-tools}/gen_compile_commands.py (100%)
create mode 100755 scripts/clang-tools/parse_compile_commands.pyMakefile +3 -0
@@ -709,6 +709,7 @@ KBUILD_CFLAGS += $(call cc-option,--param=allow-store-data-races=0) include scripts/Makefile.kcov include scripts/Makefile.gcc-plugins +include scripts/clang-tools/Makefile.clang-tools ifdef CONFIG_READABLE_ASM # Disable optimizations that make assembler listings hard to read.
@@ -1470,6 +1471,8 @@ help: @echo ' headers_check - Sanity check on exported headers' @echo ' headerdep - Detect inclusion cycles in headers' @echo ' coccicheck - Check with Coccinelle' + @echo ' clang-analyzer - Check with clang static analyzer' + @echo ' clang-tidy - Check with clang-tidy' @echo '' @echo 'Kernel selftest:' @echo ' kselftest - Build and run kernel selftest (run as root)'
scripts/clang-tools/Makefile.clang-tools new file +35 -0
@@ -0,0 +1,35 @@ +# SPDX-License-Identifier: GPL-2.0 +PHONY += clang-tidy +HAS_PARALLEL := $(shell (parallel --version 2> /dev/null) | grep 'GNU parallel' 2> /dev/null) +clang-tidy: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL + #Xargs interleaves multiprocessed output. GNU Parallel does not. + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,linuxkernel-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,linuxkernel-*' @" +endif +else + $(error Clang-tidy requires CC=clang) +endif + +PHONY += clang-analyzer +clang-analyzer: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL + #Xargs interleaves multiprocessed output. GNU Parallel does not. + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,clang-analyzer-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,clang-analyzer-*' @" +endif +else + $(error Clang-analyzer requires CC=clang) +endif
scripts/clang-tools/gen_compile_commands.py renamed from scripts/gen_compile_commands.py +0 -0
scripts/clang-tools/parse_compile_commands.py new file +47 -0
47 lines not on this page · raw message
Signature · 1 line
2.22.0.709.g102302147b-goog
RFC PATCH 2/2 Add clang-tidy irq-balancing annotations for arm64
This patch is not intended to be merged. It simply shows the scale of the effort involved in applying annotations in the case of intentionally unbalanced calls to local_irq_disable.
--- arch/arm64/kernel/debug-monitors.c | 1 + arch/arm64/kernel/machine_kexec.c | 1 + arch/arm64/kernel/process.c | 4 ++++ arch/arm64/kernel/smp.c | 1 + drivers/tty/sysrq.c | 1 + include/linux/compiler_attributes.h | 7 +++++++ init/main.c | 1 + kernel/irq/handle.c | 1 + kernel/power/suspend.c | 2 ++ kernel/sched/core.c | 1 + kernel/sched/fair.c | 1 + kernel/sched/idle.c | 6 ++++++ kernel/stop_machine.c | 1 + mm/migrate.c | 1 + net/core/dev.c | 1 + 15 files changed, 30 insertions(+)
arch/arm64/kernel/debug-monitors.c +1 -0
@@ -221,6 +221,7 @@ static int call_step_hook(struct pt_regs *regs, unsigned int esr) }NOKPROBE_SYMBOL (call_step_hook); +__unbalanced_irq static void send_user_sigtrap(int si_code) {struct pt_regs *regs =current_pt_regs ();
arch/arm64/kernel/machine_kexec.c +1 -0
@@ -258,6 +258,7 @@ static void machine_kexec_mask_interrupts(void)/** * machine_crash_shutdown - shutdown non-crashing cpus and save registers */ +__unbalanced_irq void machine_crash_shutdown(struct pt_regs *regs) {local_irq_disable ();
arch/arm64/kernel/process.c +4 -0
@@ -111,6 +111,7 @@ void cpu_do_idle(void)/* * This is our default idle handler. */ +__unbalanced_irq void arch_cpu_idle(void) {/*
@@ -149,6 +150,7 @@ void machine_shutdown(void)* activity (executing tasks, handling interrupts). smp_send_stop() * achieves this. */ +__unbalanced_irq void machine_halt(void) {local_irq_disable ();
@@ -162,6 +164,7 @@ void machine_halt(void)* achieves this. When the system power is turned off, it will take all CPUs * with it. */ +__unbalanced_irq void machine_power_off(void) {local_irq_disable ();
@@ -179,6 +182,7 @@ void machine_power_off(void)* executing pre-reset code, and using RAM that the primary CPU's code wishes * to use. Implementing such co-ordination would be essentially impossible. */ +__unbalanced_irq void machine_restart(char *cmd) {/* Disable interrupts first */
arch/arm64/kernel/smp.c +1 -0
@@ -852,6 +852,7 @@ static void ipi_cpu_stop(unsigned int cpu)static atomic_t waiting_for_crash_ipi =ATOMIC_INIT (0 );#endif +__unbalanced_irq static void ipi_cpu_crash_stop(unsigned int cpu, struct pt_regs *regs) {#ifdef CONFIG_KEXEC_CORE
drivers/tty/sysrq.c +1 -0
@@ -146,6 +146,7 @@ static struct sysrq_key_op sysrq_crash_op = { .enable_mask = SYSRQ_ENABLE_DUMP, }; +__unbalanced_irq static void sysrq_handle_reboot(int key) {lockdep_off ();
include/linux/compiler_attributes.h +7 -0
@@ -253,4 +253,11 @@*/ #define __weak __attribute__((__weak__)) + +#if __has_attribute (__annotate__) +# define __unbalanced_irq __attribute__((annotate("ignore_irq_balancing"))) +#else +# define __unbalanced_irq +#endif +#endif /* __LINUX_COMPILER_ATTRIBUTES_H */
init/main.c +1 -0
@@ -902,6 +902,7 @@ static inline void do_trace_initcall_finish(initcall_t fn, int ret) }#endif /* !TRACEPOINTS_ENABLED */ +__unbalanced_irq int __init_or_module do_one_initcall(initcall_t fn) {int count =preempt_count ();
kernel/irq/handle.c +1 -0
@@ -134,6 +134,7 @@ void __irq_wake_thread(struct irq_desc *desc, struct irqaction *action)wake_up_process (action->thread); } +__unbalanced_irq irqreturn_t __handle_irq_event_percpu(struct irq_desc *desc, unsigned int *flags) {irqreturn_t retval =IRQ_NONE ;
kernel/power/suspend.c +2 -0
@@ -380,12 +380,14 @@ static int suspend_prepare(suspend_state_t state) }/* default implementation */ +__unbalanced_irq void __weak arch_suspend_disable_irqs(void) {local_irq_disable (); }/* default implementation */ +__unbalanced_irq void __weak arch_suspend_enable_irqs(void) {local_irq_enable ();
kernel/sched/core.c +1 -0
@@ -3366,6 +3366,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)* * WARNING: must be called with preemption disabled! */ +__unbalanced_irq static void __sched notrace __schedule(bool preempt) {struct task_struct *prev, *next;
kernel/sched/fair.c +1 -0
@@ -9334,6 +9334,7 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance)* least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ +__unbalanced_irq static int active_load_balance_cpu_stop(void *data) {struct rq *busiest_rq = data;
kernel/sched/idle.c +6 -0
@@ -52,6 +52,7 @@ static int __init cpu_idle_nopoll_setup(char *__unused)__setup ("hlt" , cpu_idle_nopoll_setup);#endif +__unbalanced_irq static noinline int __cpuidle cpu_idle_poll(void) {rcu_idle_enter ();
@@ -74,6 +75,7 @@ void __weak arch_cpu_idle_prepare(void) { }void __weak arch_cpu_idle_enter (void ) { }void __weak arch_cpu_idle_exit (void ) { }void __weak arch_cpu_idle_dead (void ) { } +__unbalanced_irq void __weak arch_cpu_idle(void) { cpu_idle_force_poll =1 ;
@@ -85,6 +87,7 @@ void __weak arch_cpu_idle(void)* * To use when the cpuidle framework cannot be used. */ +__unbalanced_irq void __cpuidle default_idle_call(void) {if (current_clr_polling_and_test ()) {
@@ -96,6 +99,7 @@ void __cpuidle default_idle_call(void) } } +__unbalanced_irq static int call_cpuidle(struct cpuidle_driver *drv, struct cpuidle_device *dev,int next_state) {
@@ -126,6 +130,7 @@ static int call_cpuidle(struct cpuidle_driver *drv, struct cpuidle_device *dev,* set, and it returns with polling set. If it ever stops polling, it * must clear the polling bit. */ +__unbalanced_irq static void cpuidle_idle_call(void) {struct cpuidle_device *dev =cpuidle_get_device ();
@@ -222,6 +227,7 @@ static void cpuidle_idle_call(void)* * Called with polling cleared. */ +__unbalanced_irq static void do_idle(void) {int cpu =smp_processor_id ();
kernel/stop_machine.c +1 -0
@@ -178,6 +178,7 @@ static void ack_state(struct multi_stop_data *msdata) }/* This is the cpu_stop function which stops the CPU. */ +__unbalanced_irq static int multi_cpu_stop(void *data) {struct multi_stop_data *msdata = data;
mm/migrate.c +1 -0
@@ -396,6 +396,7 @@ static int expected_page_refs(struct address_space *mapping, struct page *page)* 2 for pages with a mapping * 3 for pages with a mapping and PagePrivate/PagePrivate2 set. */ +__unbalanced_irq int migrate_page_move_mapping(struct address_space *mapping,struct page *newpage,struct page *page,enum migrate_mode mode,int extra_count)
net/core/dev.c +1 -0
@@ -5885,6 +5885,7 @@ static void net_rps_send_ipi(struct softnet_data *remsd)* net_rps_action_and_irq_enable sends any pending IPI's for rps. * Note: called with local irq disabled, but exits with local irq enabled. */ +__unbalanced_irq static void net_rps_action_and_irq_enable(struct softnet_data *sd) {#ifdef CONFIG_RPS
Signature · 1 line
2.22.0.709.g102302147b-goog
[ Added kbuild and Masahiro to CC ... ]
Quoted 2 lines · Nathan Huckleberry
Signed-off-by Nathan Huckleberry <[email protected]>---
These two lines should be at the end of your commit log. :)
Quoted 98 lines
These patches add clang-tidy and the clang static-analyzer as make targets. The goal of these patches is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++.
The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease.
===Clang-tidy===
Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST
(https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings.Here is an example clang-tidy check to report functions that have calls to
local_irq_disablewithout calls tolocal_irq_enableand vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis.The full patch can be found here (https://reviews.llvm.org/D65828)
``` void IrqUnbalancedCheck::registerMatchers(MatchFinder *Finder) { // finds calls to "arch_local_irq_disable" in a function body auto disable = forEachDescendant( callExpr( hasDeclaration( namedDecl( hasName("arch_local_irq_disable")))).bind("disable")); // finds calls to "arch_local_irq_enable" in a function body auto enable = forEachDescendant( callExpr( hasDeclaration( namedDecl( hasName("arch_local_irq_enable")))).bind("enable")); // Looks for function body that has the following property: // ((disable && !enable) || (enable && !disable)) auto matcher = functionDecl( anyOf(allOf(disable, unless(enable)), allOf(enable, unless(disable)))); Finder->addMatcher(matcher.bind("func"), this); } std::string annotation = "ignore_irq_balancing"; void IrqUnbalancedCheck::check(const MatchFinder::MatchResult &Result) { const auto *MatchedDecl = Result.Nodes.getNodeAs<FunctionDecl>("func"); const auto *DisableCall = Result.Nodes.getNodeAs<CallExpr>("disable"); const auto *EnableCall = Result.Nodes.getNodeAs<CallExpr>("enable"); // If the function has __attribute((annotate("ignore_irq_balancing")) for (const auto *Attr : MatchedDecl->attrs()) { if (Attr->getKind() == clang::attr::Annotate) { if(dyn_cast<AnnotateAttr>(Attr)->getAnnotation().str() == annotation) { return; } } } if(EnableCall) { diag(MatchedDecl->getLocation(), "call to 'enable_local_irq' without 'disable_local_irq' in %0 ") << MatchedDecl; diag(EnableCall->getBeginLoc(), "call to 'enable_local_irq'", DiagnosticIDs::Note) << MatchedDecl; } if(DisableCall) { diag(MatchedDecl->getLocation(), "call to 'disable_local_irq' without 'enable_local_irq' in %0 ") << MatchedDecl; diag(DisableCall->getBeginLoc(), "call to 'disable_local_irq'", DiagnosticIDs::Note) << MatchedDecl; } } ```===Clang static analyzer===
The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel.
The clang static analyzer is well documented and designed to be extensible.
(https://clang-analyzer.llvm.org/checker_dev_manual.html)
(https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf)
Quoted 9 lines
Why add clang-tidy and the clang static analyzer when other static analyzers are already in the kernel?
The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase.
^ i.e. they should be right here...
Quoted 22 lines
Makefile | 3 ++ scripts/clang-tools/Makefile.clang-tools | 35 ++++++++++++++ .../{ => clang-tools}/gen_compile_commands.py | 0 scripts/clang-tools/parse_compile_commands.py | 47 +++++++++++++++++++ 4 files changed, 85 insertions(+) create mode 100644 scripts/clang-tools/Makefile.clang-tools rename scripts/{ => clang-tools}/gen_compile_commands.py (100%) create mode 100755 scripts/clang-tools/parse_compile_commands.pyMakefile +1
-0@@ -709,6 +709,7 @@ KBUILD_CFLAGS += $(call cc-option,--param=allow-store-data-races=0) include scripts/Makefile.kcov include scripts/Makefile.gcc-plugins +include scripts/clang-tools/Makefile.clang-tools ifdef CONFIG_READABLE_ASM # Disable optimizations that make assembler listings hard to read.
Makefile help: · line 1475
Quoted 5 lines
Makefile +2
-0@@ -1470,6 +1471,8 @@ help: @echo ' headers_check - Sanity check on exported headers' @echo ' headerdep - Detect inclusion cycles in headers' @echo ' coccicheck - Check with Coccinelle' + @echo ' clang-analyzer - Check with clang static analyzer' + @echo ' clang-tidy - Check with clang-tidy'
I think this is great; more people will be able to run these tests.
Quoted 9 lines
Makefile +0
-0@echo '' @echo 'Kernel selftest:' @echo ' kselftest - Build and run kernel selftest (run as root)'scripts/clang-tools/Makefile.clang-tools new file +0
-0
scripts/clang-tools/Makefile.clang-tools line 7
Quoted 7 lines
scripts/clang-tools/Makefile.clang-tools new file +7
-0@@ -0,0 +1,35 @@ +# SPDX-License-Identifier: GPL-2.0 +PHONY += clang-tidy +HAS_PARALLEL := $(shell (parallel --version 2> /dev/null) | grep 'GNU parallel' 2> /dev/null) +clang-tidy: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL
Is it worth falling back to xargs? Are there builders where clang-tidy or clang-analyzer are installed but parallel isn't? (i.e. it might just be better to simply require parallel.) Note that there's no test for python3 -- we just try to run it. :)
Quoted 75 lines
scripts/clang-tools/Makefile.clang-tools new file +28
-0+ #Xargs interleaves multiprocessed output. GNU Parallel does not. + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,linuxkernel-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,linuxkernel-*' @" +endif +else + $(error Clang-tidy requires CC=clang) +endif + +PHONY += clang-analyzer +clang-analyzer: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL + #Xargs interleaves multiprocessed output. GNU Parallel does not. + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,clang-analyzer-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,clang-analyzer-*' @" +endif +else + $(error Clang-analyzer requires CC=clang) +endifscripts/clang-tools/gen_compile_commands.py renamed from scripts/gen_compile_commands.py +0
-0scripts/clang-tools/parse_compile_commands.py new file +37
-0@@ -0,0 +1,47 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) Google LLC, 2019 +# +# Author: Nathan Huckleberry <[email protected]> +# +"""A helper routine for make clang-tidy to parse compile_commands.json.""" + +import argparse +import json +import logging +import os +import re + +def parse_arguments(): + """Set up and parses command-line arguments. + Returns: + file_path: Path to compile_commands.json file + """ + usage = """Parse a compilation database and return a list of files + included in the database""" + parser = argparse.ArgumentParser(description=usage) + + file_path_help = ('Path to the compilation database to parse') + parser.add_argument('file', type=str, help=file_path_help) + + args = parser.parse_args() + + return args.file + + +def main(): + filename = parse_arguments() + + #Read JSON data into the datastore variable + if filename:
scripts/clang-tools/parse_compile_commands.py line 47
Quoted 10 lines
scripts/clang-tools/parse_compile_commands.py new file +10
-0+ with open(filename, 'r') as f: + datastore = json.load(f) + + #Use the new datastore datastructure + for entry in datastore: + print(entry['file']) + + +if __name__ == '__main__': + main()
And this is nice to have because now there is a real consumer of gen_compile_commands.py.
Signature · 1 line
Kees Cook
Quoted 154 lines · Nathan Huckleberry
Signed-off-by Nathan Huckleberry <[email protected]>---
These patches add clang-tidy and the clang static-analyzer as make targets. The goal of these patches is to make static analysis tools usable and extendable by any developer or researcher who is familiar with basic c++.The current static analysis tools require intimate knowledge of the internal workings of the static analysis. Clang-tidy and the clang static analyzers expose an easy to use api and allow users unfamiliar with clang to write new checks with relative ease.
===Clang-tidy===
Clang-tidy is an easily extendable 'linter' that runs on the AST. Clang-tidy checks are easy to write and understand. A check consists of two parts, a matcher and a checker. The matcher is created using a domain specific language that acts on the AST
(https://clang.llvm.org/docs/LibASTMatchersReference.html). When AST nodes are found by the matcher a callback is made to the checker. The checker can then execute additional checks and issue warnings.Here is an example clang-tidy check to report functions that have calls to
local_irq_disablewithout calls tolocal_irq_enableand vice-versa. Functions flagged with __attribute((annotation("ignore_irq_balancing"))) are ignored for analysis.The full patch can be found here (https://reviews.llvm.org/D65828)
``` void IrqUnbalancedCheck::registerMatchers(MatchFinder *Finder) { // finds calls to "arch_local_irq_disable" in a function body auto disable = forEachDescendant( callExpr( hasDeclaration( namedDecl( hasName("arch_local_irq_disable")))).bind("disable")); // finds calls to "arch_local_irq_enable" in a function body auto enable = forEachDescendant( callExpr( hasDeclaration( namedDecl( hasName("arch_local_irq_enable")))).bind("enable")); // Looks for function body that has the following property: // ((disable && !enable) || (enable && !disable)) auto matcher = functionDecl( anyOf(allOf(disable, unless(enable)), allOf(enable, unless(disable)))); Finder->addMatcher(matcher.bind("func"), this); } std::string annotation = "ignore_irq_balancing"; void IrqUnbalancedCheck::check(const MatchFinder::MatchResult &Result) { const auto *MatchedDecl = Result.Nodes.getNodeAs<FunctionDecl>("func"); const auto *DisableCall = Result.Nodes.getNodeAs<CallExpr>("disable"); const auto *EnableCall = Result.Nodes.getNodeAs<CallExpr>("enable"); // If the function has __attribute((annotate("ignore_irq_balancing")) for (const auto *Attr : MatchedDecl->attrs()) { if (Attr->getKind() == clang::attr::Annotate) { if(dyn_cast<AnnotateAttr>(Attr)->getAnnotation().str() == annotation) { return; } } } if(EnableCall) { diag(MatchedDecl->getLocation(), "call to 'enable_local_irq' without 'disable_local_irq' in %0 ") << MatchedDecl; diag(EnableCall->getBeginLoc(), "call to 'enable_local_irq'", DiagnosticIDs::Note) << MatchedDecl; } if(DisableCall) { diag(MatchedDecl->getLocation(), "call to 'disable_local_irq' without 'enable_local_irq' in %0 ") << MatchedDecl; diag(DisableCall->getBeginLoc(), "call to 'disable_local_irq'", DiagnosticIDs::Note) << MatchedDecl; } } ```===Clang static analyzer===
The clang static analyzer is a more powerful static analysis tool that uses symbolic execution to find bugs. Currently there is a check that looks for potential security bugs from invalid uses of kmalloc and kfree. There are several more general purpose checks that are useful for the kernel.
The clang static analyzer is well documented and designed to be extensible.
(https://clang-analyzer.llvm.org/checker_dev_manual.html)
(https://github.com/haoNoQ/clang-analyzer-guide/releases/download/v0.1/clang-analyzer-guide-v0.1.pdf)Why add clang-tidy and the clang static analyzer when other static analyzers are already in the kernel?
The main draw of the clang tools is how accessible they are. The clang documentation is very nice and these tools are built specifically to be easily extendable by any developer. They provide an accessible method of bug-finding and research to people who are not overly familiar with the kernel codebase.
Makefile | 3 ++ scripts/clang-tools/Makefile.clang-tools | 35 ++++++++++++++ .../{ => clang-tools}/gen_compile_commands.py | 0 scripts/clang-tools/parse_compile_commands.py | 47 +++++++++++++++++++ 4 files changed, 85 insertions(+) create mode 100644 scripts/clang-tools/Makefile.clang-tools rename scripts/{ => clang-tools}/gen_compile_commands.py (100%) create mode 100755 scripts/clang-tools/parse_compile_commands.pyMakefile +3
-0@@ -709,6 +709,7 @@ KBUILD_CFLAGS += $(call cc-option,--param=allow-store-data-races=0) include scripts/Makefile.kcov include scripts/Makefile.gcc-plugins +include scripts/clang-tools/Makefile.clang-tools ifdef CONFIG_READABLE_ASM # Disable optimizations that make assembler listings hard to read.@@ -1470,6 +1471,8 @@ help: @echo ' headers_check - Sanity check on exported headers' @echo ' headerdep - Detect inclusion cycles in headers' @echo ' coccicheck - Check with Coccinelle' + @echo ' clang-analyzer - Check with clang static analyzer' + @echo ' clang-tidy - Check with clang-tidy' @echo '' @echo 'Kernel selftest:' @echo ' kselftest - Build and run kernel selftest (run as root)'scripts/clang-tools/Makefile.clang-tools new file +8
-0@@ -0,0 +1,35 @@ +# SPDX-License-Identifier: GPL-2.0 +PHONY += clang-tidy +HAS_PARALLEL := $(shell (parallel --version 2> /dev/null) | grep 'GNU parallel' 2> /dev/null) +clang-tidy: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL + #Xargs interleaves multiprocessed output. GNU Parallel does not.
scripts/clang-tools/Makefile.clang-tools line 18
Quoted 10 lines
scripts/clang-tools/Makefile.clang-tools new file +10
-0+ scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,linuxkernel-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,linuxkernel-*' @" +endif +else + $(error Clang-tidy requires CC=clang) +endif
What is the benefit for wiring this to the Makefile?
Is there any drawback if you implement this in a small independent shell script or something?
You are already adding a new python script just for reading the json file.
I would invoke clang-{tidy,analyzer} from that python script.
If you unify these into a python script,
you can support the multiprocessing with a python library ('import multiprocessing') without GNU parallel.
Thanks.
Quoted 74 lines
scripts/clang-tools/Makefile.clang-tools new file +16
-0+PHONY += clang-analyzer +clang-analyzer: +ifdef CONFIG_CC_IS_CLANG + $(PYTHON3) scripts/clang-tools/gen_compile_commands.py +ifdef HAS_PARALLEL + #Xargs interleaves multiprocessed output. GNU Parallel does not. + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | parallel -k -j $(shell nproc) 'echo {} && clang-tidy -p . "-checks=-*,clang-analyzer-*" {}' +else + @echo "GNU parallel is not installed. Defaulting to non-parallelized runs" + scripts/clang-tools/parse_compile_commands.py compile_commands.json \ + | xargs -L 1 -I@ sh -c "echo '@' && clang-tidy -p . '-checks=-*,clang-analyzer-*' @" +endif +else + $(error Clang-analyzer requires CC=clang) +endifscripts/clang-tools/gen_compile_commands.py renamed from scripts/gen_compile_commands.py +0
-0scripts/clang-tools/parse_compile_commands.py new file +47
-1@@ -0,0 +1,47 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (C) Google LLC, 2019 +# +# Author: Nathan Huckleberry <[email protected]> +# +"""A helper routine for make clang-tidy to parse compile_commands.json.""" + +import argparse +import json +import logging +import os +import re + +def parse_arguments(): + """Set up and parses command-line arguments. + Returns: + file_path: Path to compile_commands.json file + """ + usage = """Parse a compilation database and return a list of files + included in the database""" + parser = argparse.ArgumentParser(description=usage) + + file_path_help = ('Path to the compilation database to parse') + parser.add_argument('file', type=str, help=file_path_help) + + args = parser.parse_args() + + return args.file + + +def main(): + filename = parse_arguments() + + #Read JSON data into the datastore variable + if filename: + with open(filename, 'r') as f: + datastore = json.load(f) + + #Use the new datastore datastructure + for entry in datastore: + print(entry['file']) + + +if __name__ == '__main__': + main() --
Quoted 2 lines
2.22.0.709.g102302147b-goog
--
Best Regards
Masahiro Yamada