Kprobes: Add kprobes smoke tests that run on boot

From: Ananth N Mavinakayanahalli <[email protected]>

Here is a quick and naive smoke test for kprobes. This is intended to just verify if some unrelated change broke the *probes subsystem. It is self contained, architecture agnostic and isn't of any great use by itself.

This needs to be built in the kernel and runs a basic set of tests to verify if kprobes, jprobes and kretprobes run fine on the kernel. In case of an error, it'll print out a message with a "BUG" prefix.

This is a start; we intend to add more tests to this bucket over time.

Thanks to Jim Keniston and Masami Hiramatsu for comments and suggestions.

Tested on x86 (32/64) and powerpc.

---
 include/linux/kprobes.h |   10 ++
 kernel/Makefile         |    1 
 kernel/kprobes.c        |    2 
 kernel/test_kprobes.c   |  216 ++++++++++++++++++++++++++++++++++++++++++++++++
 lib/Kconfig.debug       |   12 ++
 5 files changed, 241 insertions(+)

Index: linux-2.6.24-rc6/kernel/Makefile
===================================================================
@@ -57,6 +57,7 @@ obj-$(CONFIG_GENERIC_HARDIRQS) += irq/
 obj-$(CONFIG_SECCOMP) += seccomp.o
 obj-$(CONFIG_DEBUG_SYNCHRO_TEST) += synchro-test.o
 obj-$(CONFIG_RCU_TORTURE_TEST) += rcutorture.o
+obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o
 obj-$(CONFIG_CLASSIC_RCU) += rcuclassic.o
 obj-$(CONFIG_PREEMPT_RCU) += rcupreempt.o
 ifeq ($(CONFIG_PREEMPT_RCU),y)

Index: linux-2.6.24-rc6/lib/Kconfig.debug
===================================================================
@@ -511,6 +511,18 @@ config RCU_TORTURE_TEST
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
 
+config KPROBES_SANITY_TEST
+	bool "Kprobes sanity tests"
+	depends on DEBUG_KERNEL
+	depends on KPROBES
+	default n
+	help
+	  This option provides for testing basic kprobes functionality on
+	  boot. A sample kprobe, jprobe and kretprobe are inserted and
+	  verified for functionality.
+
+	  Say N if you are unsure.
+
 config LKDTM
 	tristate "Linux Kernel Dump Test Tool Module"
 	depends on DEBUG_KERNEL

Index: linux-2.6.24-rc6/include/linux/kprobes.h
===================================================================
@@ -182,6 +182,15 @@ static inline void kretprobe_assert(stru
 	}
 }
 
+#ifdef CONFIG_KPROBES_SANITY_TEST
+extern int init_test_probes(void);
+#else
+static inline int init_test_probes(void)
+{
+	return 0;
+}
+#endif /* CONFIG_KPROBES_SANITY_TEST */
+
 extern spinlock_t kretprobe_lock;
 extern struct mutex kprobe_mutex;
 extern int arch_prepare_kprobe(struct kprobe *p);

@@ -227,6 +236,7 @@ void unregister_kretprobe(struct kretpro
 
 void kprobe_flush_task(struct task_struct *tk);
 void recycle_rp_inst(struct kretprobe_instance *ri, struct hlist_head *head);
+
 #else /* CONFIG_KPROBES */
 
 #define __kprobes	/**/

Index: linux-2.6.24-rc6/kernel/kprobes.c
===================================================================
@@ -821,6 +821,8 @@ static int __init init_kprobes(void)
 	if (!err)
 		err = register_die_notifier(&kprobe_exceptions_nb);
 
+	if (!err)
+		init_test_probes();
 	return err;
 }
 

yay! Picked this up into x86.git for testing.

small detail: i moved the kernel/Makefile change to another spot (see below), this way it integrates without rejects with rcu-preempt and with the kgdb changes.

Ingo

Index: linux-x86.q/kernel/Makefile
===================================================================
@@ -43,6 +43,7 @@ obj-$(CONFIG_CPUSETS) += cpuset.o
 obj-$(CONFIG_CGROUP_NS) += ns_cgroup.o
 obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
+obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
 obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
 obj-$(CONFIG_AUDIT_TREE) += audit_tree.o

Can you put this somewhere else please? I know there are already some test files in kernel/* but imho they all belong into some other directory.

+ if (!err)
+ init_test_probes();
return err;

Couldn't this be just a normal initcall? Perhaps even a module_init(), then it would be possible to use as a module too.

-Andi

agreed - lib/* would be the right place i think - we've got lib/locking-selftest.c already.

Ingo

Why not test/ ?

I felt a little bad about not introducing arch/x86/test/ for pageattr-test.c either.

-Andi

yeah, test/ would be even better, and then move locking-selftest over there too, plus the rt-mutex tester as well i suspect, and rcutorture.

Ingo

Actually the right place would be to have a separate toplevel directory I think, and a separate menu in the kernel config machinery. Thay way tests are in one place and easily spottable.

Seconded...

Sam

I have no problem with that, but if we want to make it buildable as a module, the call to get_kprobe() needs to be replaced with some other gcc-inline-defeating mechanism, or we need to export get_probe(). I much prefer the former, since get_kprobe() is a kprobes-internal function.

Anybody know an architecture-independent way (other than noinline, which doesn't always work) of making gcc decide not to inline a function? E.g., does taking (and using) the function's address do it?

Jim Keniston

+static noinline u32 kprobe_target(u32 value)
+{
+ /*

+        * gcc ignores noinline on some architectures unless we stuff
+        * sufficient lard into the function. The get_kprobe() here is
+        * just for that.
+        *
+        * NOTE: We aren't concerned about the correctness of
get_kprobe()
+        * here; hence, this call is neither under !preempt nor with the
+        * kprobe_mutex held. This is fine(tm)
+        */

+ if (get_kprobe((void *)0xdeadbeef))
+ printk(KERN_INFO "Kprobe smoke test: probe on
0xdeadbeef!\n");
+
+ return (value / div_factor);
+}

It's still unclear where noinline does not work (do you have details on that? It sounds a little dubious) but there are lots of different other ways to prevent gcc from inlining anything.

e.g. you can use stdarg or computed goto or a variable length local array

Details?

No, that just creates another out of line copy.

-Andi

Here's a simplified version of a module we've used to gauge kprobes performance. On my system (i386, 2.6.23-rc8 at the moment, gcc version 4.1.2), the module calls the (empty) probed function (probeme) niter times, but reports zero probe hits -- indicating probeme was inlined even though it was declared noinline.

If I change probeme to a varargs function, per your suggestion -- see diffs below -- the proper number of probe hits are reported. ----- noinline.c -----
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/jiffies.h>
#include <linux/kprobes.h>

/*
* Sample usage: insmod noinline.ko niter=5000000
* printk should report 5000000 calls.
*/

static int niter = 0;
module_param(niter, int, 0);
MODULE_PARM_DESC(niter, "iterations");

static int ncalls = 0;

static int counter_hdlr(struct kprobe *p, struct pt_regs *regs)

{
	ncalls++;
	return 0;
}

// #define noinline __attribute__((noinline))
noinline static void probeme(void)
{
}

struct kprobe kp = {
.addr = (kprobe_opcode_t*) probeme,
.pre_handler = counter_hdlr
};

static int __init noinline_init(void)

{
	int ret;
	int i;
	u64 start, stop;

	ret = register_kprobe(&kp);
	if (ret != 0) {
		printk(KERN_ERR "register_kprobe returns %d\n", ret);
		return ret;
	}

	start = get_jiffies_64();
	for (i = 1; i <= niter; i++)
		probeme();
	stop = get_jiffies_64();
	printk(KERN_INFO "noinline test: %d iterations in %lld jiffies\n",
		niter, stop-start);
	printk(KERN_INFO "Counted %d calls\n", ncalls);
	return 0;
}

static void __exit noinline_exit(void)

{
	unregister_kprobe(&kp);
	printk(KERN_INFO "noinline test: exiting...\n");
}

module_init(noinline_init);
module_exit(noinline_exit);
MODULE_LICENSE("GPL");
----- Change probeme to a varargs function -----

@@ -22,8 +22,13 @@
 }
 
 // #define noinline __attribute__((noinline))
-noinline static void probeme(void)
+noinline static void probeme(int nargs, ...)
 {
+	va_list args;
+	va_start(args, nargs);
+	while (nargs-- > 0)
+		printk(KERN_INFO "%p\n", va_arg(args, char*));
+	va_end(args);
 }
 
 struct kprobe kp = {

@@ -45,9 +50,9 @@
 
 	start = get_jiffies_64();
 	for (i = 1; i <= niter; i++)
-		probeme();
+		probeme(0);
 	stop = get_jiffies_64();
-	printk(KERN_INFO "noinline test: %d iterations in %lld jiffies\n",
+	printk(KERN_INFO "noinline2 test: %d iterations in %lld jiffies\n",
 		niter, stop-start);
 	printk(KERN_INFO "Counted %d calls\n", ncalls);
 	return 0;

@@ -56,7 +61,7 @@
 static void __exit noinline_exit(void)
 {
 	unregister_kprobe(&kp);
-	printk(KERN_INFO "noinline test: exiting...\n");
+	printk(KERN_INFO "noinline2 test: exiting...\n");
 }
 
 module_init(noinline_init);

...

Thanks for the suggestions.
Jim

Yes indeed looks like a gcc bug. Have you reported it?

-Andi

There's already a gcc bug report that appears to describe the same problem:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=34563
There seems to be some doubt as to whether it's really a bug, so I added my 2 cents, including a pointer to this thread.

Jim

From: Ananth N Mavinakayanahalli <[email protected]>
Date: Tue, 8 Jan 2008 12:03:34 +0530

Thanks very much for writing this.

It will come in handy for me when I work on sparc64 kretprobe support.