Skip to contentSkip to conversation

LTTng-core 0.5.108 : core

PATCH In review Mathieu Desnoyers 3 participants 5 messages Sep 14, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 4/11 LTTng-core 0.5.108 : core

4 - Core tracer file
patch-2.6.17-lttng-core-0.5.108-core.diff

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 
MAINTAINERS +7 -0
@@ -1740,6 +1740,13 @@ W:	http://lsm.immunix.org
 T:	git kernel.org:/pub/scm/linux/kernel/git/chrisw/lsm-2.6.git
 S:	Supported
 
+LINUX TRACE TOOLKIT NEXT GENERATION
+P:	Mathieu Desnoyers
+M:	[email protected]
+L:	[email protected]
+W:	http://ltt.polymtl.ca
+S:	Maintained
+
 LM83 HARDWARE MONITOR DRIVER
 P:	Jean Delvare
 M:	[email protected]
ltt/ltt-core.c new file +895 -0

895 lines not on this page · raw message

init/main.c +4 -0
@@ -47,6 +47,7 @@ #include <linux/unistd.h>
 #include <linux/rmap.h>
 #include <linux/mempolicy.h>
 #include <linux/key.h>
+#include <linux/ltt-core.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -482,6 +483,9 @@ asmlinkage void __init start_kernel(void
 		   __stop___param - __start___param,
 		   &unknown_bootoption);
 	sort_main_extable();
+#ifdef CONFIG_LTT
+	ltt_init();
+#endif //CONFIG_LTT
 	trap_init();
 	rcu_init();
 	init_IRQ();
kernel/ltt-base.c new file +42 -0

42 lines not on this page · raw message

kernel/ltt-heartbeat.c new file +210 -0

210 lines not on this page · raw message

#1 lore raw

hoi :)

ltt/ltt-core.c line 101

Quoted 11 lines · Mathieu Desnoyers
ltt/ltt-core.c new file +11 -0
+int ltt_module_register(enum ltt_module_function name, void *function,
+		struct module *owner)
+{
+	int ret = 0;
+	
+	/* Protect these operations by disallowing them when tracing is
+	 * active */
+	if(ltt_traces.num_active_traces) {
+		ret = -EBUSY;
+		goto end;
+	}

what would happen otherwise?
can it happen that someone enables tracing between this check and the rest of the function?

ltt/ltt-core.c line 468

Quoted 4 lines
ltt/ltt-core.c new file +4 -0
+	new_trace->transport = transport;
+	new_trace->ops = &transport->ops;
+
+	err = -new_trace->ops->create_dirs(new_trace);

^ typo or intentional?

Signature · 1 line
Martin Waitz
#3 lore raw
In reply to Mathieu Desnoyers

Hi,

I'm wondering why this is safe:

Quoting Mathieu Desnoyers ([email protected]):
...

ltt/ltt-core.c line 176

Quoted 11 lines
ltt/ltt-core.c new file +11 -0
+void ltt_transport_register(struct ltt_transport *transport)
+{
+	spin_lock(&transport_list_lock);
+	list_add_tail(&transport->node, &ltt_transport_list);
+	spin_unlock(&transport_list_lock);
+}
+
+void ltt_transport_unregister(struct ltt_transport *transport)
+{
+	spin_lock(&transport_list_lock);
+	list_del(&transport->node);

You don't have a refcount you check here, and

Quoted 5 lines
ltt/ltt-core.c new file +5 -0
+	spin_unlock(&transport_list_lock);
+}
+
+EXPORT_SYMBOL_GPL(ltt_transport_register);
+EXPORT_SYMBOL_GPL(ltt_transport_unregister);

...

ltt/ltt-core.c line 410

Quoted 6 lines
ltt/ltt-core.c new file +6 -0
+static int ltt_trace_create(char *trace_name, char *trace_type,
+		enum trace_mode mode,
+		unsigned subbuf_size_low, unsigned n_subbufs_low,
+		unsigned subbuf_size_med, unsigned n_subbufs_med,
+		unsigned subbuf_size_high, unsigned n_subbufs_high)
+{

here:

Quoted 13 lines
ltt/ltt-core.c new file +13 -0
+	spin_lock(&transport_list_lock);
+	list_for_each_entry(tran, &ltt_transport_list, node) {
+		if (!strcmp(tran->name, trace_type)) {
+			transport = tran;
+			break;
+		}
+	}
+	spin_unlock(&transport_list_lock);
+
+	if (!transport) {
+		err = EINVAL;
+		printk(KERN_ERR	"LTT : Transport %s is not present.\n", trace_type);
+		goto trace_error;

ltt/ltt-core.c line 466

Quoted 10 lines
ltt/ltt-core.c new file +10 -0
+	}
+
+	if(!try_module_get(transport->owner)) {
+		err = ENODEV;
+		printk(KERN_ERR	"LTT : Can't lock transport module.\n");
+		goto trace_error;
+	}
+
+	new_trace->transport = transport;
+	new_trace->ops = &transport->ops;

you grab references to the object which may be deleted after you drop the transport_list_lock at the top of this block. Since a later patch shows the unregister being called right before the owning module is unloaded, that seems awefuly dangerous.

Is there some other magic going on making this safe?

thanks,
-serge

#4 lore raw
Quoted 11 lines · * Serge E. Hallyn ([email protected]) wrote:

Hi,

I'm wondering why this is safe:

you grab references to the object which may be deleted after you drop the transport_list_lock at the top of this block. Since a later patch shows the unregister being called right before the owning module is unloaded, that seems awefuly dangerous.

Is there some other magic going on making this safe?

The ltt_traces_sem mutex is intended to make this safe. However, the transport separation patch, contributed recently, uses its own transport_list_lock, which seems to be broken.

I will fix it by using ltt_traces_sem around :

        down(&ltt_traces_sem);
        list_for_each_entry(tran, &ltt_transport_list, node) {
                if (!strcmp(tran->name, trace_type)) {
                        transport = tran;
                        break;
                }
        }

        if (!transport) {
                err = EINVAL;
                printk(KERN_ERR "LTT : Transport %s is not present.\n", trace_type);
                goto trace_error;
        }

        if(!try_module_get(transport->owner)) {
                err = ENODEV;
                printk(KERN_ERR "LTT : Can't lock transport module.\n");
                goto trace_error;
        }
        up(&ltt_traces_sem);

And change the transport_list_lock for ltt_traces_sem everywhere else.

Thanks for spotting this bug,

Mathieu

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 
#2 lore raw
In reply to Martin Waitz
Quoted 18 lines · * Martin Waitz ([email protected]) wrote:

hoi :)

ltt/ltt-core.c line 101

Quoted 11 lines · Mathieu Desnoyers
ltt/ltt-core.c new file +11 -0
+int ltt_module_register(enum ltt_module_function name, void *function,
+		struct module *owner)
+{
+	int ret = 0;
+	
+	/* Protect these operations by disallowing them when tracing is
+	 * active */
+	if(ltt_traces.num_active_traces) {
+		ret = -EBUSY;
+		goto end;
+	}

what would happen otherwise?
can it happen that someone enables tracing between this check and the rest of the function?

The ltt-statedump module was still in my FIXME list. Let's address it.

In _ltt_trace_start, we have a try_module_get(ltt_statedump_owner) to make sure the module does not vanish when we try to call its functions. Then, in ltt_trace_start, we have a call to ltt_statedump_functor(trace), which uses functions in ltt-statedump (the code of this module is stripped from lttng-core). However, if ltt-statedump is not loaded when the try_module_get is first done, and only then is the functor is called, the module could vanish while the functor is executed.

Note that the ltt_traces.num_active_traces modifications only happen when the ltt_traces_sem mutex is taken.

However, this scenario is possible (which is bad) :

CPU 0 CPU 1
ltt_trace_start ltt_trace_stop
_ltt_trace_start : inc num_active_traces
_ltt_trace_stop : dec
num_active_traces
call statedump functor
unload ltt-statedump

A possible solution to this problem would be to increment the ltt-statedump reference count just around the function call :
in ltt_trace_start :

if(!try_module_get(ltt_statedump_owner)) {
error handling...
} else {
ltt_statedump_functor(trace);
module_put(ltt_statedump_owner);
}

And inside the ltt-statedump module, we need to make sure the kthread that is created exits before the functor returns. The side effect of this approach is that the lttctl control program will block until the end of state dump upon trace start. As it is not a problem, I will fix the code accordingly : remove the kthread.

Now for the ltt-filter module (which is not implemented yet) : as its function pointer will be called in the tracing critical path, we cannot afford to increment the module reference counter each time. This is why we have to combine making sure that the num_active_traces count is 0 and waiting for each tracing code to finish (all uses of ltt-filter are surrounded by preempt_disable : the waiting is done by synchronize_sched() while the ltt_traces_sem is taken, so no one can increment the num_active_traces).

The ltt-filter-control module (which is not implemented yet) is not a problem :
it is surrounded by try_module_get/module_put and does not depend on the traces being active or not.

The ltt-relay module is also protected by a mechanism similar to ltt-filter. It is necessary because the function pointer is on the critical path.

So, to answer your question : the if(ltt_traces.num_active_traces) check is only necessary to make sure that the filter module is not unloaded when a trace is active. However, adding a simple synchronize_sched() to ltt_module_unregister will fix this issue and add the ability to register/unregister a filter module when tracing is active.

I will remove this superfluous check in the next version.

Quoted 7 lines

ltt/ltt-core.c line 468

Quoted 4 lines
ltt/ltt-core.c new file +4 -0
+	new_trace->transport = transport;
+	new_trace->ops = &transport->ops;
+
+	err = -new_trace->ops->create_dirs(new_trace);

^ typo or intentional?

ltt_trace_create has a stardard of positive error values when ltt_relay_create_dirs has negative error values, without any need to do so. Good catch :) It will be fixed in the next version.

Thanks for the comments!

Mathieu

OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68