Skip to contentSkip to conversation

ftrace: add a tracepoint for __raise_softirq_irqoff()

PATCH Superseded Xiao Guangrong 9 participants 37 messages May 5, 2009 – Jul 9, 2009 Tracing

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 ftrace: add a tracepoint for __raise_softirq_irqoff()

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

---
 include/linux/interrupt.h  |    6 ++++++
 include/trace/events/irq.h |   18 ++++++++++++++++++
 kernel/softirq.c           |    8 ++++++++
 3 files changed, 32 insertions(+), 0 deletions(-)
include/linux/interrupt.h +6 -0
index dd574d5..3143341 100644
@@ -361,7 +361,13 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
+
+#ifdef CONFIG_TRACEPOINTS
+extern void __raise_softirq_irqoff(unsigned int nr);
+#else
 #define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+#endif
+
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 extern void wakeup_softirqd(void);
include/trace/events/irq.h +18 -0
index 32a9f7e..3c895bb 100644
@@ -128,6 +128,24 @@ TRACE_EVENT(softirq_exit,
 	TP_printk("softirq=%d action=%s", __entry->vec, __get_str(name))
 );
 
+TRACE_EVENT(irq_softirq_raise,
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+	),
+
+	TP_printk("softirq=%d action=%s is raised",
+		__entry->nr, softirq_to_name[__entry->nr])
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
kernel/softirq.c +8 -0
index b41fb71..bd0546b 100644
@@ -310,6 +310,14 @@ void irq_exit(void)
 	preempt_enable_no_resched();
 }
 
+#ifdef CONFIG_TRACEPOINTS
+inline void __raise_softirq_irqoff(unsigned int nr)
+{
+	trace_irq_softirq_raise(nr);
+	or_softirq_pending(1UL << (nr));
+}
+#endif
+
 /*
  * This function must run with irqs disabled!
  */
Signature · 1 line
1.6.1.2
#1 lore raw
Quoted 1 line

+TRACE_EVENT(irq_softirq_raise,

I think 'softirq_raise' is better.

Quoted 5 lines
include/trace/events/irq.h +5 -0
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+

include/trace/events/irq.h TRACE_EVENT(softirq_exit, · line 146

Quoted 10 lines
include/trace/events/irq.h +10 -0
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+	),
+
+	TP_printk("softirq=%d action=%s is raised",
+		__entry->nr, softirq_to_name[__entry->nr])

"softirq=%d action=%s" is sufficient.

Please see TRACE_EVENT(softirq_entry) and TRACE_EVENT(softirq_exit).

Quoted 24 lines
include/trace/events/irq.h +2 -0
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
kernel/softirq.c +8 -0
index b41fb71..bd0546b 100644
@@ -310,6 +310,14 @@ void irq_exit(void)
 	preempt_enable_no_resched();
 }
 
+#ifdef CONFIG_TRACEPOINTS
+inline void __raise_softirq_irqoff(unsigned int nr)
+{
+	trace_irq_softirq_raise(nr);
+	or_softirq_pending(1UL << (nr));
+}
+#endif
+
 /*
  * This function must run with irqs disabled!
  */
#3 lore raw
In reply to Xiao Guangrong
Quoted 30 lines · * Xiao Guangrong ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

---
 include/linux/interrupt.h  |    6 ++++++
 include/trace/events/irq.h |   18 ++++++++++++++++++
 kernel/softirq.c           |    8 ++++++++
 3 files changed, 32 insertions(+), 0 deletions(-)
include/linux/interrupt.h +0 -0
index dd574d5..3143341 100644

include/linux/interrupt.h asmlinkage void do_softirq(void); · line 368

Quoted 8 lines
include/linux/interrupt.h +4 -0
@@ -361,7 +361,13 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
+
+#ifdef CONFIG_TRACEPOINTS
+extern void __raise_softirq_irqoff(unsigned int nr);
+#else
 #define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)

Can you put the

	trace_irq_softirq_raise(nr);

directly in the define rather than adding this weird CONFIG_TRACEPOINTS? (and change the define for a static inline), something like :

static inline void __raise_softirq_irqoff(unsigned int nr)

{
	trace_irq_softirq_raise(nr);
	or_softirq_pending(1UL << (nr);
}

This would ensure we don't add a function call on the __raise_softirq_irqoff() fast-path.

Beware of circular include dependencies though. The tracepoints are meant not to have this kind of problems (I try to keep the dependencies very minimalistic), but I wonder if Steven's TRACE_EVENT is now ok on this aspect.

If TRACE_EVENT happens to pose problems with circular header dependencies, then try moving to the DECLARE_TRACE/DEFINE_TRACE scheme which has been more thoroughly tested as a first step.

Mathieu

Quoted 56 lines
include/linux/interrupt.h +2 -0
+#endif
+
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 extern void wakeup_softirqd(void);
include/trace/events/irq.h +18 -0
index 32a9f7e..3c895bb 100644
@@ -128,6 +128,24 @@ TRACE_EVENT(softirq_exit,
 	TP_printk("softirq=%d action=%s", __entry->vec, __get_str(name))
 );
 
+TRACE_EVENT(irq_softirq_raise,
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+	),
+
+	TP_printk("softirq=%d action=%s is raised",
+		__entry->nr, softirq_to_name[__entry->nr])
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
kernel/softirq.c +8 -0
index b41fb71..bd0546b 100644
@@ -310,6 +310,14 @@ void irq_exit(void)
 	preempt_enable_no_resched();
 }
 
+#ifdef CONFIG_TRACEPOINTS
+inline void __raise_softirq_irqoff(unsigned int nr)
+{
+	trace_irq_softirq_raise(nr);
+	or_softirq_pending(1UL << (nr));
+}
+#endif
+
 /*
  * This function must run with irqs disabled!
  */
Signature · 1 line
1.6.1.2
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#35 lore raw
In reply to Xiao Guangrong
Quoted 12 lines · Xiao Guangrong

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Quoted 7 lines

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

hi,

Could you please add 'docbook' style comments above TRACE_EVENTi()? We recently added a tracepoint docbook to the -tip and it already contains an 'irq' chapter, so you just need to add the comments for the raise softirq tracepoint. Please see:

http://marc.info/?l=linux-kernel&m=124118471411710&w=2

thanks,

-Jason

#2 lore raw
In reply to Li Zefan
Quoted 22 lines · Li Zefan
Quoted 1 line

+TRACE_EVENT(irq_softirq_raise,

I think 'softirq_raise' is better.

Quoted 5 lines
include/trace/events/irq.h +5 -0
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+

include/trace/events/irq.h TRACE_EVENT(softirq_exit, · line 146

Quoted 10 lines
include/trace/events/irq.h +10 -0
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+	),
+
+	TP_printk("softirq=%d action=%s is raised",
+		__entry->nr, softirq_to_name[__entry->nr])

"softirq=%d action=%s" is sufficient.

Quoted 2 lines

Please see TRACE_EVENT(softirq_entry) and TRACE_EVENT(softirq_exit).

Thanks, I will modify it as your suggestions. ;-)

Quoted 3 lines
Quoted 3 lines
include/trace/events/irq.h +2 -0
+);
+
#36 lore raw
In reply to Jason Baron
Quoted 21 lines · Jason Baron
Quoted 12 lines · Xiao Guangrong

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Quoted 7 lines

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 9 lines

hi,

Could you please add 'docbook' style comments above TRACE_EVENTi()? We recently added a tracepoint docbook to the -tip and it already contains an 'irq' chapter, so you just need to add the comments for the raise softirq tracepoint. Please see:

http://marc.info/?l=linux-kernel&m=124118471411710&w=2

Hi Jason:

I will add the comments for this tracepoint.
thanks!

Quoted 7 lines

thanks,

-Jason

#4 lore raw
In reply to Mathieu Desnoyers
Quoted 15 lines · Mathieu Desnoyers

include/linux/interrupt.h asmlinkage void do_softirq(void); · line 368

Quoted 7 lines · * Xiao Guangrong ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>

include/linux/interrupt.h +3 -0
+#ifdef CONFIG_TRACEPOINTS
+extern void __raise_softirq_irqoff(unsigned int nr);
+#else
 #define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)

Can you put the

	trace_irq_softirq_raise(nr);

directly in the define rather than adding this weird CONFIG_TRACEPOINTS? (and change the define for a static inline), something like :

Quoted 9 lines

static inline void __raise_softirq_irqoff(unsigned int nr)

{
	trace_irq_softirq_raise(nr);
	or_softirq_pending(1UL << (nr);
}

This would ensure we don't add a function call on the __raise_softirq_irqoff() fast-path.

We did this in v2, and we think it is better for same reason. But ...

Quoted 5 lines

Beware of circular include dependencies though. The tracepoints are meant not to have this kind of problems (I try to keep the dependencies very minimalistic), but I wonder if Steven's TRACE_EVENT is now ok on this aspect.

We encount this type of problem in v2.
So we move to this version(v3).

Quoted 4 lines

If TRACE_EVENT happens to pose problems with circular header dependencies, then try moving to the DECLARE_TRACE/DEFINE_TRACE scheme which has been more thoroughly tested as a first step.

IMHO, TRACE_EVENT framework is better for its more generic as ingo said, and it also provide ftrace support which means user can view tracepoint information from /debug/tracing/events.

Although this TRACE_EVENT happens to expose problems with circular header dependencies, we should not refuse using TRACE_EVENT, instead we should try to fix it for the whole TRACE_EVENT facility later.

Thanks

Quoted 2 lines

Mathieu

#5 lore raw
Quoted 45 lines · * Xiao Guangrong ([email protected]) wrote:
Quoted 15 lines · Mathieu Desnoyers

include/linux/interrupt.h asmlinkage void do_softirq(void); · line 368

Quoted 7 lines · * Xiao Guangrong ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>

include/linux/interrupt.h +3 -0
+#ifdef CONFIG_TRACEPOINTS
+extern void __raise_softirq_irqoff(unsigned int nr);
+#else
 #define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)

Can you put the

	trace_irq_softirq_raise(nr);

directly in the define rather than adding this weird CONFIG_TRACEPOINTS? (and change the define for a static inline), something like :

Quoted 9 lines

static inline void __raise_softirq_irqoff(unsigned int nr)

{
	trace_irq_softirq_raise(nr);
	or_softirq_pending(1UL << (nr);
}

This would ensure we don't add a function call on the __raise_softirq_irqoff() fast-path.

We did this in v2, and we think it is better for same reason. But ...

Quoted 5 lines

Beware of circular include dependencies though. The tracepoints are meant not to have this kind of problems (I try to keep the dependencies very minimalistic), but I wonder if Steven's TRACE_EVENT is now ok on this aspect.

We encount this type of problem in v2.
So we move to this version(v3).

Quoted 4 lines

If TRACE_EVENT happens to pose problems with circular header dependencies, then try moving to the DECLARE_TRACE/DEFINE_TRACE scheme which has been more thoroughly tested as a first step.

Quoted 8 lines

IMHO, TRACE_EVENT framework is better for its more generic as ingo said, and it also provide ftrace support which means user can view tracepoint information from /debug/tracing/events.

Although this TRACE_EVENT happens to expose problems with circular header dependencies, we should not refuse using TRACE_EVENT, instead we should try to fix it for the whole TRACE_EVENT facility later.

I partially agree with you :

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

Mathieu

Quoted 4 lines

Thanks

Quoted 2 lines

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#6 lore raw
Quoted 7 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

The majority of tracepoints happen is C files. Those few cases where they are used in headers is where the issues arise.

But...

I did not want to uglify all trace event headers with:

#ifdef CREATE_FOO_TRACE_POINTS
#undef CREATE_FOO_TRACE_POINTS
#include <trace/define_trace.h>
#endif

We would only need to do that for those trace points that need to be included in header files. Then the declaration C file would need to define both CREATE_FOO_TRACE_POINTS and CREATE_TRACE_POINTS

#define CREATE_FOO_TRACE_POINTS
#define CRATE_TRACE_POINTS
#include <trace/events/foo.h>

But this is pretty trivial to solve, and I do not consider it a show stopper or a major header dependency problem.

-- Steve

#7 lore raw
Quoted 31 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

The majority of tracepoints happen is C files. Those few cases where they are used in headers is where the issues arise.

But...

I did not want to uglify all trace event headers with:

#ifdef CREATE_FOO_TRACE_POINTS
#undef CREATE_FOO_TRACE_POINTS
#include <trace/define_trace.h>
#endif

We would only need to do that for those trace points that need to be included in header files. Then the declaration C file would need to define both CREATE_FOO_TRACE_POINTS and CREATE_TRACE_POINTS

#define CREATE_FOO_TRACE_POINTS
#define CRATE_TRACE_POINTS
#include <trace/events/foo.h>

Quoted 5 lines

But this is pretty trivial to solve, and I do not consider it a show stopper or a major header dependency problem.

-- Steve

Hrm, is there any way to solve it elegantly ?

What we really need is to see the cases where TRACE_EVENT() is used as a declaration vs the case where it expands TP_STRUCT__entry/TP_fast_assign/TP_printk as having different dependencies. The problem comes when we bring the include dependencies of the TP_fast_assign part into the tracepoint header and it becomes a dependency of the TRACE_EVENT() declaration-only part.

Can we do the following ?

All tracepoint headers could surround the include dependencies by :

#ifdef BUILD_EVENTS
#include <veryannoyingheaderdependency.h>
#endif

And then we follow this by the TRACE_EVENT() declarations.

BUILD_EVENTS would only be defined in kernel/trace/events.c.

I think it should work, but it looks a bit too simple, so I may have missed something... ?

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#8 lore raw
Quoted 51 lines · Mathieu Desnoyers
Quoted 31 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

The majority of tracepoints happen is C files. Those few cases where they are used in headers is where the issues arise.

But...

I did not want to uglify all trace event headers with:

#ifdef CREATE_FOO_TRACE_POINTS
#undef CREATE_FOO_TRACE_POINTS
#include <trace/define_trace.h>
#endif

We would only need to do that for those trace points that need to be included in header files. Then the declaration C file would need to define both CREATE_FOO_TRACE_POINTS and CREATE_TRACE_POINTS

#define CREATE_FOO_TRACE_POINTS
#define CRATE_TRACE_POINTS
#include <trace/events/foo.h>

Quoted 5 lines

But this is pretty trivial to solve, and I do not consider it a show stopper or a major header dependency problem.

-- Steve

Hrm, is there any way to solve it elegantly ?

What we really need is to see the cases where TRACE_EVENT() is used as a declaration vs the case where it expands TP_STRUCT__entry/TP_fast_assign/TP_printk as having different dependencies. The problem comes when we bring the include dependencies of the TP_fast_assign part into the tracepoint header and it becomes a dependency of the TRACE_EVENT() declaration-only part.

Can we do the following ?

All tracepoint headers could surround the include dependencies by :

Quoted 7 lines

#ifdef BUILD_EVENTS
#include <veryannoyingheaderdependency.h>
#endif

And then we follow this by the TRACE_EVENT() declarations.

BUILD_EVENTS would only be defined in kernel/trace/events.c.

That file no longer exists. All the events are created with the define_trace.h or ftrace.h file.

We could do something like your

#ifdef BUILD_EVENTS /* or better name BUILDING_EVENTS ?? */ #include <allmyannoyingheaders.h>
[...]
#endif

And then have the define_trace.h file add define it. But this again adds more annoying CPP commands to all trace_event headers.

-- Steve

Quoted 11 lines

I think it should work, but it looks a bit too simple, so I may have missed something... ?

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#9 lore raw
Quoted 69 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 51 lines · Mathieu Desnoyers
Quoted 31 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

The majority of tracepoints happen is C files. Those few cases where they are used in headers is where the issues arise.

But...

I did not want to uglify all trace event headers with:

#ifdef CREATE_FOO_TRACE_POINTS
#undef CREATE_FOO_TRACE_POINTS
#include <trace/define_trace.h>
#endif

We would only need to do that for those trace points that need to be included in header files. Then the declaration C file would need to define both CREATE_FOO_TRACE_POINTS and CREATE_TRACE_POINTS

#define CREATE_FOO_TRACE_POINTS
#define CRATE_TRACE_POINTS
#include <trace/events/foo.h>

Quoted 5 lines

But this is pretty trivial to solve, and I do not consider it a show stopper or a major header dependency problem.

-- Steve

Hrm, is there any way to solve it elegantly ?

What we really need is to see the cases where TRACE_EVENT() is used as a declaration vs the case where it expands TP_STRUCT__entry/TP_fast_assign/TP_printk as having different dependencies. The problem comes when we bring the include dependencies of the TP_fast_assign part into the tracepoint header and it becomes a dependency of the TRACE_EVENT() declaration-only part.

Can we do the following ?

All tracepoint headers could surround the include dependencies by :

Quoted 7 lines

#ifdef BUILD_EVENTS
#include <veryannoyingheaderdependency.h>
#endif

And then we follow this by the TRACE_EVENT() declarations.

BUILD_EVENTS would only be defined in kernel/trace/events.c.

That file no longer exists. All the events are created with the define_trace.h or ftrace.h file.

We could do something like your

Quoted 8 lines

#ifdef BUILD_EVENTS /* or better name BUILDING_EVENTS ?? */ #include <allmyannoyingheaders.h>
[...]
#endif

And then have the define_trace.h file add define it. But this again adds more annoying CPP commands to all trace_event headers.

Yes, but this would solve most of the include dependency problems at once. We would only have dependency on preempt.h left, which alone is easier to deal with. And if headers don't need to include any annoying headers, they don't need to use the ifdef BUILDING_EVENTS. But my point is that when it's needed (and we already see two cases where it's needed, with pvops instrumentation and softirq instrumentation, and I guess we'll see much more), it's good to have this infrastructure in place, so people don't end up trying to do hacks to the kernel code changing inlines for function calls to try to deal with the circular include issue.

I think it's

a) needed
b) it does not hurt anyone who does not need it.

So I would definitely recommend adding such define.

Mathieu

Quoted 13 lines

-- Steve

Quoted 11 lines

I think it should work, but it looks a bit too simple, so I may have missed something... ?

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#10 lore raw
Quoted 22 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
Quoted 4 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Yes, but this would solve most of the include dependency problems at once. We would only have dependency on preempt.h left, which alone is easier to deal with. And if headers don't need to include any annoying headers, they don't need to use the ifdef BUILDING_EVENTS. But my point is that when it's needed (and we already see two cases where it's needed, with pvops instrumentation and softirq instrumentation, and I guess we'll see much more), it's good to have this infrastructure in place, so people don't end up trying to do hacks to the kernel code changing inlines for function calls to try to deal with the circular include issue.

Quoted 7 lines

I think it's

a) needed
b) it does not hurt anyone who does not need it.

So I would definitely recommend adding such define.

Sorry for my poor English.
Problem is not only in TP_printk, but also in TP_PROTO if we add a tracer in header file.
So, it can't be solved only by "get rid of including ftrace parts"

Take v2 patch for example:
(it is at: http://marc.info/?l=linux-kernel&m=124081169727739&w=2)

In order to trace __raise_softirq_irqoff(), we should add a trace function in __raise_softirq_irqoff() and include <include/trace/irq.h> in interrupte.h.
If we put <include/trace/irq.h> on top of linux/irq.h, we will see a warning of "struct softirq_action declared inside parameter list" in compile. It is because struct irqaction's definition is bypasswd before TP_PROTO.

To say it simple:
sched.c:
include interrupte.h

	|
	|-->include irq.h
	|	|-->include interrupte.h (bypassed)
	|	|-->TP_PROTO(int irq, struct irqaction *action),
	|	    (but struct softirq_action is not declared before,
	|	     it raise a compile warning:struct softirq_action declared
	|	     inside parameter list.
	|-> ......

I can't see the solvent for this in your discuss or I understand wrong?

Xiao

Quoted 15 lines

Mathieu

Quoted 12 lines

-- Steve

Quoted 10 lines

I think it should work, but it looks a bit too simple, so I may have missed something... ?

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#11 lore raw
Quoted 41 lines · * Xiao Guangrong ([email protected]) wrote:
Quoted 22 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 5 lines · Mathieu Desnoyers
Quoted 4 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Yes, but this would solve most of the include dependency problems at once. We would only have dependency on preempt.h left, which alone is easier to deal with. And if headers don't need to include any annoying headers, they don't need to use the ifdef BUILDING_EVENTS. But my point is that when it's needed (and we already see two cases where it's needed, with pvops instrumentation and softirq instrumentation, and I guess we'll see much more), it's good to have this infrastructure in place, so people don't end up trying to do hacks to the kernel code changing inlines for function calls to try to deal with the circular include issue.

Quoted 7 lines

I think it's

a) needed
b) it does not hurt anyone who does not need it.

So I would definitely recommend adding such define.

Sorry for my poor English.
Problem is not only in TP_printk, but also in TP_PROTO if we add a tracer in header file.
So, it can't be solved only by "get rid of including ftrace parts"

Take v2 patch for example:
(it is at: http://marc.info/?l=linux-kernel&m=124081169727739&w=2)

Quoted 8 lines

In order to trace __raise_softirq_irqoff(), we should add a trace function in __raise_softirq_irqoff() and include <include/trace/irq.h> in interrupte.h.
If we put <include/trace/irq.h> on top of linux/irq.h, we will see a warning of "struct softirq_action declared inside parameter list" in compile. It is because struct irqaction's definition is bypasswd before TP_PROTO.

Then add a forward declaration of

struct softirqaction;

At the top of trace/irq.h. I did it in quite a few places in the LTTng tree. TP_PROTO just needs a forward declaration, not the full structure declaration.

Only the ftrace-specific parts will need the #include <linux/irq.h>, which we should put in a #ifdef :

sched.c:

#include linux/interrupt.h
        |
        |-->include <trace/irq.h>
            #ifdef BUILDING_EVENTS
            /*
             * Includes needed by TP_fast_assign, TP_printk, and
             * TP_STRUCT__entry :
             */
            # include <linux/interrupt.h>
            #else
	    /* Forward declarations for TP_PROTO */
            struct softirq_action;
            #endif

            TRACE_EVENT(......);
            |-->TP_PROTO(int irq, struct irqaction *action),
                (but struct softirq_action is not declared before,
                 it raise a compile warning:struct softirq_action declared
                 inside parameter list.
            |-> ......

Mathieu

Quoted 20 lines

I can't see the solvent for this in your discuss or I understand wrong?

Xiao

Quoted 15 lines

Mathieu

Quoted 12 lines

-- Steve

Quoted 10 lines

I think it should work, but it looks a bit too simple, so I may have missed something... ?

Mathieu

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#29 lore raw
In reply to Mathieu Desnoyers
Quoted 11 lines · Mathieu Desnoyers

I partially agree with you :

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

I partially agree with you:

Yes, Circular header dependencies is a real problem with TRACE_EVENT right now. It is also a problem with DECLARE_TRACE. It's a stubborn disease with C-Language (for complex headers). Can we fix C-Language?

o Macros in header (!CREATE_TRACE_POINTS)

When CREATE_TRACE_POINTS is not defined, TRACE_EVENT is definitely the same as DECLARE_TRACE. Actually, TRACE_EVENT is:

#define TRACE_EVENT(name, proto, args, struct, assign, print) \
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))

So TRACE_EVENT and DECLARE_TRACE are the same in header files. And so TRACE_EVENT and DECLARE_TRACE have the same advantages and disadvantages. More TRACE_EVENT equals to a known-good solution.

o Macros in c-file

tracepoint uses DEFINE_TRACE only.

ftrace uses CREATE_TRACE_POINTS + TRACE_EVENT:

	#define CREATE_TRACE_POINTS
	#include <trace/events/sched.h> (which uses TRACE_EVENT)

ftrace generates more code which uses the tracepoints.

Quoted 9 lines

Then add a forward declaration of

struct softirqaction;

At the top of trace/irq.h. I did it in quite a few places in the LTTng tree. TP_PROTO just needs a forward declaration, not the full structure declaration.

Thank you for your valuable suggestions.

You are the father of tracepoint and LTTng, your experience in LTTng is very useful for ftrace.

I'm glad for your suggestions.

Xiao Guangrong, could you add forward declarations of

struct irqaction;
struct softirq_action;

at the top of trace/irq.h as Mathieu's suggestions. (and remove "#include <linux/interrupt.h>")

Lai

#30 lore raw
Quoted 59 lines · * Lai Jiangshan ([email protected]) wrote:
Quoted 11 lines · Mathieu Desnoyers

I partially agree with you :

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

I partially agree with you:

Yes, Circular header dependencies is a real problem with TRACE_EVENT right now. It is also a problem with DECLARE_TRACE. It's a stubborn disease with C-Language (for complex headers). Can we fix C-Language?

o Macros in header (!CREATE_TRACE_POINTS)

When CREATE_TRACE_POINTS is not defined, TRACE_EVENT is definitely the same as DECLARE_TRACE. Actually, TRACE_EVENT is:

#define TRACE_EVENT(name, proto, args, struct, assign, print) \
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))

So TRACE_EVENT and DECLARE_TRACE are the same in header files. And so TRACE_EVENT and DECLARE_TRACE have the same advantages and disadvantages. More TRACE_EVENT equals to a known-good solution.

o Macros in c-file

tracepoint uses DEFINE_TRACE only.

ftrace uses CREATE_TRACE_POINTS + TRACE_EVENT:

	#define CREATE_TRACE_POINTS
	#include <trace/events/sched.h> (which uses TRACE_EVENT)

ftrace generates more code which uses the tracepoints.

Quoted 9 lines

Then add a forward declaration of

struct softirqaction;

At the top of trace/irq.h. I did it in quite a few places in the LTTng tree. TP_PROTO just needs a forward declaration, not the full structure declaration.

Thank you for your valuable suggestions.

You are the father of tracepoint and LTTng, your experience in LTTng is very useful for ftrace.

I'm glad for your suggestions.

Quoted 8 lines

Xiao Guangrong, could you add forward declarations of

struct irqaction;
struct softirq_action;

at the top of trace/irq.h as Mathieu's suggestions. (and remove "#include <linux/interrupt.h>")

You will probably still need something like :

#ifdef CREATE_TRACE_POINTS
#include <linux/interrupt.h>
#else
struct irqaction;
struct softirq_action;
#endif

So that FTRACE has the header dependencies it needs to build.

Mathieu

Quoted 3 lines

Lai

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#31 lore raw
Quoted 65 lines · Mathieu Desnoyers
Quoted 55 lines · * Lai Jiangshan ([email protected]) wrote:
Quoted 10 lines · Mathieu Desnoyers

I partially agree with you :

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

I partially agree with you:

Yes, Circular header dependencies is a real problem with TRACE_EVENT right now. It is also a problem with DECLARE_TRACE. It's a stubborn disease with C-Language (for complex headers). Can we fix C-Language?

o Macros in header (!CREATE_TRACE_POINTS)

When CREATE_TRACE_POINTS is not defined, TRACE_EVENT is definitely the same as DECLARE_TRACE. Actually, TRACE_EVENT is:

#define TRACE_EVENT(name, proto, args, struct, assign, print) \
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))

So TRACE_EVENT and DECLARE_TRACE are the same in header files. And so TRACE_EVENT and DECLARE_TRACE have the same advantages and disadvantages. More TRACE_EVENT equals to a known-good solution.

o Macros in c-file

tracepoint uses DEFINE_TRACE only.

ftrace uses CREATE_TRACE_POINTS + TRACE_EVENT:

	#define CREATE_TRACE_POINTS
	#include <trace/events/sched.h> (which uses TRACE_EVENT)

ftrace generates more code which uses the tracepoints.

Quoted 8 lines

Then add a forward declaration of

struct softirqaction;

At the top of trace/irq.h. I did it in quite a few places in the LTTng tree. TP_PROTO just needs a forward declaration, not the full structure declaration.

Thank you for your valuable suggestions.

You are the father of tracepoint and LTTng, your experience in LTTng is very useful for ftrace.

I'm glad for your suggestions.

Quoted 8 lines

Xiao Guangrong, could you add forward declarations of

struct irqaction;
struct softirq_action;

at the top of trace/irq.h as Mathieu's suggestions. (and remove "#include <linux/interrupt.h>")

Quoted 9 lines

You will probably still need something like :

#ifdef CREATE_TRACE_POINTS
#include <linux/interrupt.h>
#else
struct irqaction;
struct softirq_action;
#endif

It's not needed for trace/events/irq.h

Yes, it's a solution.

But I don't think we have to do this, CREATE_TRACE_POINTS is needed only _once_ for every <trace/events/xxxx.h>

The .c file which defines CREATE_TRACE_POINTS can provide (had provided likely) things like "#include <linux/interrupt.h>"

See kernel/softirq.c:
#include <linux/interrupt.h>
......
......
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>

I don't think it's a problem, CREATE_TRACE_POINTS is defined only once for a <trace/events/xxxx.h>.

Lai.

#32 lore raw
Quoted 24 lines · Lai Jiangshan
Quoted 5 lines · Mathieu Desnoyers
Quoted 1 line · * Lai Jiangshan ([email protected]) wrote:

Mathieu Desnoyers wrote:

#endif

It's not needed for trace/events/irq.h

Yes, it's a solution.

But I don't think we have to do this, CREATE_TRACE_POINTS is needed only _once_ for every <trace/events/xxxx.h>

The .c file which defines CREATE_TRACE_POINTS can provide (had provided likely) things like "#include <linux/interrupt.h>"

See kernel/softirq.c:
#include <linux/interrupt.h>
......
......
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>

Quoted 3 lines

I don't think it's a problem, CREATE_TRACE_POINTS is defined only once for a <trace/events/xxxx.h>.

Yes, I think Lai's argument is reasonable,
the .c file which use ftrace already include the relevant head file. I will make a patch as your suggestions.

Thanks! ;-)

Quoted 4 lines

Lai.

#12 lore raw

PATCH v4 1/1 ftrace: add a tracepoint for __raise_softirq_irqoff()

In reply to Xiao Guangrong

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

---
 include/linux/interrupt.h  |    9 ++++++++-
 include/trace/events/irq.h |   27 ++++++++++++++++++++++++++-
 2 files changed, 34 insertions(+), 2 deletions(-)
include/linux/interrupt.h +8 -1
index dd574d5..51e7909 100644
@@ -18,6 +18,7 @@
 #include <asm/atomic.h>
 #include <asm/ptrace.h>
 #include <asm/system.h>
+#include <trace/events/irq.h>
 
 /*
  * These correspond to the IORESOURCE_IRQ_* defines in
@@ -361,7 +362,13 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
-#define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+
+static inline void __raise_softirq_irqoff(unsigned int nr)
+{
+	trace_softirq_raise(nr);
+	or_softirq_pending(1UL << nr);
+}
+
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 extern void wakeup_softirqd(void);
include/trace/events/irq.h +26 -1
index 32a9f7e..08fa63b 100644
@@ -2,11 +2,13 @@
 #define _TRACE_IRQ_H
 
 #include <linux/tracepoint.h>
-#include <linux/interrupt.h>
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM irq
 
+struct irqaction;
+struct softirq_action;
+
 /**
  * irq_handler_entry - called immediately before the irq action handler
  * @irq: irq number
@@ -128,6 +130,29 @@ TRACE_EVENT(softirq_exit,
 	TP_printk("softirq=%d action=%s", __entry->vec, __get_str(name))
 );
 
+/**
+ * softirq_raise - called immediately when a softirq is raised
+ * @nr: softirq vector number
+ */
+TRACE_EVENT(softirq_raise,
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr			)
+		__string(	name,		softirq_to_name[nr]	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+		__assign_str(name, softirq_to_name[nr]);
+	),
+
+	TP_printk("softirq=%u action=%s", __entry->nr, __get_str(name))
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
Signature · 1 line
1.6.1.2
#33 lore raw
In reply to Lai Jiangshan
Quoted 86 lines · * Lai Jiangshan ([email protected]) wrote:
Quoted 65 lines · Mathieu Desnoyers
Quoted 55 lines · * Lai Jiangshan ([email protected]) wrote:
Quoted 10 lines · Mathieu Desnoyers

I partially agree with you :

Yes, we should try to fix TRACE_EVENT, but we should fix it _before_ we start using it widely. Circular header dependencies is a real problem with TRACE_EVENT right now.

Until we fix this, I will be tempted to stay with a known-good solution, which is DECLARE/DEFINE_TRACE.

I partially agree with you:

Yes, Circular header dependencies is a real problem with TRACE_EVENT right now. It is also a problem with DECLARE_TRACE. It's a stubborn disease with C-Language (for complex headers). Can we fix C-Language?

o Macros in header (!CREATE_TRACE_POINTS)

When CREATE_TRACE_POINTS is not defined, TRACE_EVENT is definitely the same as DECLARE_TRACE. Actually, TRACE_EVENT is:

#define TRACE_EVENT(name, proto, args, struct, assign, print) \
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))

So TRACE_EVENT and DECLARE_TRACE are the same in header files. And so TRACE_EVENT and DECLARE_TRACE have the same advantages and disadvantages. More TRACE_EVENT equals to a known-good solution.

o Macros in c-file

tracepoint uses DEFINE_TRACE only.

ftrace uses CREATE_TRACE_POINTS + TRACE_EVENT:

	#define CREATE_TRACE_POINTS
	#include <trace/events/sched.h> (which uses TRACE_EVENT)

ftrace generates more code which uses the tracepoints.

Quoted 8 lines

Then add a forward declaration of

struct softirqaction;

At the top of trace/irq.h. I did it in quite a few places in the LTTng tree. TP_PROTO just needs a forward declaration, not the full structure declaration.

Thank you for your valuable suggestions.

You are the father of tracepoint and LTTng, your experience in LTTng is very useful for ftrace.

I'm glad for your suggestions.

Quoted 8 lines

Xiao Guangrong, could you add forward declarations of

struct irqaction;
struct softirq_action;

at the top of trace/irq.h as Mathieu's suggestions. (and remove "#include <linux/interrupt.h>")

Quoted 9 lines

You will probably still need something like :

#ifdef CREATE_TRACE_POINTS
#include <linux/interrupt.h>
#else
struct irqaction;
struct softirq_action;
#endif

It's not needed for trace/events/irq.h

Yes, it's a solution.

But I don't think we have to do this, CREATE_TRACE_POINTS is needed only _once_ for every <trace/events/xxxx.h>

The .c file which defines CREATE_TRACE_POINTS can provide (had provided likely) things like "#include <linux/interrupt.h>"

Quoted 7 lines

See kernel/softirq.c:
#include <linux/interrupt.h>
......
......
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>

Isn't it all included under kernel/trace/events.c ? e.g. :

#include <trace/trace_events.h>

#include "trace_output.h"

#include "trace_events_stage_1.h"
#include "trace_events_stage_2.h"
#include "trace_events_stage_3.h"

Therefore, adding a #include <linux/interrupt.h> would be just weird here. We would have to do that for every tracepoint headers, and we would just lose the ability to keep track of which tracepoint header has which include dependency.

I thought the goal of TRACE_EVENT() was exactly the opposite : to declare everything in one location....

Therefore, if you agree that it's good to have TRACE_EVENT() declaring everything in one location, then I don't see why you would argue to move the include dependencies in the very remote kernel/trace/events.c file ?

Mathieu

Quoted 5 lines

I don't think it's a problem, CREATE_TRACE_POINTS is defined only once for a <trace/events/xxxx.h>.

Lai.

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#13 lore raw
In reply to Xiao Guangrong
Quoted 16 lines · * Xiao Guangrong ([email protected]) wrote:

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

Mathieu

Quoted 88 lines
---
 include/linux/interrupt.h  |    9 ++++++++-
 include/trace/events/irq.h |   27 ++++++++++++++++++++++++++-
 2 files changed, 34 insertions(+), 2 deletions(-)
include/linux/interrupt.h +8 -1
index dd574d5..51e7909 100644
@@ -18,6 +18,7 @@
 #include <asm/atomic.h>
 #include <asm/ptrace.h>
 #include <asm/system.h>
+#include <trace/events/irq.h>
 
 /*
  * These correspond to the IORESOURCE_IRQ_* defines in
@@ -361,7 +362,13 @@ asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
-#define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
+
+static inline void __raise_softirq_irqoff(unsigned int nr)
+{
+	trace_softirq_raise(nr);
+	or_softirq_pending(1UL << nr);
+}
+
 extern void raise_softirq_irqoff(unsigned int nr);
 extern void raise_softirq(unsigned int nr);
 extern void wakeup_softirqd(void);
include/trace/events/irq.h +26 -1
index 32a9f7e..08fa63b 100644
@@ -2,11 +2,13 @@
 #define _TRACE_IRQ_H
 
 #include <linux/tracepoint.h>
-#include <linux/interrupt.h>
 
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM irq
 
+struct irqaction;
+struct softirq_action;
+
 /**
  * irq_handler_entry - called immediately before the irq action handler
  * @irq: irq number
@@ -128,6 +130,29 @@ TRACE_EVENT(softirq_exit,
 	TP_printk("softirq=%d action=%s", __entry->vec, __get_str(name))
 );
 
+/**
+ * softirq_raise - called immediately when a softirq is raised
+ * @nr: softirq vector number
+ */
+TRACE_EVENT(softirq_raise,
+
+	TP_PROTO(unsigned int nr),
+
+	TP_ARGS(nr),
+
+	TP_STRUCT__entry(
+		__field(	unsigned int,	nr			)
+		__string(	name,		softirq_to_name[nr]	)
+	),
+
+	TP_fast_assign(
+		__entry->nr	= nr;
+		__assign_str(name, softirq_to_name[nr]);
+	),
+
+	TP_printk("softirq=%u action=%s", __entry->nr, __get_str(name))
+);
+
 #endif /*  _TRACE_IRQ_H */
 
 /* This part must be outside protection */
Signature · 1 line
1.6.1.2
Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#34 lore raw
In reply to Mathieu Desnoyers
Quoted 20 lines · Mathieu Desnoyers
Quoted 12 lines
Quoted 1 line

It's not needed for trace/events/irq.h

Yes, it's a solution.

But I don't think we have to do this, CREATE_TRACE_POINTS is needed only _once_ for every <trace/events/xxxx.h>

The .c file which defines CREATE_TRACE_POINTS can provide (had provided likely) things like "#include <linux/interrupt.h>"

Quoted 7 lines

See kernel/softirq.c:
#include <linux/interrupt.h>
......
......
#define CREATE_TRACE_POINTS
#include <trace/events/irq.h>

Quoted 1 line

Isn't it all included under kernel/trace/events.c ? e.g. :

Mathieu,

I've told you this two or three times before. That file no longer exists. Maybe it is still there for .30, but these updates are for .31 (too late for .30). The code to make TRACE_EVENT work for modules removed the events.c file and got rid of those nasty stages.

-- Steve

Quoted 33 lines

#include <trace/trace_events.h>

#include "trace_output.h"

#include "trace_events_stage_1.h"
#include "trace_events_stage_2.h"
#include "trace_events_stage_3.h"

Therefore, adding a #include <linux/interrupt.h> would be just weird here. We would have to do that for every tracepoint headers, and we would just lose the ability to keep track of which tracepoint header has which include dependency.

I thought the goal of TRACE_EVENT() was exactly the opposite : to declare everything in one location....

Therefore, if you agree that it's good to have TRACE_EVENT() declaring everything in one location, then I don't see why you would argue to move the include dependencies in the very remote kernel/trace/events.c file ?

Mathieu

Quoted 5 lines

I don't think it's a problem, CREATE_TRACE_POINTS is defined only once for a <trace/events/xxxx.h>.

Lai.

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
maintainer
#14 lore raw
In reply to Mathieu Desnoyers
Quoted 23 lines · Mathieu Desnoyers
Quoted 16 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

-- Steve

#15 lore raw
Quoted 30 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 28 lines · Mathieu Desnoyers
Quoted 16 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Quoted 5 lines
Quoted 2 lines

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

OK, thanks for the update :)

Mathieu

Quoted 2 lines

-- Steve

Signature · 2 lines
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
#16 lore raw
In reply to Steven Rostedt
Quoted 28 lines · Steven Rostedt
Quoted 27 lines · Mathieu Desnoyers
Quoted 16 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:

	http://marc.info/?l=linux-kernel&m=123791201816245&w=2

This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Quoted 5 lines
Quoted 2 lines

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Thanks for your review, is great help to me. ;-)

Quoted 4 lines

-- Steve

#17 lore raw

* From: "Xiao Guangrong" <[email protected]>

Quoted 32 lines
Quoted 31 lines · Steven Rostedt
Quoted 29 lines · Mathieu Desnoyers
Quoted 16 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

Quoted 8 lines
Quoted 2 lines

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

Thanks
Zhaolei

Quoted 6 lines
Quoted 5 lines

-- Steve

���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay�ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�

#18 lore raw
Quoted 40 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 39 lines
Quoted 28 lines · Steven Rostedt
Quoted 27 lines · Mathieu Desnoyers
Quoted 16 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Quoted 6 lines

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Quoted 5 lines
Quoted 2 lines

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 6 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

#19 lore raw
Quoted 40 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

Hello, Ingo

Thanks for your reply.

Since it is unrelated to tracing, why not add this tracepoint first and fix the softirq code later? The tracepoint won't add any burden to the fix work.

Thanks
Zhaolei

Quoted 48 lines

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

#20 lore raw
In reply to Ingo Molnar
Quoted 72 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

Quoted 7 lines

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

It's a nice solution. But I think it would be nicer when it is changed a little.

The new API raise_softirq_check() will become a very hard _burden_ to the users of raise_softirq*(). They have to find out a proper place to place the "raise_softirq_check();". It's not an easy things, functions are complicated and hard to be determined WHERE is the process context and WHEN(a function may be called from multi kinds of context).

Instead, I prefer that raise_softirq_check() is hidden from users. We call raise_softirq_check() from schedule(), it will handle the un-handle softirqs in time(if un-handle softirqs are too much, it'll wakeup the ksoftirqd).

Lai

Quoted 19 lines

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

#21 lore raw

RFC PATCH 1/1 softirq: fix ksoftirq starved

Quoted 88 lines · Lai Jiangshan
Quoted 71 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

Quoted 7 lines

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

It's a nice solution. But I think it would be nicer when it is changed a little.

The new API raise_softirq_check() will become a very hard _burden_ to the users of raise_softirq*(). They have to find out a proper place to place the "raise_softirq_check();". It's not an easy things, functions are complicated and hard to be determined WHERE is the process context and WHEN(a function may be called from multi kinds of context).

Quoted 28 lines

Instead, I prefer that raise_softirq_check() is hidden from users. We call raise_softirq_check() from schedule(), it will handle the un-handle softirqs in time (if un-handle softirqs are too much, it'll wakeup the ksoftirqd).

Lai

Quoted 19 lines

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

Subject: [PATCH RFC] softirq: fix ksoftirq starved

The API:
void raise_softirq_irqoff(unsigned int nr)

{
	__raise_softirq_irqoff(nr);

	/*
	 * If we're in an interrupt or softirq, we're done
	 * (this also catches softirq-disabled code). We will
	 * actually run the softirq once we return from
	 * the irq or softirq.
	 *
	 * Otherwise we wake up ksoftirqd to make sure we
	 * schedule the softirq soon.
	 */
	if (!in_interrupt())
		wakeup_softirqd();
}

is broken with RT tasks, as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

To make pending softirq is called in time, we check and run the pending softirq in schedule().

---

include/linux/interrupt.h +1 -0
index dd574d5..f59e552 100644
@@ -359,6 +359,7 @@ struct softirq_action
 
 asmlinkage void do_softirq(void);
 asmlinkage void __do_softirq(void);
+extern void schedule_softirq_check(void);
 extern void open_softirq(int nr, void (*action)(struct softirq_action *));
 extern void softirq_init(void);
 #define __raise_softirq_irqoff(nr) do { or_softirq_pending(1UL << (nr)); } while (0)
kernel/sched.c +1 -0
index cfb0456..21b5f67 100644
@@ -5307,6 +5307,7 @@ need_resched:
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
 
+	schedule_softirq_check();
 	schedule_debug(prev);
 
 	if (sched_feat(HRTICK))
kernel/softirq.c +24 -3
index b41fb71..59d142b 100644
@@ -55,6 +55,7 @@ EXPORT_SYMBOL(irq_stat);
 static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp;
 
 static DEFINE_PER_CPU(struct task_struct *, ksoftirqd);
+static DEFINE_PER_CPU(int, schedule_softirq_check);
 
 char *softirq_to_name[NR_SOFTIRQS] = {
 	"HI", "TIMER", "NET_TX", "NET_RX", "BLOCK",
@@ -76,6 +77,14 @@ void wakeup_softirqd(void)
 		wake_up_process(tsk);
 }
 
+static void set_schedule_softirq_check(void)
+{
+	if (current != __get_cpu_var(ksoftirqd)) {
+		__get_cpu_var(schedule_softirq_check) = 1;
+		set_tsk_need_resched(current);
+	}
+}
+
 /*
  * This one is for softirq.c-internal use,
  * where hardirqs are disabled legitimately:
@@ -243,6 +252,7 @@ restart:
 
 	lockdep_softirq_exit();
 
+	__get_cpu_var(schedule_softirq_check) = 0;
 	account_system_vtime(current);
 	_local_bh_enable();
 }
@@ -269,6 +279,12 @@ asmlinkage void do_softirq(void)
 
 #endif
 
+void schedule_softirq_check(void)
+{
+	if (__get_cpu_var(schedule_softirq_check))
+		do_softirq();
+}
+
 /*
  * Enter an interrupt context.
  */
@@ -323,11 +339,16 @@ inline void raise_softirq_irqoff(unsigned int nr)
 	 * actually run the softirq once we return from
 	 * the irq or softirq.
 	 *
-	 * Otherwise we wake up ksoftirqd to make sure we
-	 * schedule the softirq soon.
+	 * Otherwise we check and run the pending softirq in schedule().
+	 *
+	 * NOTE: It's sometimes helpless to wakeup the ksoftirqd here.
+	 *       Because when current task is a real-time task(which
+	 *       has higher priority) or a real-time task is on standby
+	 *       in this cpu, ksoftirqd won't execute, we starve softirq
+	 *       execution.
 	 */
 	if (!in_interrupt())
-		wakeup_softirqd();
+		set_schedule_softirq_check();
 }
 
 void raise_softirq(unsigned int nr)
#22 lore raw

Re: [PATCH RFC] softirq: fix ksoftirq starved

kernel/sched.c need_resched: · line 5311

Quoted 8 lines · Lai Jiangshan
kernel/sched.c +1 -0
@@ -5307,6 +5307,7 @@ need_resched:
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
 
+	schedule_softirq_check();
 	schedule_debug(prev);

hm, this slows down the scheduler fast-path ...

Ingo

#23 lore raw

Re: [PATCH RFC] softirq: fix ksoftirq starved

Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Lai Jiangshan
kernel/sched.c +0 -0
@@ -5307,6 +5307,7 @@ need_resched:
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
 
Quoted 8 lines

kernel/sched.c need_resched: · line 5311

Quoted 2 lines
kernel/sched.c +1 -0
+	schedule_softirq_check();
 	schedule_debug(prev);

hm, this slows down the scheduler fast-path ...

Ingo

It's true. But:

The overheads are:

Overhead-A: the function call statement "schedule_softirq_check();" It can be gotten rid off by a macro or inline function.

Overhead-B: __get_cpu_var() and the test statement.

Overhead-C: do_softirq()
In my patch, we test a variable and then call do_softirq() when the variable is true. do_softirq() can be called from process context or from schedule() or by any other ways, but it must be called and avoids starvation in this condition.
So we need pay this overhead. It is no worse than before.

Is it a critical thing when it slows down the scheduler fast-path because of the "Overhead-B"?

Or I misunderstand something?

Thanks, Lai.

#24 lore raw

Re: [PATCH RFC] softirq: fix ksoftirq starved

Quoted 34 lines · Lai Jiangshan
Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Lai Jiangshan
kernel/sched.c +0 -0
@@ -5307,6 +5307,7 @@ need_resched:
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
 
Quoted 8 lines

kernel/sched.c need_resched: · line 5311

Quoted 2 lines
kernel/sched.c +1 -0
+	schedule_softirq_check();
 	schedule_debug(prev);

hm, this slows down the scheduler fast-path ...

Ingo

It's true. But:

The overheads are:

Overhead-A: the function call statement "schedule_softirq_check();" It can be gotten rid off by a macro or inline function.

Overhead-B: __get_cpu_var() and the test statement.

Overhead-C: do_softirq()
In my patch, we test a variable and then call do_softirq() when the variable is true. do_softirq() can be called from process context or from schedule() or by any other ways, but it must be called and avoids starvation in this condition.
So we need pay this overhead. It is no worse than before.

Quoted 4 lines

Is it a critical thing when it slows down the scheduler fast-path because of the "Overhead-B"?

Or I misunderstand something?

I think its overhead-c, actually calling do_softirq from the schedule() path that a really whacky idea.

Why not make whoemever needs the softirq to happen (eg. network ops) poll this flag, so that regular RT processes don't get penalized by random softirq muck?

It seems to me this approach basically gives softirqs higher prio than RT processes, not something to be done lightly.

#25 lore raw

Re: [PATCH RFC] softirq: fix ksoftirq starved

In reply to Lai Jiangshan
Quoted 34 lines · Lai Jiangshan
Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Lai Jiangshan
kernel/sched.c +0 -0
@@ -5307,6 +5307,7 @@ need_resched:
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
 
Quoted 8 lines

kernel/sched.c need_resched: · line 5311

Quoted 2 lines
kernel/sched.c +1 -0
+	schedule_softirq_check();
 	schedule_debug(prev);

hm, this slows down the scheduler fast-path ...

Ingo

It's true. But:

The overheads are:

Overhead-A: the function call statement "schedule_softirq_check();" It can be gotten rid off by a macro or inline function.

Overhead-B: __get_cpu_var() and the test statement.

Overhead-C: do_softirq()
In my patch, we test a variable and then call do_softirq() when the variable is true. do_softirq() can be called from process context or from schedule() or by any other ways, but it must be called and avoids starvation in this condition.
So we need pay this overhead. It is no worse than before.

Quoted 4 lines

Is it a critical thing when it slows down the scheduler fast-path because of the "Overhead-B"?

Or I misunderstand something?

The thing is that _any_ extra instruction in the scheduler fast-path should be avoided. I dont think it can be claimed that this problem can only be solved that way.

Ingo

#26 lore raw
In reply to Ingo Molnar
Quoted 90 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

Quoted 6 lines

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

Hi, Ingo

Could you give me the .config file that this bug occurs?

Lai.

#27 lore raw
Quoted 95 lines · Lai Jiangshan
Quoted 94 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Quoted 6 lines
Quoted 2 lines

Ingo

Hi, Ingo

Could you give me the .config file that this bug occurs?

I havent reproduced this - it was reported in a recent thread on lkml. I dont have an URI for it - does anyone on the Cc: remember the details?

Ingo

#28 lore raw
Quoted 100 lines · Ingo Molnar
Quoted 98 lines · Lai Jiangshan
Quoted 89 lines · Ingo Molnar
Quoted 38 lines · Zhaolei

* From: "Xiao Guangrong" <[email protected]>

Quoted 37 lines
Quoted 23 lines · Steven Rostedt
Quoted 22 lines · Mathieu Desnoyers
Quoted 22 lines

From: Mathieu Desnoyers <[email protected]>

This patch is modified from Mathieu Desnoyers' patch. The original patch can be found here:
http://marc.info/?l=linux-kernel&m=123791201816245&w=2
This tracepoint can trace the time stamp when softirq action is raised.

Changelog for v1 -> v2:
1: Use TRACE_EVENT instead of DEFINE_TRACE
2: Move the tracepoint from raise_softirq_irqoff() to
__raise_softirq_irqoff()

Changelog for v2 -> v3:
Move the definition of __raise_softifq_irqoff() to .c file when CONFIG_TRACEPOINTS is enabled, to avoid recursive includes

Changelog for v3 -> v4:
1: Come back to v2, and use forward declarations to avoid
recursive includes as Mathieu's suggestion
2: Modifiy the tracepoint name
3: Add comments for this tracepoint

Quoted 9 lines
Quoted 7 lines

This is a step in the right direction, but please see my email to Lai about the fact that this assumes correct and undocumented include dependencies in kernel/trace/events.c. Not explicitely stating the include dependencies is a build error waiting to happen.

Including interrupt.h under a ifdef would allow keeping track of TRACE_EVENT specific build dependencies neatly on a per header basis.

This is all moot, the events.c file no longer exists and as not an issue.

As Steve's says, use ftrace in ftrace.h not in events.c now. So, this mistake does not exist.
Dose this patch has other error? I expect for your views.

Quoted 5 lines
Quoted 1 line

Thanks for your review, is great help to me. ;-)

Hello,

It seems Mathieu has no other comments on this patch now. Ingo, what is your opinion on this patch?

There's a complication: this area of the softirq code needs fixes (unrelated to tracing).

This API:

inline void raise_softirq_irqoff(unsigned int nr)
{
__raise_softirq_irqoff(nr);

/*
* If we're in an interrupt or softirq, we're done
* (this also catches softirq-disabled code). We will
* actually run the softirq once we return from
* the irq or softirq.
*
* Otherwise we wake up ksoftirqd to make sure we
* schedule the softirq soon.
*/
if (!in_interrupt())
wakeup_softirqd();
}

is broken with RT tasks (as recently reported to lkml), as when a real-time task wakes up ksoftirqd (which has lower priority) it wont execute and we starve softirq execution.

The proper solution would be to have a new API:

raise_softirq_check()

and to remove the wakeup_softirqd() hack from raise_softirq_irqoff()
- and put raise_softirq_check() to all places that use raise_softirq*() from process context.

raise_softirq_check() would execute softirq handlers from process context, if there's any pending ones. It has to be called outside of bh critical sections - i.e. often a bit after the raise_softirq() has been done.

__raise_softirq_irqoff() would be made private to kernel/softirq.c, and we'd only have two public APIs to trigger softirqs:
raise_softirq() and raise_softirq_irqoff(). Both just set the pending flag and dont do any wakeup.

Quoted 6 lines

As a side-effect of these fixes, the tracepoints will be sorted out as well - there wont be any need to hack into __raise_softirq_irqoff().

Ingo

Hi, Ingo

Quoted 10 lines
Quoted 1 line

Could you give me the .config file that this bug occurs?

I havent reproduced this - it was reported in a recent thread on lkml. I dont have an URI for it - does anyone on the Cc: remember the details?

Ingo

I haven't found the email you referred, I've searched it from when you pointed out this softirq problem till now. If anyone know it, please let me know.

If my analysis is correct, I will say this problem does NOT exist:
(non-PREEMPT_RT)

1) the (outmost) interrupt will call do_softirq() when it exits. softirq will not be starved when interrupts occur.

2) the clocks will provide tick-interrupt periodic.

3) even when CONFIG_NO_HZ=y, the cpu still provide periodic tick when current task is RT-task(which is non-idle task).

So any softirq will be called in time(one jiffy). Maybe my analysis is incorrect or maybe the RT task disables the irq very long time or maybe there is other bug in kernel which has influence and impacts softirq.

A) I will be still finding the email or the .config and probe this problem. The endeavors to ensure all softirq handled are not useful and it may not solve the problem. Because the tick-interrupt does ensure pending softirq handled.

B) If my thinking is correct, it's the best that we insert tracepoints at raise_softirq() functions, it's very helpful and one of its benefit is help us to probe problems which happen here.

Lai