Skip to contentSkip to conversation

tracing/events: stable tracepoints

RFC PATCH In review Steven Rostedt 7 participants 24 messages Nov 17, 2010 – Nov 23, 2010

Subsystems (3)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover tracing/events: stable tracepoints

[ RFC ONLY - Not for inclusion ]

As discussed at Kernel Summit, there was some issues about what to do with tracepoints.

Basically, anyone, anywhere, any developer, can create a tracepoint and have it appear in /sys/kernel/debug/tracing/events/...

These events automatically appear in both perf and ftrace as events. And any tool can tap into them. That's where the problem rises.

What happens when a tool starts to depend on a tracepoint? Will that tracepoint always be there? Will it ever change?

The problem also extends to the fact that we can't guarantee that tracepoints will stay as is. There are literally hundreds of tracepoints, and they are used by developers to have in field debugging tools. As the kernel changes, so will these tracepoints. A developer can use these to ask a customer that has run into some problem to enable a trace and send the developer back the trace so they can go off and analyze it.

But for tools, this is a different story. They want and depend on a tracepoint to be stable. If it changes under them, then it makes tracepoints completely useless for tools.

This patch series is a start and RFC for the creation of
stable tracepoints. I will now call the current tracepoints raw or in-field-debugging tracepoints or events. What I call stable tracepoints are those that are to answer questions about the OS and not for a developer to debug their code.

What I propose is to create a new format and a new filesystem called eventfs. Like debugfs, when enabled, a directory will be created:

/sys/kernel/events

Which would be the normal place to mount the eventfs filesystem.

The old format for events looked like this:

$ cat /debug/tracing/events/sched/sched_switch/format 
name: sched_switch
ID: 57
format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;
	field:int common_lock_depth;	offset:8;	size:4;	signed:1;
	field:char prev_comm[TASK_COMM_LEN];	offset:12;	size:16;	signed:1;
	field:pid_t prev_pid;	offset:28;	size:4;	signed:1;
	field:int prev_prio;	offset:32;	size:4;	signed:1;
	field:long prev_state;	offset:40;	size:8;	signed:1;
	field:char next_comm[TASK_COMM_LEN];	offset:48;	size:16;	signed:1;
	field:pid_t next_pid;	offset:64;	size:4;	signed:1;
	field:int next_prio;	offset:68;	size:4;	signed:1;

print fmt: "prev_comm=%s prev_pid=%d prev_prio=%d prev_state=%s ==> next_comm=%s next_pid=%d next_prio=%d", REC->prev_comm, REC->prev_pid, REC->prev_prio, REC->prev_state ? __print_flags(REC->prev_state, "|", { 1, "S"} , { 2, "D" }, { 4, "T" }, { 8, "t" }, { 16, "Z" }, { 32, "X" }, { 64, "x" }, { 128, "W" }) : "R", REC->next_comm, REC->next_pid, REC->next_prio

The "common" fields were ftrace (and because perf attached to it, also perf) specific. Also the size is in bytes, which would limit the ability to use bit fields. We also don't know about arch specific alignment that may be needed to write to these fields.

We also have name (redundant), ID (should be agnostic), and print_fmt (lots of issues).

So the new format looks like this:

[root@bxf ~]# cat /sys/kernel/event/sched_switch/format

	array:prev_comm	type:char	size:8	count:16	align:1	signed:1;
	field:prev_pid	type:pid_t	size:32	align:4	signed:1;
	field:prev_state	type:char	size:8	align:1	signed:1;
	array:next_comm	type:char	size:8	count:16	align:1	signed:1;
	field:next_pid	type:pid_t	size:32	align:4	signed:1;

Some notes:

o The size is in bits.
o We added an align, that is the natural alignment for the arch of that type.
o We added an "array" type, that specifies the size of an element as well as a "count", where total size can be align(size) * count.
o We separated the field name from the type.

Not in this series, but for future (after we agree on all this) I would like to move the raw tracepoints into /debug/events/... and have the same format as here.

This patch series uses some of the same tricks as the TRACE_EVENT() code. It has magic macros to do all the redundant code. But it has a bit of manual work.

Right now, when a STABLE_EVENT() is created, the format appears. But nothing hooks into it yet. perf, trace, or ftrace could register a handle that is created, either manually, or it can use the same magic macro tricks to automate all the stable events. The design has been made to allow for that too.

The last two patches create two stable tracepoints. sched_switch and sched_migrate_task (for examples as well as to get the ball rolling). As you may have already noticed, there is currently no hierarchy with the stable events. We want to limit the # of stable events, as they should only be created to help answer general questions about the OS. All events reside at the top layer of the eventfs filesystem. (I do not plan on doing this for the raw events though).

Another note is that all stable events need a corresponding raw event. The raw event does not need to be of the same format as the stable event, it just needs to provide all the information that the stable event needs, but the raw event may supply much more. This should not be a problem, since the tracepoint that represents a stable event should, by definition, always be stable :-)

Because the stable events piggy back on top of the raw events, the trace_...() function in the kernel can be used by both. No changes are needed there. As long as there's already a tracepoint represented by a raw event, a stable event can be placed on top.

The raw event may change at anytime, as long as it always supplies the stable event with what is needed. It will require the hooks between them to be updated. The way tracepoints work, if they become out of sync, the code will fail to compile.

Time to get out the hose!

-- Steve

The following patches are in:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git

branch: rfc/events

Steven Rostedt (5):
events: Add EVENT_FS the event filesystem tracing/events: Add code to (un)register stable events tracing/events: Add infrastructure to show stable event formats tracing/events: Add stable event sched_switch tracing/events: Add sched_migrate_task stable event

----

 fs/Kconfig                   |    6 +
 fs/Makefile                  |    1 +
 fs/eventfs/Makefile          |    4 +
 fs/eventfs/file.c            |   53 +++++
 fs/eventfs/inode.c           |  433 ++++++++++++++++++++++++++++++++++++++++++
 include/linux/eventfs.h      |   83 ++++++++
 include/linux/magic.h        |    3 +-
 include/trace/stable.h       |   72 +++++++
 include/trace/stable/sched.h |   33 ++++
 include/trace/stable_list.h  |    3 +
 kernel/Makefile              |    1 +
 kernel/events/Makefile       |    1 +
 kernel/events/event_format.c |   74 +++++++
 kernel/events/event_format.h |   64 ++++++
 kernel/events/event_reg.h    |   79 ++++++++
 kernel/events/events.c       |   48 +++++
 kernel/trace/Kconfig         |    1 +
 17 files changed, 958 insertions(+), 1 deletions(-)
author maintainer
#1 lore raw

RFC PATCH 1/5 events: Add EVENT_FS the event filesystem

From: Steven Rostedt <[email protected]>

Copied mostly from debugfs, the eventfs is the filesystem that will include stable tracepoints. Currently nothing enables this filesystem as of this patch.

---
 fs/Kconfig              |    6 +
 fs/Makefile             |    1 +
 fs/eventfs/Makefile     |    4 +
 fs/eventfs/file.c       |   53 ++++++
 fs/eventfs/inode.c      |  433 +++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/eventfs.h |   83 +++++++++
 include/linux/magic.h   |    3 +-
 7 files changed, 582 insertions(+), 1 deletions(-)
 create mode 100644 fs/eventfs/Makefile
 create mode 100644 fs/eventfs/file.c
 create mode 100644 fs/eventfs/inode.c
 create mode 100644 include/linux/eventfs.h
fs/Kconfig +6 -0
index 771f457..ffd3a8b 100644
@@ -250,6 +250,12 @@ source "fs/partitions/Kconfig"
 endmenu
 endif
 
+config EVENT_FS
+       bool
+	help
+	  The event filesystem, usually mounted at /sys/kernel/events.
+	  This option is selected by other configs that require it.
+
 source "fs/nls/Kconfig"
 source "fs/dlm/Kconfig"
 
fs/Makefile +1 -0
index a7f7cef..4fe02d4 100644
@@ -116,6 +116,7 @@ obj-$(CONFIG_HOSTFS)		+= hostfs/
 obj-$(CONFIG_HPPFS)		+= hppfs/
 obj-$(CONFIG_CACHEFILES)	+= cachefiles/
 obj-$(CONFIG_DEBUG_FS)		+= debugfs/
+obj-$(CONFIG_EVENT_FS)		+= eventfs/
 obj-$(CONFIG_OCFS2_FS)		+= ocfs2/
 obj-$(CONFIG_BTRFS_FS)		+= btrfs/
 obj-$(CONFIG_GFS2_FS)           += gfs2/
fs/eventfs/Makefile new file +4 -0
new file mode 100644
index 0000000..5e991a5
@@ -0,0 +1,4 @@
+eventfs-objs	:= inode.o file.o
+
+obj-$(CONFIG_EVENT_FS)	+= eventfs.o
+
fs/eventfs/file.c new file +53 -0

53 lines not on this page · raw message

fs/eventfs/inode.c new file +433 -0

433 lines not on this page · raw message

include/linux/eventfs.h new file +83 -0

83 lines not on this page · raw message

include/linux/magic.h +2 -1
index ff690d0..0126169 100644
@@ -8,7 +8,8 @@
 #define CODA_SUPER_MAGIC	0x73757245
 #define CRAMFS_MAGIC		0x28cd3d45	/* some random number */
 #define CRAMFS_MAGIC_WEND	0x453dcd28	/* magic number with the wrong endianess */
-#define DEBUGFS_MAGIC          0x64626720
+#define DEBUGFS_MAGIC		0x64626720
+#define EVENTFS_MAGIC		0x65667310
 #define SYSFS_MAGIC		0x62656572
 #define SECURITYFS_MAGIC	0x73636673
 #define SELINUX_MAGIC		0xf97cff8c
Signature · 1 line
1.7.1
#2 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 5 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

Copied mostly from debugfs, the eventfs is the filesystem that will include stable tracepoints. Currently nothing enables this filesystem as of this patch.

What? Wait, I wrote tracefs a long time ago just for this, why not take that code and use it instead?

{sigh} And I just deleted that old tracefs git tree today as I thought that idea was long gone and dead....

Quoted 9 lines
---
 fs/Kconfig              |    6 +
 fs/Makefile             |    1 +
 fs/eventfs/Makefile     |    4 +
 fs/eventfs/file.c       |   53 ++++++
 fs/eventfs/inode.c      |  433 +++++++++++++++++++++++++++++++++++++++++++++++

This seems a bit big, I don't think you need all of this for some reason. Are you sure you can't make it smaller?

thanks,

greg k-h

#3 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 8 lines · Greg KH
Quoted 4 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

Copied mostly from debugfs, the eventfs is the filesystem that will include stable tracepoints. Currently nothing enables this filesystem as of this patch.

What? Wait, I wrote tracefs a long time ago just for this, why not take that code and use it instead?

Yeah, and i know that i suggested 'eventfs' to Steve and others in a prior thread a few months ago - and i suspect Steve was following up on that suggestion with this patch? So i guess it's partly my fault ;-)

[ Also, i think our _real_ problems with tracing lie entirely elsewhere, but i've
explained that numerous times. Maintaining instrumentation bits is the ultimate cat herding experience ;-) ]

I also explained it in that eventfs suggestion thread that eventfs (or, indeed tracefs) is IMO only a second tier approach compared to the real thing: proper enumeration of events in sysfs.

[ Beyond the obvious compatibility detail that we are _NOT_ getting rid of
/debug/tracing/events/, as existing tooling depends on it. So unless eventfs or sysfs integration brings some real tangible benefits over what we have already we dont want to force tooling to migrate to yet another API. ]

Lin Ming and PeterZ are working on sysfs integration and they have posted several iterations of that work which extends event details to sysfs. That work is not complete yet and they need help. (I've Cc:-ed them.)

The sysfs approach has numerous upsides:

- Design: sysfs is a mature, multi-year project with tons of meaningful hardware and software hieararchies already well established. Attaching events to these existing nodes optionally is an obvious advantage and avoids duplication and forces people to think about structure.

- Concentration of structure: subsystem and driver authors/maintainers already care about their sysfs layout - and when they define new tracepoints for subsystem or driver instrumentation it would be very natural for those events to go somewhere nearby, in the existing sysfs hieararchy.

- Practicalities: sysfs is already mounted on all distros so tooling could rely on it universally. It's the ultimate 'describe system structure' store.

- Long term maintenance: we want to be strict with events, i.e. keep the descriptors read only and single-line structured. You sysfs folks are enforcing that pretty well - with eventfs we'd always have the nasty lure to apply API hacks to eventfs components when we really shouldnt ...

Eventfs has a couple of downsides:

- Design: it's slapping events into a separate, partly duplicated, partly unique, partly inconsistent set of hierarchies. We can deal with it, but it's not particularly intelligent and i'd like us to try harder.

- Practicalities: eventfs has to be mounted on every distro. It's an uphill climb in general and the appeal of an approach has to be _strong_ for this to be feasible.

So putting it into sysfs looks like a pretty intelligent solution all around and i'd prefer it.

Steve, would you be interested in helping out Lin Ming and PeterZ with the sysfs work - or at least help them come to the conclusion that we want eventfs?

Thanks,

Ingo

author maintainer
#4 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 8 lines · Ingo Molnar
Quoted 6 lines · Greg KH
Quoted 4 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

Copied mostly from debugfs, the eventfs is the filesystem that will include stable tracepoints. Currently nothing enables this filesystem as of this patch.

Quoted 6 lines
Quoted 2 lines

What? Wait, I wrote tracefs a long time ago just for this, why not take that code and use it instead?

Yeah, and i know that i suggested 'eventfs' to Steve and others in a prior thread a few months ago - and i suspect Steve was following up on that suggestion with this patch? So i guess it's partly my fault ;-)

And we brought this up at Kernel Summit.

Quoted 5 lines

[ Also, i think our _real_ problems with tracing lie entirely elsewhere, but i've
explained that numerous times. Maintaining instrumentation bits is the ultimate cat herding experience ;-) ]

Quoted 8 lines

I also explained it in that eventfs suggestion thread that eventfs (or, indeed tracefs) is IMO only a second tier approach compared to the real thing: proper enumeration of events in sysfs.

[ Beyond the obvious compatibility detail that we are _NOT_ getting rid of
/debug/tracing/events/, as existing tooling depends on it. So unless eventfs or sysfs integration brings some real tangible benefits over what we have already we dont want to force tooling to migrate to yet another API. ]

One benefit is that we have a way to distinguish between in-field-debugging tracepoints and tracepoints that are only for analysis tools.

Quoted 4 lines

Lin Ming and PeterZ are working on sysfs integration and they have posted several iterations of that work which extends event details to sysfs. That work is not complete yet and they need help. (I've Cc:-ed them.)

Actually, I suck at adding anything to the sysfs/kobject code. I always screw it up. I only got the /sys/kernel/events working because I copied it directly from Greg. I doubt I'd be much help.

Quoted 13 lines

The sysfs approach has numerous upsides:

- Design: sysfs is a mature, multi-year project with tons of meaningful hardware and software hieararchies already well established. Attaching events to these existing nodes optionally is an obvious advantage and avoids duplication and forces people to think about structure.

- Concentration of structure: subsystem and driver authors/maintainers already care about their sysfs layout - and when they define new tracepoints for subsystem or driver instrumentation it would be very natural for those events to go somewhere nearby, in the existing sysfs hieararchy.

Quoted 7 lines

- Practicalities: sysfs is already mounted on all distros so tooling could rely on it universally. It's the ultimate 'describe system structure' store.

- Long term maintenance: we want to be strict with events, i.e. keep the descriptors read only and single-line structured. You sysfs folks are enforcing that pretty well - with eventfs we'd always have the nasty lure to apply API hacks to eventfs components when we really shouldnt ...

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

Also, if we just blindly label a tracepoint as "stable" then we must keep all its contents. For example, the sched_switch will contain the priority. As Peter has stated several times, that may go away. We also do not want to lose getting that information, as a lot of us use it.

Quoted 10 lines

Eventfs has a couple of downsides:

- Design: it's slapping events into a separate, partly duplicated, partly unique, partly inconsistent set of hierarchies. We can deal with it, but it's not particularly intelligent and i'd like us to try harder.

- Practicalities: eventfs has to be mounted on every distro. It's an uphill climb in general and the appeal of an approach has to be _strong_ for this to be feasible.

Some distros already mount debugfs by default. It's a oneliner in fstab.

Quoted 3 lines

So putting it into sysfs looks like a pretty intelligent solution all around and i'd prefer it.

Another downside is that you need to scan hundreds of directories to find tracepoints. And again, are they all now stable?

Quoted 3 lines

Steve, would you be interested in helping out Lin Ming and PeterZ with the sysfs work - or at least help them come to the conclusion that we want eventfs?

I don't think I would be much help with the former, and I'm thinking I'm losing the later.

Hmm, seems that every decision that we came to agreement with at Kernel Summit has been declined in practice. Makes me think that Kernel Summit is pointless, and was a waste of my time. :-(

-- Steve

Quoted 4 lines

Thanks,

Ingo

#5 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 3 lines · Steven Rostedt

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

I think you are worrying about the wrong things.

I think Arjan's complaints at the KS stemmed from prior sporadic declarations on lkml that there is no tracepoint ABI _at all_, and that powertop/latencytop could break anytime.

But in reality i strongly disagree with such declarations, and tracepoint data that is used by PowerTop/timechart/latencytop or perf is and was an ABI, simple as that - and i've been enforcing that for two years. (We have so few good instrumentation tools that we _really_ dont want to break them.)

At that point, realizing that we have an ABI for existing tools, i think it's fundamentally misguided to go out on a limb trying to put barriers in the way of other tools that do not even exist to begin with ...

Our real problem with tracing is lack of relevance, lack of utility, lack of punch-through analytical power.

Trying to create a sandbox to _reduce utility_ is like the last step, and a really optional step, when we have such variety that we want some control over it. It's always expensive, it always reduces the tool space as collateral damage.

So please dont think of sysfs or eventfs as a tool to restrict. Think of it as a tool to _organize_.

Again, i'd _LOVE_ to have the 'problem' of us having so many tools that analyze application and kernel behavior in such a rich way that they use tracepoints that were not supposed to be 'stable'.

I simply dont see the 'problem' that is being solved here. We had a stable ABI and we didnt break sysprof or powertop/latencytop in the past and wont break it in the future either.

Thanks,

Ingo

maintainer
#6 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 3 lines · Ingo Molnar

I think Arjan's complaints at the KS stemmed from prior sporadic declarations on lkml that there is no tracepoint ABI _at all_, and that powertop/latencytop could break anytime.

And it will, afaik Arjan refused to even parse the format file which is part of the tracepoint abi and I'll be changing those for the scheduler.

I really object to not being able to make sane changes just because some tool is too lazy to even implement the full ABI that was exposed.

Quoted 3 lines

I think Arjan's complaints at the KS stemmed from prior sporadic declarations on lkml that there is no tracepoint ABI _at all_, and that powertop/latencytop could break anytime.

I fully intent to break powertop/latencytop if they refuse to use the format file, deal with it.

Also, in the unlikely event we need to re-order the task->state bits I'll do so without a moments hesitation, regardless of who consumes them through the scheduler tracepoints, that's simply not stuff that should be tied down.

The same for anything that tries to interpret task->prio through the tracepoints.

author maintainer
#7 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Ingo Molnar
Quoted 31 lines · Ingo Molnar
Quoted 3 lines · Steven Rostedt

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

I think you are worrying about the wrong things.

I think Arjan's complaints at the KS stemmed from prior sporadic declarations on lkml that there is no tracepoint ABI _at all_, and that powertop/latencytop could break anytime.

But in reality i strongly disagree with such declarations, and tracepoint data that is used by PowerTop/timechart/latencytop or perf is and was an ABI, simple as that - and i've been enforcing that for two years. (We have so few good instrumentation tools that we _really_ dont want to break them.)

At that point, realizing that we have an ABI for existing tools, i think it's fundamentally misguided to go out on a limb trying to put barriers in the way of other tools that do not even exist to begin with ...

Our real problem with tracing is lack of relevance, lack of utility, lack of punch-through analytical power.

Trying to create a sandbox to _reduce utility_ is like the last step, and a really optional step, when we have such variety that we want some control over it. It's always expensive, it always reduces the tool space as collateral damage.

So please dont think of sysfs or eventfs as a tool to restrict. Think of it as a tool to _organize_.

Quoted 7 lines

Again, i'd _LOVE_ to have the 'problem' of us having so many tools that analyze application and kernel behavior in such a rich way that they use tracepoints that were not supposed to be 'stable'.

I simply dont see the 'problem' that is being solved here. We had a stable ABI and we didnt break sysprof or powertop/latencytop in the past and wont break it in the future either.

What about a tool that picks up tracepoints that were only used by a developer for in-field debugging, and then that tracepoint disappears because of a design change. Is it OK for that tool to break with it?

Do all tools that use tracepoints require a "check" feature?

I guess the problem is that creators of the tools to analyze the kernel have no idea of what they can count on and what they can't. Do we need a process to have these tool creators request to developers to "keep this tracepoint"?

-- Steve

maintainer
#8 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 5 lines · Steven Rostedt

What about a tool that picks up tracepoints that were only used by a developer for in-field debugging, and then that tracepoint disappears because of a design change. Is it OK for that tool to break with it?

Do all tools that use tracepoints require a "check" feature?

Not sure what you mean with a 'check' feature, but I do think its useful to tools authors to clearly delineate between stable and debug tracepoints, that also facilitates

Quoted 4 lines

I guess the problem is that creators of the tools to analyze the kernel have no idea of what they can count on and what they can't. Do we need a process to have these tool creators request to developers to "keep this tracepoint"?

the process mentioned here, if they cannot find the data they want through existing stable tracepoints they have to request it, or otherwise clearly suffer breakage whenever we feel like changing stuff.

A nice aspect of devs coming to us and requesting data is that we get a fairly good idea of what tools are available and get to discuss the problem they're trying to solve.

Esp. that latter point is a very good one imho, we might have a totally different view on some of the problems :-)

#9 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar

I think Arjan's complaints at the KS stemmed from prior sporadic declarations on lkml that there is no tracepoint ABI _at all_, and that powertop/latencytop could break anytime.

But in reality i strongly disagree with such declarations, and tracepoint data that is used by PowerTop/timechart/latencytop or perf is and was an ABI, simple as that - and i've been enforcing that for two years. (We have so few good instrumentation tools that we _really_ dont want to break them.)

There was general consensus at the kernel summit that the tracepoints, being deliberately in /sys/kernel/debug was unstable and today exposes internal implementation details --- and that that people putting different traceponts in Documentation/ABI/{testing,stable,unstable} simply doesn't work.

Heck, I've recently had to make changes to tracepoints because otherwise perf would fall over dead when it tripped over an ext4 tracepoint, due to its limitations dealing with what we could put into TP_PRINTK(). Saying that tracepoints are stable ABI that must never change, forever and ever, amen, is simply a non-starter.

Quoted 3 lines

I simply dont see the 'problem' that is being solved here. We had a stable ABI and we didnt break sysprof or powertop/latencytop in the past and wont break it in the future either.

I think the general consensus is that we didn't have a stable ABI, and the tension between what kernel developers need for debugging, which of necessity is kernel version specific, and what gets exposed as a stable ABI that tools can depend upon, is not one that can be resolved.

Regards,

- Ted

maintainer
#10 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 4 lines · Ingo Molnar

I also explained it in that eventfs suggestion thread that eventfs (or, indeed tracefs) is IMO only a second tier approach compared to the real thing: proper enumeration of events in sysfs.

Quoted 8 lines
Quoted 4 lines

[ Beyond the obvious compatibility detail that we are _NOT_ getting rid of
/debug/tracing/events/, as existing tooling depends on it. So unless eventfs or sysfs integration brings some real tangible benefits over what we have already we dont want to force tooling to migrate to yet another API. ]

One benefit is that we have a way to distinguish between in-field-debugging tracepoints and tracepoints that are only for analysis tools.

Right, and there was a clear consensus that that was a desired thing, have a strong signal towards tool authors to request 'stable' tracepoints or live with the fact they have to cope with the thing changing.

As to the whole /debug/tracing/events thing, we can deprecate it like we have done so many interfaces, provided we have indeed provided an alternative interface.

Quoted 7 lines
Quoted 3 lines

Lin Ming and PeterZ are working on sysfs integration and they have posted several iterations of that work which extends event details to sysfs. That work is not complete yet and they need help. (I've Cc:-ed them.)

Actually, I suck at adding anything to the sysfs/kobject code. I always screw it up. I only got the /sys/kernel/events working because I copied it directly from Greg. I doubt I'd be much help.

Yeah, I suck at it too, sysfs is scary. Luckily Kay is helping me out.

Quoted 1 line

Are these events now going to be labeled as stable?

For now I'm concentrating on the hardware events, those are more or less stable in that if you run it on the same hardware you get the same thing counted. Different hardware may miss some events since it simply cannot provide anything resembling, or count something related but slightly different (speculative vs retired events, or different cache level, or a slightly different definition of miss etc..)

Basically the same we already have with the perf 'generic' events.

Quoted 8 lines

Is every tracepoint
we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

Also, if we just blindly label a tracepoint as "stable" then we must keep all its contents. For example, the sched_switch will contain the priority. As Peter has stated several times, that may go away. We also do not want to lose getting that information, as a lot of us use it.

For now I've not at all looked at representing tracepoints in sysfs, for the hardware bits I'm looking to place the event_source (formerly known as PMU) in the hardware topology already present in /sys/devices/.

One suggestion was to place the software and tracepoint events in /sys/kernel/ some place. Another was to place driver specific, say wifi things near the wifi driver node.

None of the proposals have dealt with the stable vs debug thing, simply because none of them were post KS.

Quoted 1 line

Some distros already mount debugfs by default. It's a oneliner in fstab.

I haven't yet seen that.. maybe its automounted on /sys/kernel/debug/ or some daft place like that by magic initscipts outside of fstab but I'd not notice that.

Quoted 6 lines
Quoted 3 lines

So putting it into sysfs looks like a pretty intelligent solution all around and i'd prefer it.

Another downside is that you need to scan hundreds of directories to find tracepoints. And again, are they all now stable?

Not really, they'd be accessible through the bus structure, something like:

/sys/bus/event_source/*/events/*

Sure, that's more than 1 directory, but then so
is /debug/tracing/events/*/*/

Quoted 5 lines
Quoted 2 lines

Steve, would you be interested in helping out Lin Ming and PeterZ with the sysfs work - or at least help them come to the conclusion that we want eventfs?

I don't think I would be much help with the former, and I'm thinking I'm losing the later.

Yeah, eventfs simply won't work for what we want to do with hardware events.

Quoted 3 lines

Hmm, seems that every decision that we came to agreement with at Kernel Summit has been declined in practice. Makes me think that Kernel Summit is pointless, and was a waste of my time. :-(

Well, I don't know, clearly Ingo seems to disagree, but then he wasn't at KS. Thomas and me were, and neither of us really see a problem with the stable vs debug things (at least, I didn't hear tglx protest).

author maintainer
#11 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Not sure if the stable vs debug tracepoints debate is dead.

Quoted 21 lines · Peter Zijlstra
Quoted 2 lines

Are these events now going to be labeled as stable?

For now I'm concentrating on the hardware events, those are more or less stable in that if you run it on the same hardware you get the same thing counted. Different hardware may miss some events since it simply cannot provide anything resembling, or count something related but slightly different (speculative vs retired events, or different cache level, or a slightly different definition of miss etc..)

Basically the same we already have with the perf 'generic' events.

Quoted 8 lines

Is every tracepoint
we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

Also, if we just blindly label a tracepoint as "stable" then we must keep all its contents. For example, the sched_switch will contain the priority. As Peter has stated several times, that may go away. We also do not want to lose getting that information, as a lot of us use it.

Quoted 7 lines

For now I've not at all looked at representing tracepoints in sysfs, for the hardware bits I'm looking to place the event_source (formerly known as PMU) in the hardware topology already present in /sys/devices/.

One suggestion was to place the software and tracepoint events in /sys/kernel/ some place. Another was to place driver specific, say wifi things near the wifi driver node.

Should we create a /sys/kernel/tracepoints/

directory for the stable tracepoints. And then have these directories have the format of those tracepoints? Or is that against the "sysfs" rule of only having a single line per file? The format is multi lines.

Quoted 10 lines

None of the proposals have dealt with the stable vs debug thing, simply because none of them were post KS.

Quoted 1 line

Some distros already mount debugfs by default. It's a oneliner in fstab.

I haven't yet seen that.. maybe its automounted on /sys/kernel/debug/ or some daft place like that by magic initscipts outside of fstab but I'd not notice that.

I'd still like to keep the general tracepoints in something like /sys/kernel/debug/events/... using the same format that we come up with for stable tracepoints. The fact that you need to mount the debugfs system to use it, should help keep some tools from using it.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So putting it into sysfs looks like a pretty intelligent solution all around and i'd prefer it.

Quoted 7 lines
Quoted 2 lines

Another downside is that you need to scan hundreds of directories to find tracepoints. And again, are they all now stable?

Not really, they'd be accessible through the bus structure, something like:

/sys/bus/event_source/*/events/*

That's for every event? Or just software ones?

Quoted 5 lines

Sure, that's more than 1 directory, but then so
is /debug/tracing/events/*/*/

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Steve, would you be interested in helping out Lin Ming and PeterZ with the sysfs work - or at least help them come to the conclusion that we want eventfs?

I don't think I would be much help with the former, and I'm thinking I'm losing the later.

Yeah, eventfs simply won't work for what we want to do with hardware events.

Well, hardware events are something that only depends on what you have for hardware. I wasn't thinking of putting them in with the software events. They probably should be separate.

Can the hardware events be traced? That is, not just profiled, but have them traced for when they occur individually.

I'll go work on other things until we can come up with an agreement. I hate to keep wasting days of work just to have someone NAK it again.

-- Steve

#12 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Steven Rostedt

* Steven Rostedt ([email protected]) wrote:
[...]

Quoted 3 lines

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

I'd like to bring up the point of KVM tracepoints here. KVM can be configured as a module, and may clearly contain tracepoints that we'd like to be stable.

My thought is that what we really want to enforce is "no stable tracepoints in drivers" rather than in "modules", but I might be wrong.

Thoughts ?

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
author maintainer
#13 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 6 lines · Mathieu Desnoyers

* Steven Rostedt ([email protected]) wrote:
[...]

Quoted 3 lines

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

Quoted 7 lines

I'd like to bring up the point of KVM tracepoints here. KVM can be configured as a module, and may clearly contain tracepoints that we'd like to be stable.

My thought is that what we really want to enforce is "no stable tracepoints in drivers" rather than in "modules", but I might be wrong.

Thoughts ?

I still say no to stable tracepoints in modules. Once you open that door, everyone will have it.

But, that doesn't mean that a raw traepoint can't be stable. If the maintainer of that tracepoint states it is stable, then by all means, let tools use it.

-- Steve

#14 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 13 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 12 lines · Mathieu Desnoyers

* Steven Rostedt ([email protected]) wrote:
[...]

Quoted 3 lines

Are these events now going to be labeled as stable? Is every tracepoint we have, much have the same data? Linus specifically said at Kernel Summit that he wants absolutely NO modules to have a stable tracepoint.

I'd like to bring up the point of KVM tracepoints here. KVM can be configured as a module, and may clearly contain tracepoints that we'd like to be stable.

My thought is that what we really want to enforce is "no stable tracepoints in drivers" rather than in "modules", but I might be wrong.

Quoted 8 lines
Quoted 1 line

Thoughts ?

I still say no to stable tracepoints in modules. Once you open that door, everyone will have it.

But, that doesn't mean that a raw traepoint can't be stable. If the maintainer of that tracepoint states it is stable, then by all means, let tools use it.

I'd really like to hear Avi's thoughts on this.

Thanks,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#15 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 10 lines · Mathieu Desnoyers
Quoted 7 lines · * Steven Rostedt ([email protected]) wrote:

I still say no to stable tracepoints in modules. Once you open that door, everyone will have it.

But, that doesn't mean that a raw traepoint can't be stable. If the maintainer of that tracepoint states it is stable, then by all means, let tools use it.

I'd really like to hear Avi's thoughts on this.

Steven's scheme is fine with me. (kvm tracepoints are more or less stable, but all tools so far read the tracepoint definitions dynamically, and can cope with tracepoints being added/changed/removed).

Signature · 1 line
error compiling committee.c: too many arguments to function
#16 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Steven Rostedt
Quoted 3 lines · Steven Rostedt

I still say no to stable tracepoints in modules. Once you open that door, everyone will have it.

What about having KVM define the stable tracepoints in the built-in part of the kernel, if CONFIG_KVM is Y or M, and then export the tracepoints, such that the tracepoints can be called from a module?

That way the tracepoints aren't being *defined* in a module, they are just being *called* from the module. Does that seem like a reasonable compromise?

- Ted

#17 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

Quoted 5 lines · * Ted Ts'o ([email protected]) wrote:
Quoted 3 lines · Steven Rostedt

I still say no to stable tracepoints in modules. Once you open that door, everyone will have it.

Quoted 7 lines

What about having KVM define the stable tracepoints in the built-in part of the kernel, if CONFIG_KVM is Y or M, and then export the tracepoints, such that the tracepoints can be called from a module?

That way the tracepoints aren't being *defined* in a module, they are just being *called* from the module. Does that seem like a reasonable compromise?

As Avi replied separately, he does not seem to think stable ABI is needed for KVM, given that his tools can deal with tracepoint addition/removal pretty well. However, let's keep this idea in mind if we ever face this issue with other "core" modules.

Thanks Ted,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
author maintainer
#18 lore raw

Re: [RFC][PATCH 1/5] [PATCH 1/5] events: Add EVENT_FS the event filesystem

In reply to Greg KH
Quoted 9 lines · Greg KH
Quoted 5 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

Copied mostly from debugfs, the eventfs is the filesystem that will include stable tracepoints. Currently nothing enables this filesystem as of this patch.

What? Wait, I wrote tracefs a long time ago just for this, why not take that code and use it instead?

Because:

1) I couldn't find it (I thought you called it eventfs, so I was searching for the wrong thing).

2) You never answered your ping on IRC

3) I was in a rush to get something out.

;-)

I'm perfectly fine in swapping this out. This is only an RFC anyway. This is also why I Cc'd you on this patch.

Quoted 3 lines

{sigh} And I just deleted that old tracefs git tree today as I thought that idea was long gone and dead....

{sigh} and it may still be dead :-(

-- Steve

Quoted 17 lines
Quoted 10 lines
---
 fs/Kconfig              |    6 +
 fs/Makefile             |    1 +
 fs/eventfs/Makefile     |    4 +
 fs/eventfs/file.c       |   53 ++++++
 fs/eventfs/inode.c      |  433 +++++++++++++++++++++++++++++++++++++++++++++++

This seems a bit big, I don't think you need all of this for some reason. Are you sure you can't make it smaller?

thanks,

greg k-h

author maintainer
#19 lore raw

RFC PATCH 2/5 tracing/events: Add code to (un)register stable events

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Add the framework to create and register stable events. To create a stable event, add a file into:

include/trace/stable/myfile.h

With the following format:

STABLE_EVENT(myevent,

	EVENT_STRUCT(
		__field(type, item)
		__array(type, item, count)
	)

)

This will create the function prototype:

typedef void (*trace_proto_myevent)(type item, type *item);

And the functions:

int register_stable_trace_myevent(trace_proto_myevent func, void *data); void unregister_stable_trace_myevent(trace_proto_myevent func, void *data);

This will allow developers to register a callback of a stable event.

The stable events must have a matching raw (in field debugging) trace event. Although, only the names must match, the information in the stable event is just a subset of the information that can be extracted from the raw event.

---
 include/trace/stable.h      |   72 +++++++++++++++++++++++++++++++++++++++
 include/trace/stable_list.h |    2 +
 kernel/Makefile             |    1 +
 kernel/events/Makefile      |    1 +
 kernel/events/event_reg.h   |   79 +++++++++++++++++++++++++++++++++++++++++++
 kernel/events/events.c      |   16 +++++++++
 kernel/trace/Kconfig        |    1 +
 7 files changed, 172 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/stable.h
 create mode 100644 include/trace/stable_list.h
 create mode 100644 kernel/events/Makefile
 create mode 100644 kernel/events/event_reg.h
 create mode 100644 kernel/events/events.c
include/trace/stable.h new file +72 -0
new file mode 100644
index 0000000..ee10f41
@@ -0,0 +1,72 @@
+#ifndef _TRACE_STABLE_H
+#define _TRACE_STABLE_H
+/*
+ * stable.h - create structure and format for stable events
+ *
+ * Copyright (C) 2010 Steven Rostedt <[email protected]>, Red Hat Inc
+ *
+ * This file creates the registering function prototypes
+ * to hook to the stable events and the function prototypes of those
+ * hooks.
+ *
+ * The stable events themselves reside in the directory:
+ *   include/trace/stable/
+ *
+ * Each of these files must be added to the header:
+ *   include/trace/stable_list.
+ *
+ * Code that registers and unregisters stable events only needs
+ * to include this file:
+ *   #include <trace/stable.h>
+ *
+ * and all the stable structures and registering functions will
+ * also be included for all stable events.
+ *
+ * The current elements that can be used by stable events are:
+ *
+ *    __field(type, item)
+ *    __array(type, time, size)
+ *
+ *  Example:
+ *   STABLE_EVENT(myevent,
+ * 	EVENT_STRUCT(
+ *      __field(	pid_t,		pid			)
+ *      __array(	char,		comm,	TASK_COMM_LEN	)
+ *	)
+ *   )
+ *
+ * The above would create the following:
+ *
+ *  typedef void (*trace_proto_myevent)(pid_t pid, char comm[TASK_COMM_LEN]);
+ *  int register_stable_trace_myevent(trace_proto_myevent func, void *data);
+ *  void unregister_stable_trace_myevent(trace_proto_myevent func, void *data);
+ *
+ * These functions will be created in kernel/events/events.c
+ */
+
+#define STABLE_HEADER_MULTI_READ
+
+#undef __SEP__
+#define __SEP__ ,
+
+#undef __field
+#define __field(type, item)				type item
+
+#undef __array
+#define __array(type, item, size)			type *item
+
+#undef EVENT_STRUCT
+#define EVENT_STRUCT(s)	s
+
+#undef STABLE_EVENT
+#define STABLE_EVENT(name, estruct)					\
+	typedef void (*trace_proto_##name)(void *data, estruct);	\
+	int register_stable_trace_##name(trace_proto_##name func, void *data); \
+	void unregister_stable_trace_##name(trace_proto_##name func, void *data)
+
+/* New stable defines must be added here */
+#include <trace/stable_list.h>
+
+#undef STABLE_HEADER_MULTI_READ
+
+#endif /* _TRACE_STABLE_H */
include/trace/stable_list.h new file +2 -0
new file mode 100644
index 0000000..996932a
@@ -0,0 +1,2 @@
+
+/* New stable defines must be added here */
kernel/Makefile +1 -0
index 0b5ff08..8c1cbbd 100644
@@ -106,6 +106,7 @@ obj-$(CONFIG_PERF_EVENTS) += perf_event.o
 obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
 obj-$(CONFIG_USER_RETURN_NOTIFIER) += user-return-notifier.o
 obj-$(CONFIG_PADATA) += padata.o
+obj-y += events/
 
 ifneq ($(CONFIG_SCHED_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is
kernel/events/Makefile new file +1 -0
new file mode 100644
index 0000000..86fd7c1
@@ -0,0 +1 @@
+obj-y += events.o
kernel/events/event_reg.h new file +79 -0

79 lines not on this page · raw message

kernel/events/events.c new file +16 -0
new file mode 100644
index 0000000..6868bf1
@@ -0,0 +1,16 @@
+/*
+ * events.c - code to register stable events
+ *
+ * Copyright (C) 2010 Steven Rostedt <[email protected]> Red Hat Inc
+ *
+ * This file holds the hooks that are required to convert
+ * a raw tracepoing into a stable one. The conversion names
+ * must be named: hook_##tracepoint_name.
+ */
+#include <linux/tracepoint.h>
+#include <linux/module.h>
+#include <trace/stable.h>
+
+static struct mutex stable_event_mutex;
+
+#include "event_reg.h"
kernel/trace/Kconfig +1 -0
index e04b8bc..0533182 100644
@@ -94,6 +94,7 @@ config TRACING
 	select NOP_TRACER
 	select BINARY_PRINTF
 	select EVENT_TRACING
+	select EVENT_FS
 
 config GENERIC_TRACER
 	bool
Signature · 1 line
1.7.1
author maintainer
#20 lore raw

RFC PATCH 3/5 tracing/events: Add infrastructure to show stable event formats

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Add the infrastructure to connect to the eventfs filesystem and create a event directory. Currently, all events are flat. That is there is no hierarchy. Each stable event has its own directory in /sys/kernel/events (after the eventfs is mounted there).

Currently only one file exists in each of the event directories. This file is the format file. The format shows each item in the stable event. The name of that item, the item's type, the size of the item (in bits), the count of entries of the item (if that item is an array), the alignment needed for that item to be read (arch specific) and the sign of the item.

---
 kernel/events/Makefile       |    2 +-
 kernel/events/event_format.c |   74 ++++++++++++++++++++++++++++++++++++++++++
 kernel/events/event_format.h |   64 ++++++++++++++++++++++++++++++++++++
 3 files changed, 139 insertions(+), 1 deletions(-)
 create mode 100644 kernel/events/event_format.c
 create mode 100644 kernel/events/event_format.h
kernel/events/Makefile +1 -1
index 86fd7c1..943bda5 100644
@@ -1 +1 @@
-obj-y += events.o
+obj-y += events.o event_format.o
kernel/events/event_format.c new file +74 -0

74 lines not on this page · raw message

kernel/events/event_format.h new file +64 -0

64 lines not on this page · raw message

Signature · 1 line
1.7.1
author maintainer
#21 lore raw

RFC PATCH 4/5 tracing/events: Add stable event sched_switch

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Add the stable event for sched_switch.

[root@bxf ~]# cat /sys/kernel/event/sched_switch/format

	array:prev_comm	type:char	size:8	count:16	align:1	signed:1;
	field:prev_pid	type:pid_t	size:32	align:4	signed:1;
	field:prev_state	type:char	size:8	align:1	signed:1;
	array:next_comm	type:char	size:8	count:16	align:1	signed:1;
	field:next_pid	type:pid_t	size:32	align:4	signed:1;
---
 include/trace/stable/sched.h |   20 ++++++++++++++++++++
 include/trace/stable_list.h  |    1 +
 kernel/events/events.c       |   22 ++++++++++++++++++++++
 3 files changed, 43 insertions(+), 0 deletions(-)
 create mode 100644 include/trace/stable/sched.h
include/trace/stable/sched.h new file +20 -0
new file mode 100644
index 0000000..b5f4fd7
@@ -0,0 +1,20 @@
+#if !defined(_STABLE_SCHED_H) || defined(STABLE_HEADER_MULTI_READ)
+#define _STABLE_SCHED_H
+
+#include <linux/sched.h>
+
+/*
+ * Tracepoint for task switches, performed by the scheduler:
+ */
+STABLE_EVENT(sched_switch,
+
+	EVENT_STRUCT(
+		__array(	char,	prev_comm,	TASK_COMM_LEN	)__SEP__
+		__field(	pid_t,	prev_pid			)__SEP__
+		__field(	char,	prev_state			)__SEP__
+		__array(	char,	next_comm,	TASK_COMM_LEN	)__SEP__
+		__field(	pid_t,	next_pid			)
+	)
+);
+
+#endif /* _STABLE_SCHED_H */
include/trace/stable_list.h +1 -0
index 996932a..9cbc006 100644
@@ -1,2 +1,3 @@
 
 /* New stable defines must be added here */
+#include <trace/stable/sched.h>
kernel/events/events.c +22 -0
index 6868bf1..f69e720 100644
@@ -13,4 +13,26 @@
 
 static struct mutex stable_event_mutex;
 
+#include <trace/events/sched.h>
+
+DECLARE_TRACE(stable_sched_switch,
+	      TP_PROTO(char *prev_comm, pid_t prev_pid, char prev_state,
+		       char *next_comm, pid_t next_pid),
+	      TP_ARGS(prev_comm, prev_pid, prev_state, next_comm, next_pid));
+DEFINE_TRACE(stable_sched_switch);
+
+static const char stat_nam[] = TASK_STATE_TO_CHAR_STR;
+static void hook_sched_switch(void *ignore,
+			      struct task_struct *prev,
+			      struct task_struct *next)
+{
+	unsigned state;
+
+	state = prev->state ? __ffs(prev->state) + 1 : 0;
+	state = state < sizeof(stat_nam) - 1 ? stat_nam[state] : '?';
+
+	trace_stable_sched_switch(prev->comm, prev->pid, state,
+				  next->comm, next->pid);
+}
+
 #include "event_reg.h"
Signature · 1 line
1.7.1
author maintainer
#22 lore raw

RFC PATCH 5/5 tracing/events: Add sched_migrate_task stable event

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

Add the stable event for sched_migrate_task.

[root@bxf ~]# cat /sys/kernel/event/sched_migrate_task/format

	array:comm	type:char	size:8	count:16	align:1	signed:1;
	field:pid	type:pid_t	size:32	align:4	signed:1;
	field:orig_cpu	type:int	size:32	align:4	signed:1;
	field:dest_cpu	type:int	size:32	align:4	signed:1;
---
 include/trace/stable/sched.h |   13 +++++++++++++
 kernel/events/events.c       |   10 ++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)
include/trace/stable/sched.h +13 -0
index b5f4fd7..55d0e6f 100644
@@ -17,4 +17,17 @@ STABLE_EVENT(sched_switch,
 	)
 );
 
+/*
+ * Tracepoint for a task being migrated:
+ */
+STABLE_EVENT(sched_migrate_task,
+
+	EVENT_STRUCT(
+		__array(	char,	comm,	TASK_COMM_LEN	)__SEP__
+		__field(	pid_t,	pid			)__SEP__
+		__field(	int,	orig_cpu		)__SEP__
+		__field(	int,	dest_cpu		)
+	)
+);
+
 #endif /* _STABLE_SCHED_H */
kernel/events/events.c +10 -0
index f69e720..6ca2d2d 100644
@@ -35,4 +35,14 @@ static void hook_sched_switch(void *ignore,
 				  next->comm, next->pid);
 }
 
+DECLARE_TRACE(stable_sched_migrate_task,
+	      TP_PROTO(char *comm, pid_t pid, int orig_cpu, int dest_cpu),
+	      TP_ARGS(comm, pid, orig_cpu, dest_cpu));
+DEFINE_TRACE(stable_sched_migrate_task);
+
+static void hook_sched_migrate_task(void *ignore, struct task_struct *p, int dest_cpu)
+{
+	trace_stable_sched_migrate_task(p->comm, p->pid, task_cpu(p), dest_cpu);
+}
+
 #include "event_reg.h"
Signature · 1 line
1.7.1
#23 lore raw
In reply to Steven Rostedt
Quoted 11 lines · * Steven Rostedt ([email protected]) wrote:

We also have name (redundant), ID (should be agnostic), and print_fmt (lots of issues).

So the new format looks like this:

[root@bxf ~]# cat /sys/kernel/event/sched_switch/format

	array:prev_comm	type:char	size:8	count:16	align:1	signed:1;
	field:prev_pid	type:pid_t	size:32	align:4	signed:1;
	field:prev_state	type:char	size:8	align:1	signed:1;
	array:next_comm	type:char	size:8	count:16	align:1	signed:1;
	field:next_pid	type:pid_t	size:32	align:4	signed:1;

Hrm, this is mixing field and type definitions. How about we organize this in something that will be both parseable and extensible ?

First, I don't see what exporting the kernel-internal type "pid_t" in there gives you. Userspace knows nothing about this, so it seems pretty useless.

What do you think of this alternative layout ?

Named types below:

% cat /sys/kernel/event/types/char
parent = integer;
size = 8;
signed = true;
align = 8;

% cat /sys/kernel/event/types/pid_t
parent = integer;
size = 32;
signed = true;
align = 32; /* Or 8 if the architecture supports unaligned writes
efficiently */

% cat /sys/kernel/event/sched_switch/format
type { /* Nameless type */

        parent = struct;
        fields = {
                {
                        type {  /* Nameless type */
                                parent = array;
                                length = 16;
                                elem_type = char; /* refers to named type */
                        },
                        prev_comm,
                },
                { pid_t, prev_pid, },
                { char, prev_state, },
                {
                        type {  /* Nameless type */
                                parent = array;
                                length = 16;
                                elem_type = char; /* refers to named type */
                        },
                        next_comm,
                },
                { pid_t, next_pid, },
        };
}

With this layout, we can declare types like enumerations, e.g.

% cat /sys/kernel/event/types/trap_id_t

type {
        parent = enum;
        size = 5;       /* 5-bit bitfield to hold the enumeration */
        signed = false;
        align = 1;      /* bit-packed */
        map = {
                { 0, "divide error" },
                { 2, "nmi stack" },
                { 4, "overflow" },
                ....
        };
}

So we can refer to this "named type" in all events for which we want to save trap ID ? We therefore get the mapping to a human-understandable name for free.

Quoted 3 lines

Some notes:

o The size is in bits.

Yep, this will immensely help when dealing with bitfields.

Quoted 2 lines

o We added an align, that is the natural alignment for the arch of that type.

Just watch out, in your initial example, I think your align field is in bytes rather than bits. Ideally we'd like everything to be consistent.

Thanks,

Mathieu

Quoted 4 lines

o We added an "array" type, that specifies the size of an element as well as a "count", where total size can be align(size) * count.
o We separated the field name from the type.

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com