Skip to contentSkip to conversation

Pulling away from the tracing ABI quicksands

Ksummit-discuss Mathieu Desnoyers 7 participants 32 messages Jun 29, 2017 – Jul 28, 2017

lore
Share & tools
lore
#0 lore raw

Our attempts at extending the Linux scheduler Tracepoints [1,2,3] to cover sched rt and deadline policies went fairly well until we reached the point where we had to extend the tracing ABI exposed to user-space by Ftrace through the DebugFS tracing sub-directory. We aim to expose more accurate and complete scheduler information (based on the scheduling class), and eventually deprecate implementation-specific fields that should never have been exposed to user-space in the first place.

Example of problems we are facing:

* Humans on the receiving end of a kernel ABI

Should we limit the design of kernel ABIs based on their direct use by humans through echo and redirections, or can we aim design of those interfaces at scripts and user-space programs/libraries ?

A simple and extensible kernel ABI is rarely an easy to use end-user interface.

* How can we deprecate, remove, or re-purpose a field in an event ? For instance, the "prio" field in the scheduler instrumentation is an internal implementation detail.

Perhaps it would be good to use the opportunity to meet at the Kernel Summit and discuss this issue.

Thanks,

Mathieu and Julien

[1] [RFC PATCH 5/6] tracing: extend scheduling tracepoints

    http://lkml.kernel.org/r/[email protected]
[2] [RFC PATCH v2 0/5] Additional scheduling information in tracepoints
    http://lkml.kernel.org/r/[email protected]
[3] [RFC PATCH v3 0/2] Extend scheduling tracepoints
    http://lkml.kernel.org/r/[email protected]
Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#1 lore raw
Quoted 8 lines · On Thu, 29 Jun 2017 21:20:54 +0000 (UTC) Mathieu Desnoyers…

Our attempts at extending the Linux scheduler Tracepoints [1,2,3] to cover sched rt and deadline policies went fairly well until we reached the point where we had to extend the tracing ABI exposed to user-space by Ftrace through the DebugFS tracing sub-directory. We aim to expose more accurate and complete scheduler information (based on the scheduling class), and eventually deprecate implementation-specific fields that should never have been exposed to user-space in the first place.

Quoted 7 lines

Example of problems we are facing:

* Humans on the receiving end of a kernel ABI

Should we limit the design of kernel ABIs based on their direct use by humans through echo and redirections, or can we aim design of those interfaces at scripts and user-space programs/libraries ?

Well, that human is mainly Peter ;-)

But we also can't break tools that user the sched_switch tracepoint, namely trace-cmd and perf, as well as powertop.

Quoted 7 lines

A simple and extensible kernel ABI is rarely an easy to use end-user interface.

* How can we deprecate, remove, or re-purpose a field in an event ? For instance, the "prio" field in the scheduler instrumentation is an internal implementation detail.

One way is to fix all tools that use it and make sure they get out to the distros before making the change.

Quoted 3 lines

Perhaps it would be good to use the opportunity to meet at the Kernel Summit and discuss this issue.

Agreed. In this topic, I would like to focus more on extending, or implementations on overlaying tracepoints where one tracepoint exists in the code, but there's multiple ways to display it.

-- Steve

Quoted 13 lines

Thanks,

Mathieu and Julien

[1] [RFC PATCH 5/6] tracing: extend scheduling tracepoints

    http://lkml.kernel.org/r/[email protected]
[2] [RFC PATCH v2 0/5] Additional scheduling information in tracepoints
    http://lkml.kernel.org/r/[email protected]
[3] [RFC PATCH v3 0/2] Extend scheduling tracepoints
    http://lkml.kernel.org/r/[email protected]
#2 lore raw
Quoted 7 lines · Steven Rostedt
Quoted 4 lines

* How can we deprecate, remove, or re-purpose a field in an event ? For instance, the "prio" field in the scheduler instrumentation is an internal implementation detail.

One way is to fix all tools that use it and make sure they get out to the distros before making the change.

OR DO THE THING THAT PEOPLE HAVE BEEN TOLD TO DO AT LEAST THREE KERNEL SUMMITS NOW: LEAVE THE DAMN FIELD ALONE, AND FILL IT WITH ZERO. OR ONE. OR BRAN MUFFINS. I DON'T CARE. BUT DON'T REMOVE IT, AND STOP USING IT AS AN EXCUSE FOR WHY NOTHING CAN EVER BE DONE.

Really. I don't want to have this stupid tracing discussion one more time. We've had it. Several times. This exact issue has come up. Several times.

So stop wasting everybodys time one more year. I'm going to walk out if people start discussing this thing again.

Linus "grumpy"

#3 lore raw
Quoted 6 lines · On Thu, 29 Jun 2017 17:03:05 -0700 Linus Torvalds…
Quoted 5 lines · Steven Rostedt
Quoted 4 lines

* How can we deprecate, remove, or re-purpose a field in an event ? For instance, the "prio" field in the scheduler instrumentation is an internal implementation detail.

Quoted 7 lines
Quoted 2 lines

One way is to fix all tools that use it and make sure they get out to the distros before making the change.

OR DO THE THING THAT PEOPLE HAVE BEEN TOLD TO DO AT LEAST THREE KERNEL SUMMITS NOW: LEAVE THE DAMN FIELD ALONE, AND FILL IT WITH ZERO. OR ONE. OR BRAN MUFFINS. I DON'T CARE. BUT DON'T REMOVE IT, AND STOP USING IT AS AN EXCUSE FOR WHY NOTHING CAN EVER BE DONE.

Well, we actually were able to in the past remove a field after getting the one user up to date (powertop) remember? I fixed powertop, waited a few years until the fix was in Debian stable, and then removed the field. Nobody noticed. I thought that was the point. If user space breaks, and nobody is around to complain about it, did it really break?

The reason that was important to remove, is that it was a field in *every* tracepoint. It was only 4 bytes, but when you have 4 million tracepoints in the buffers, that's 4 megs of memory wasted (a normal tracepoint is about 24 bytes, which makes 4 bytes a big percentage). It's similar to wasted fields in the page struct. It bloats up fast.

Quoted 4 lines

Really. I don't want to have this stupid tracing discussion one more time. We've had it. Several times. This exact issue has come up. Several times.

This is actually something quite different, and new. It sounds similar, but its not.

I should have been the one to post the topic, because what Mathieu wrote, makes it sound very much like what we've discussed to death in the past.

What we use to talk about at ksummit was about stable ABIs and such. How to get new tracepoints into the kernel subsystems like the file system and not worry that these tracepoints will cause harm later to development. THAT IS NOT WHAT WE ARE TALKING ABOUT NOW. (just to get your attention ;-)

Quoted 3 lines

So stop wasting everybodys time one more year. I'm going to walk out if people start discussing this thing again.

Here's what the new issue is. We have a single tracepoint in the scheduler that denotes sched switch. It currently looks like this:

name: sched_switch
ID: 287
format:

	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:char prev_comm[16];	offset:8;	size:16;	signed:1;
	field:pid_t prev_pid;	offset:24;	size:4;	signed:1;
	field:int prev_prio;	offset:28;	size:4;	signed:1;
	field:long prev_state;	offset:32;	size:8;	signed:1;
	field:char next_comm[16];	offset:40;	size:16;	signed:1;
	field:pid_t next_pid;	offset:56;	size:4;	signed:1;
	field:int next_prio;	offset:60;	size:4;	signed:1;

The issue is that we now have a new scheduling class called SCHED_DEADLINE, were prio is completely useless. We would like to add the dynamic fields of "remaining runtime", "next deadline", "next period".

Now sched_switch is also one of the most commonly used tracepoints, as it lets a user see what preempts their process, what system services are running and for how long, etc etc. The thing is, we don't want to bloat that tracepoint. Adding fields for a scheduling class that is used by a very small niche class, is a waste for everyone else.

One of the ideas I've had is to allow for "overlays". That is, we don't want to add another trace_sched_switch() in the scheduler, as that will add a little more overhead to the normal non tracing case. Thus, since we already have that hook (the trace_sched_switch) it would be good to tap into it, and have another way to extract more data from the tracepoint. That is, the overlay.

The problem we have is how to implement it?

We could make one tracepoint hook location have several different "tracepoints" in the tracefs directory letting the user choose how much information they want to trace. Have different tracepoints that can be enabled for a single location, where it may show extended fields.

I know people would like to have a way to cut down some fields, as real-estate in the ring buffer is of high value, and the smaller the events are, the more data one can collect. People who use tracing really do care about any wasted space (which is why we like to avoid writing zeros in fields no longer valued, it makes it harder to get the data you are after).

In summary, this is not another beat the dead horse how to do stable tracepoints. The focus is, how to make tracepoints more user customizable for their use cases.

-- Steve

#4 lore raw
Quoted 3 lines · Steven Rostedt

This is actually something quite different, and new. It sounds similar, but its not.

No, I remember the particular scheduler tracepoint discussion.

And the answer remains exactly the same. If some internal detail no longer exists, just zero out the field.

And no, don't add new, even more specialized fields.

Just get over this discussion already.

Linus

#5 lore raw
Quoted 5 lines · On Thu, 29 Jun 2017 17:41:29 -0700 Linus Torvalds…
Quoted 3 lines · Steven Rostedt

This is actually something quite different, and new. It sounds similar, but its not.

Quoted 9 lines

No, I remember the particular scheduler tracepoint discussion.

And the answer remains exactly the same. If some internal detail no longer exists, just zero out the field.

And no, don't add new, even more specialized fields.

Just get over this discussion already.

How does this solve adding key information that lots of people want added. SCHED_DEADLINE is new, and lots of people want to use the tracing infrastructure to analyze their applications. I know when I was at Red Hat, we had our own tracepoints that we added to our kernel to do just that. Right now, everyone has to patch their kernel to get the necessary information. Red Hat isn't the only one. Mathieu is doing it too. Perhaps SuSE is as well.

It doesn't solve anything.

Basically, what you are saying is the opposite of what we feared might happen. People have always been afraid that a tracepoint might inhibit kernel development because of some key information it exposes. Now you are saying, don't touch the tracepoint, and just let it rot. Now the usefulness of a tracepoint has become limited due to advancements in the development of the kernel.

-- Steve

#6 lore raw
In reply to Steven Rostedt
Quoted 14 lines · On Thu, 29 Jun 2017 20:32:24 -0400 Steven Rostedt <[email protected]> wrote:

The problem we have is how to implement it?

We could make one tracepoint hook location have several different "tracepoints" in the tracefs directory letting the user choose how much information they want to trace. Have different tracepoints that can be enabled for a single location, where it may show extended fields.

I know people would like to have a way to cut down some fields, as real-estate in the ring buffer is of high value, and the smaller the events are, the more data one can collect. People who use tracing really do care about any wasted space (which is why we like to avoid writing zeros in fields no longer valued, it makes it harder to get the data you are after).

Quoted 3 lines

In summary, this is not another beat the dead horse how to do stable tracepoints. The focus is, how to make tracepoints more user customizable for their use cases.

Just to explain what Mathieu was talking about with echo and such, is that Peter Zijlstra has been against multiple tracepoints for that one sched switch location. He would prefer a dynamic case, instead of having to enable all tracepoints, to get full functionality, as he tends to use echo / cat to interact with ftrace than by using the tools. He doesn't want the hassle of enabling more than one tracepoint for sched_switch. One could call him lazy ;) but he's also the maintainer of that tracepoint!

What would be really nice is a way to have sched_switch be a dynamic tracepoint where it can trace differently depending on what is being scheduled in and out. I'm sure other tracepoints locations may want a similar feature.

But the devil is in the details of implementation, and we need to do all this without breaking the existing tools that use the tracing system. (See I have been listening to you!)

-- Steve

#7 lore raw
Quoted 4 lines · Steven Rostedt

Just to explain what Mathieu was talking about with echo and such, is that Peter Zijlstra has been against multiple tracepoints for that one sched switch location.

I am too.

Dammit, if somebody cares about one partiocular scheduler, then that person can add dynamic tracepoints.

Leave the existing one alone. Really. Zero out any fields that no longer make sense. Really. Don't beat this damn horse again. It's been dead for three years, and it's not just smelling bad, it's bloating in some scary ways.

The only reason for static tracepoints are for major tools like powertop. There is no way in hell such a tool will care about fields that only exist for one particular scheduler implementation. Don't add new random crap.

If somebody is interested in *one* particular odd low-level scheduler, he damn well can add the dynamic points.

This is the last I want to ever hear about it, and I particularly do not want to have this be a kernel summit discussion. We've had it before. Get over it.

Linus

#8 lore raw
Quoted 7 lines · On Thu, 29 Jun 2017 18:00:42 -0700 Linus Torvalds…
Quoted 4 lines · Steven Rostedt

Just to explain what Mathieu was talking about with echo and such, is that Peter Zijlstra has been against multiple tracepoints for that one sched switch location.

I am too.

I didn't realize you even used the tracepoints.

Quoted 3 lines

Dammit, if somebody cares about one partiocular scheduler, then that person can add dynamic tracepoints.

You mean kprobes? Or perhaps eBPF?

The information about SCHED_DEADLINE is not trivial enough to extract with them.

Quoted 10 lines

Leave the existing one alone. Really. Zero out any fields that no longer make sense. Really. Don't beat this damn horse again. It's been dead for three years, and it's not just smelling bad, it's bloating in some scary ways.

The only reason for static tracepoints are for major tools like powertop. There is no way in hell such a tool will care about fields that only exist for one particular scheduler implementation. Don't add new random crap.

Well, the world does have people that use tools besides powertop.

Quoted 3 lines

If somebody is interested in *one* particular odd low-level scheduler, he damn well can add the dynamic points.

Again, not a trivial task. It's much easier to patch the kernel. Which, I guess is what will be needed from now on.

Quoted 4 lines

This is the last I want to ever hear about it, and I particularly do not want to have this be a kernel summit discussion. We've had it before. Get over it.

OK, this is my last email on the subject. Too bad you feel this way.

Just one last note. I've tried very hard to keep tracing as contained as possible. That is, not to let implementation details and such creep into the rest of the kernel. I worked on making it as robust as possible. All solutions to this had one major requirement. That is, it must be contained, and not something that would cause any disruption in any other part of the kernel or even other tracepoints. But oh well, this idea is now dead. Well, at least until the demand for it boils up into your visibility.

-- Steve

#9 lore raw

I know I said that was going to be my last email, but I just noticed something by re-reading the thread.

Quoted 6 lines · On Thu, 29 Jun 2017 21:16:41 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 6 lines · Steven Rostedt
Quoted 4 lines

Just to explain what Mathieu was talking about with echo and such, is that Peter Zijlstra has been against multiple tracepoints for that one sched switch location.

Quoted 6 lines
Quoted 4 lines

I am too.

Dammit, if somebody cares about one partiocular scheduler, then that person can add dynamic tracepoints.

You mean kprobes? Or perhaps eBPF?

I missed this because you cut it out. What I originally wrote:

Just to explain what Mathieu was talking about with echo and such, is that Peter Zijlstra has been against multiple tracepoints for that one sched switch location. He would prefer a dynamic case, instead of having to enable all tracepoints, to get full functionality, as he tends to use echo / cat to interact with ftrace than by using the tools. He doesn't want the hassle of enabling more than one tracepoint for sched_switch.

Where I mentioned that he prefers a dynamic tracepoint. Is that what you mean? Because what he prefers doesn't exist yet. It's not kprobes nor eBPF. It is something we've been talking about implementing, and we were going to discuss the implementation at ksummit.

Are you OK with that?

-- Steve

#10 lore raw
Quoted 7 lines · Steven Rostedt

Where I mentioned that he prefers a dynamic tracepoint. Is that what you mean? Because what he prefers doesn't exist yet. It's not kprobes nor eBPF. It is something we've been talking about implementing, and we were going to discuss the implementation at ksummit.

Are you OK with that?

I'm not going to be at that discussion, and I don't think it should be a ksummit thing, at least not the maintainership part. It's a technical detail that should be discussed with the people who are interested in that technical detail. Maybe a BOF?

But yes, I was talking about something very similar to what I think Peter is talking about - the ability to attach a ebpf script to kprobes and extract data dynamically. We've supported ebpf tracepoints for years afaik, what is actually missing from using that for whatever particular extension people want to use?

Wasn't that the whole promise (and premise) of using ebpf programs in tracing? Exactly the ability to generate sane statistics and traces dynamically. I know that was what I was sold on, even if it might not actually have worked out that way.

Linus

#11 lore raw
Quoted 7 lines · On Thu, 29 Jun 2017 18:51:14 -0700 Linus Torvalds…
Quoted 6 lines · Steven Rostedt

Where I mentioned that he prefers a dynamic tracepoint. Is that what you mean? Because what he prefers doesn't exist yet. It's not kprobes nor eBPF. It is something we've been talking about implementing, and we were going to discuss the implementation at ksummit.

Quoted 6 lines
Quoted 1 line

Are you OK with that?

I'm not going to be at that discussion, and I don't think it should be a ksummit thing, at least not the maintainership part. It's a technical detail that should be discussed with the people who are interested in that technical detail. Maybe a BOF?

The thread was posted as "TECH TOPIC" which was suppose to be only for those that are interested. It wasn't posted for "MAINTAINERS SUMMIT". Thus, we are fine with that.

IIUC, there's two things happening for "ksummit". There's the "Maintainer's summit" which is your hand picked group. And then there's the extension of individual technical topics for a larger body of people (also this helps get people to the location that may need to be available for anything that comes out of you Maintainer's Summit, but were not invited). This mailing list was for both, with the subject tags to differentiate which is for which.

Quoted 6 lines

But yes, I was talking about something very similar to what I think Peter is talking about - the ability to attach a ebpf script to kprobes and extract data dynamically. We've supported ebpf tracepoints for years afaik, what is actually missing from using that for whatever particular extension people want to use?

Well, I don't want to put words in his mouth, but as he's probably currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We were talking about making the static tracepoints more "dynamic". I'm not sure he's ever used eBPF with tracing.

Quoted 5 lines

Wasn't that the whole promise (and premise) of using ebpf programs in tracing? Exactly the ability to generate sane statistics and traces dynamically. I know that was what I was sold on, even if it might not actually have worked out that way.

eBPF is still very limited in tracing. Currently it is only implemented for perf. Although, it has been on my todo list to get it working for ftrace as well, and implementing eBPF for ftrace can also be on the agenda. The issue is eBPF still needs easier users tools to get it working, as it needs to be compiled down. But that's all details that can be sorted out.

I still like to have a TECH TOPIC on all these issues at Kernel Summit. You don't need to be there, and it would only be for those that are interested in the discussion.

-- Steve

#12 lore raw
Quoted 5 lines · Steven Rostedt

Well, I don't want to put words in his mouth, but as he's probably currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We were talking about making the static tracepoints more "dynamic". I'm not sure he's ever used eBPF with tracing.

I don't know how else you would make them dynamic, though. Realistically, ebpf seems to be working really well for the networking people, and seems to be the obvious solution.

Now, the networking people have obviously *made* it work for them. So it's not like it's some kind of "ebpf automatically solves all problems" thing. ebpf needs some infrastructure too, to be able to get to the interesting data sanely (and safely).

Quoted 4 lines

eBPF is still very limited in tracing. Currently it is only implemented for perf. Although, it has been on my todo list to get it working for ftrace as well, and implementing eBPF for ftrace can also be on the agenda.

Oh, I thought it worked outside of perf already. My bad. I'm actually surprised it doesn't interact with ftrace, since it seems like the perfect use case.

Linus

#13 lore raw
Quoted 3 lines · On Thu, 29 Jun 2017 19:34:53 -0700 Linus Torvalds…

Oh, I thought it worked outside of perf already. My bad. I'm actually surprised it doesn't interact with ftrace, since it seems like the perfect use case.

Agreed. I need to push that up in priority in my todo list.

Getting ebpf functionality into tracing could also help filtering and such, which is still done with tree logic.

Anyway, I would like to get people together in discussing some of these enhancements at ksummit as well. How to load ebpf into ftrace is another ABI issue that we need to do right, otherwise that could bite us years from now.

-- Steve

#14 lore raw
In reply to Linus Torvalds
Quoted 16 lines · Linus Torvalds
Quoted 5 lines · Steven Rostedt

Well, I don't want to put words in his mouth, but as he's probably currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We were talking about making the static tracepoints more "dynamic". I'm not sure he's ever used eBPF with tracing.

I don't know how else you would make them dynamic, though. Realistically, ebpf seems to be working really well for the networking people, and seems to be the obvious solution.

Now, the networking people have obviously *made* it work for them. So it's not like it's some kind of "ebpf automatically solves all problems" thing. ebpf needs some infrastructure too, to be able to get to the interesting data sanely (and safely).

Quoted 8 lines
Quoted 4 lines

eBPF is still very limited in tracing. Currently it is only implemented for perf. Although, it has been on my todo list to get it working for ftrace as well, and implementing eBPF for ftrace can also be on the agenda.

Oh, I thought it worked outside of perf already. My bad. I'm actually surprised it doesn't interact with ftrace, since it seems like the perfect use case.

If Steven has a use case for bpf in ftrace, I don't mind, but I don't see a value yet. Everything we wanted to see inside the kernel we can already do with perf_events and bpf scripts.
Here is one scheduler related script:
https://github.com/iovisor/bcc/blob/master/tools/runqlen.py#L79
it builds a histogram of task->se.cfs_rq->nr_running like

# ./runqlat 
Tracing run queue latency... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 233      |***********                             |
         2 -> 3          : 742      |************************************    |
         4 -> 7          : 203      |**********                              |
         8 -> 15         : 173      |********                                |
        16 -> 31         : 24       |*                                       |
        32 -> 63         : 0        |                                        |

This particular script just samples the whole system at given frequency and uses probe_read() to walk kernel internal data strctures. It's obviously unstable interace and scripts break from time to time. The worst offender over the last years was constantly changing internals of 'struct request' on block side, so the biosnoop.py bpf script has ugly code like this:

#ifdef REQ_WRITE
    data.rwflag = !!(req->cmd_flags & REQ_WRITE);
#elif defined(REQ_OP_SHIFT)
    data.rwflag = !!((req->cmd_flags >> REQ_OP_SHIFT) == REQ_OP_WRITE);
#else
    data.rwflag = !!((req->cmd_flags & REQ_OP_MASK) == REQ_OP_WRITE);

#endif
to be able to run with different kernel versions, but that's fine. When one wants to look inside kernel structures they need to be ready to change their scripts with every kernel version and that is well understood.

Re-reading Mathieu's original email I don't really understand what he's trying to solve that is not solved already.

Also I'm not planning to fly to Prague just for tracing discussion. There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better suited to discuss tracing related topics.

#15 lore raw
Quoted 4 lines · On Thu, 29 Jun 2017 19:58:54 -0700 Alexei Starovoitov…

Also I'm not planning to fly to Prague just for tracing discussion. There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better suited to discuss tracing related topics.

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not moving at all. If it is to be accepted, it needs some talk proposals, and fast!

-- Steve

#16 lore raw
Quoted 4 lines · On Thu, 29 Jun 2017 23:02:51 -0400 Steven Rostedt <[email protected]> wrote:

On Thu, 29 Jun 2017 19:58:54 -0700
Alexei Starovoitov <[email protected]> wrote:

Quoted 8 lines
Quoted 4 lines

Also I'm not planning to fly to Prague just for tracing discussion. There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better suited to discuss tracing related topics.

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not moving at all. If it is to be accepted, it needs some talk proposals, and fast!

Also note, Mathieu has stated he wont be attending Plumbers, and I'm not sure Peter will be either as he has smaller things to attend to.

-- Steve

#17 lore raw
Quoted 12 lines · ----- On Jun 29, 2017, at 11:20 PM, rostedt [email protected] wrote:
Quoted 9 lines · On Thu, 29 Jun 2017 23:02:51 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 4 lines · On Thu, 29 Jun 2017 19:58:54 -0700 Alexei Starovoitov…

Also I'm not planning to fly to Prague just for tracing discussion. There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better suited to discuss tracing related topics.

Quoted 6 lines
Quoted 3 lines

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not moving at all. If it is to be accepted, it needs some talk proposals, and fast!

Also note, Mathieu has stated he wont be attending Plumbers, and I'm not sure Peter will be either as he has smaller things to attend to.

I take it back. Work permit delays postpone my conflicting house renovation work, so I will likely be able to make it to LPC finally. :)

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#18 lore raw
Quoted 5 lines · On Thu, 27 Jul 2017 14:35:45 +0000 (UTC) Mathieu Desnoyers…
Quoted 2 lines

Also note, Mathieu has stated he wont be attending Plumbers, and I'm not sure Peter will be either as he has smaller things to attend to.

I take it back. Work permit delays postpone my conflicting house renovation work, so I will likely be able to make it to LPC finally. :)

Great! I see you updated the Wiki that you are attending as well.

http://wiki.linuxplumbersconf.org/2017:tracing

Thanks, looking forward in seeing you there.

-- Steve

#19 lore raw
In reply to Steven Rostedt

On 6/29/17, 11:02 PM, "Steven Rostedt" <[email protected]> wrote:

Quoted 4 lines · On Thu, 29 Jun 2017 19:58:54 -0700 Alexei Starovoitov…

Also I'm not planning to fly to Prague just for tracing discussion. There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better suited to discuss tracing related topics.

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not moving at all. If it is to be accepted, it needs some talk proposals, and fast!

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

Josef

#20 lore raw
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

-- Steve

#21 lore raw

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 24 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

-- Steve

#22 lore raw

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 13 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

Quoted 9 lines

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

Yeah plumbers is a weird venue for tracing, I always hope that we are going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it. It doesn’t really seem to happen that way tho, and for things like tracing ABI there just aren’t the right people in the room to have that kind of discussion. My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour. If it turns out nobody else has much to talk about then we can just declare tracing is feature complete and we can talk about something else ;). Thanks,

Josef

#23 lore raw
Quoted 28 lines · On Fri, 30 Jun 2017 18:37:59 +0000 Josef Bacik <[email protected]> wrote:

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 13 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

Quoted 9 lines

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

Quoted 12 lines

Yeah plumbers is a weird venue for tracing, I always hope that we are going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it. It doesn’t really seem to happen that way tho, and for things like tracing ABI there just aren’t the right people in the room to have that kind of discussion. My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour. If it turns out nobody else has much to talk about then we can just declare tracing is feature complete and we can talk about something else ;). Thanks,

At this rate, I'm guessing that Tracing is not going to be on the Plumbers' agenda.

-- Steve

#24 lore raw
Quoted 46 lines · ----- On Jul 6, 2017, at 3:10 PM, rostedt [email protected] wrote:
Quoted 30 lines · On Fri, 30 Jun 2017 18:37:59 +0000 Josef Bacik <[email protected]> wrote:

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 15 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 3 lines · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting with you! I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

Quoted 9 lines

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

Quoted 12 lines

Yeah plumbers is a weird venue for tracing, I always hope that we are going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it. It doesn’t really seem to happen that way tho, and for things like tracing ABI there just aren’t the right people in the room to have that kind of discussion. My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour. If it turns out nobody else has much to talk about then we can just declare tracing is feature complete and we can talk about something else ;). Thanks,

Quoted 2 lines

At this rate, I'm guessing that Tracing is not going to be on the Plumbers' agenda.

Since the Kernel Summit and Plumbers do not seem like a good fit to have discussions involving both tracing end users and developers, we have adapted the Tracing Summit schedule to have half day of the usual presentations, and half day dedicated to such discussions. Steven has volunteered to run the discussion part.

The Tracing Summit will take place on October 27th in Prague, on the Friday right after Kernel Summit.

So if you have tracing topics that you would like to discuss at this event, the CFP/CFD and all the information are available here:

http://tracingsummit.org/wiki/TracingSummit2017

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#25 lore raw
Quoted 56 lines · Mathieu Desnoyers
Quoted 53 lines · ----- On Jul 6, 2017, at 3:10 PM, rostedt [email protected] wrote:
Quoted 48 lines · On Fri, 30 Jun 2017 18:37:59 +0000 Josef Bacik <[email protected]> wrote:

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 23 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:

On Fri, 30 Jun 2017 18:24:12 +0000
Josef Bacik <[email protected]> wrote:

Quoted 5 lines

Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have too many.

That said, I could probably come up with a few things, starting with this trace event issue. But it will be pointless if Peter Zijlstra and Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like to discuss where we go from there.

Quoted 5 lines

One last thing. I don't want to have too many responsibilities, as I'm on the LPC program committee and I need to make sure I have time to fulfill any action items I'm responsible for during the conference.

Yeah plumbers is a weird venue for tracing, I always hope that we are going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it.  It doesn’t really seem to happen that way tho, and for things like tracing ABI there just aren’t the right people in the room to have that kind of discussion.  My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro- conference to listen to me bitch for an hour.  If it turns out nobody else has much to talk about then we can just declare tracing is feature complete and we can talk about something else ;).  Thanks,

Quoted 5 lines
Quoted 2 lines

At this rate, I'm guessing that Tracing is not going to be on the Plumbers' agenda.

Since the Kernel Summit and Plumbers do not seem like a good fit to have discussions involving both tracing end users and developers

First the disclaimer: being on the Plumbers Programme Committee, I'm biased.  However, I have to say that the design of Plumbers is to bring together everyone interested in the plumbing of Linux.  That means end users as well, so it's not correct to say it's not a good fit.

It also looks like there's been some renewed interest in having a Tracing MC at Plumbers, so my best guess now is that it will happen. That's not to say the two events can't easily co-exist: being on different continents means better opportunities for attendees with international travel restrictions.

James

#26 lore raw
In reply to Mathieu Desnoyers

Actually, Brendan Gregg got enough proposals together and there will be a tracing MC at Plumbers this year.

-- Steve

Quoted 85 lines · Mathieu Desnoyers
Quoted 5 lines · ----- On Jul 6, 2017, at 3:10 PM, rostedt [email protected] wrote:
Quoted 2 lines · On Fri, 30 Jun 2017 18:37:59 +0000 Josef Bacik <[email protected]> wrote:

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting

with you! I’ll

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

put up my proposal in the next day or two, I think Brendan has

something he’s

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have

too

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

many.

That said, I could probably come up with a few things, starting

with

Quoted 1 line
Quoted 1 line
Quoted 1 line

this trace event issue. But it will be pointless if Peter Zijlstra

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like

to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as

I'm

Quoted 1 line
Quoted 1 line
Quoted 1 line

on the LPC program committee and I need to make sure I have time

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

fulfill any action items I'm responsible for during the

conference.

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

Yeah plumbers is a weird venue for tracing, I always hope that we

are

Quoted 3 lines
Quoted 3 lines

going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it. It doesn’t really seem to happen that way tho,

and

Quoted 1 line
Quoted 1 line

for things like tracing ABI there just aren’t the right people in

the

Quoted 4 lines
Quoted 4 lines

room to have that kind of discussion. My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour. If it turns out nobody else has much to talk about

then

Quoted 1 line
Quoted 1 line

we can just declare tracing is feature complete and we can talk

about

Quoted 5 lines
Quoted 2 lines

something else ;). Thanks,

At this rate, I'm guessing that Tracing is not going to be on the Plumbers' agenda.

Since the Kernel Summit and Plumbers do not seem like a good fit to have
discussions involving both tracing end users and developers, we have adapted the Tracing Summit schedule to have half day of the usual presentations, and half day dedicated to such discussions. Steven has volunteered to run the discussion part.

The Tracing Summit will take place on October 27th in Prague, on the Friday right after Kernel Summit.

So if you have tracing topics that you would like to discuss at this event, the CFP/CFD and all the information are available here:

http://tracingsummit.org/wiki/TracingSummit2017

Thanks,

Mathieu

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#27 lore raw

Do we want to talk about ABI at the micro conference? Facebook uses tracing everywhere in production so I can talk about it from both a user and maintainer standpoint. Thanks,

Josef

Sent from my iPhone

Quoted 96 lines · Steven Rostedt

Actually, Brendan Gregg got enough proposals together and there will be a tracing MC at Plumbers this year.

-- Steve

Quoted 86 lines · Mathieu Desnoyers
Quoted 5 lines · ----- On Jul 6, 2017, at 3:10 PM, rostedt [email protected] wrote:
Quoted 2 lines · On Fri, 30 Jun 2017 18:37:59 +0000 Josef Bacik <[email protected]> wrote:

[ I forgot to add Tom to the Cc list. Sending again. ]

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines · On Fri, 30 Jun 2017 14:29:56 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 1 line · On Fri, 30 Jun 2017 18:24:12 +0000 Josef Bacik <[email protected]> wrote:

Yup I’ll start bugging people to submit talk proposals, starting

with you! I’ll

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

put up my proposal in the next day or two, I think Brendan has

something he’s

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

going to talk about. Thanks,

I shouldn't have used the term "talk", as it really is all about discussions. In fact, if you need more than one slide, you have

too

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

many.

That said, I could probably come up with a few things, starting

with

Quoted 1 line
Quoted 1 line
Quoted 1 line

this trace event issue. But it will be pointless if Peter Zijlstra

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always interesting. Not to mention talking about Tom Zanussi's latest histogram work. It may be pretty much completed, but I would like

to

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as

I'm

Quoted 1 line
Quoted 1 line
Quoted 1 line

on the LPC program committee and I need to make sure I have time

to

Quoted 1 line
Quoted 1 line
Quoted 1 line

fulfill any action items I'm responsible for during the

conference.

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

Yeah plumbers is a weird venue for tracing, I always hope that we

are

Quoted 3 lines
Quoted 3 lines

going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it. It doesn’t really seem to happen that way tho,

and

Quoted 1 line
Quoted 1 line

for things like tracing ABI there just aren’t the right people in

the

Quoted 4 lines
Quoted 4 lines

room to have that kind of discussion. My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour. If it turns out nobody else has much to talk about

then

Quoted 1 line
Quoted 1 line

we can just declare tracing is feature complete and we can talk

about

Quoted 5 lines
Quoted 2 lines

something else ;). Thanks,

At this rate, I'm guessing that Tracing is not going to be on the Plumbers' agenda.

Since the Kernel Summit and Plumbers do not seem like a good fit to have
discussions involving both tracing end users and developers, we have adapted the Tracing Summit schedule to have half day of the usual presentations, and half day dedicated to such discussions. Steven has volunteered to run the discussion part.

The Tracing Summit will take place on October 27th in Prague, on the Friday right after Kernel Summit.

So if you have tracing topics that you would like to discuss at this event, the CFP/CFD and all the information are available here:

https://urldefense.proofpoint.com/v2/url?u=http-3A__tracingsummit.org_wiki_TracingSummit2017&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=sDzg6MvHymKOUgI8SFIm4Q&m=i0WUwxGtMNqWf0sUfcJGu8mjy4EmALhzGj4FSSAj_10&s=uSqVNCjkvfgDy8m4bV0fRhLxXZzd2b5MOnzIs5uAugM&e=

Thanks,

Mathieu

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#28 lore raw
Quoted 3 lines · On Sun, 23 Jul 2017 16:24:09 +0000 Josef Bacik <[email protected]> wrote:

Do we want to talk about ABI at the micro conference? Facebook uses tracing everywhere in production so I can talk about it from both a user and maintainer standpoint. Thanks,

Yes, please add that to the wiki.

-- Steve

#29 lore raw
In reply to Steven Rostedt

Yay, tracing fight!! :/

Quoted 9 lines · Steven Rostedt
Quoted 5 lines · On Thu, 29 Jun 2017 18:51:14 -0700 Linus Torvalds…

But yes, I was talking about something very similar to what I think Peter is talking about - the ability to attach a ebpf script to kprobes and extract data dynamically. We've supported ebpf tracepoints for years afaik, what is actually missing from using that for whatever particular extension people want to use?

Quoted 4 lines

Well, I don't want to put words in his mouth, but as he's probably currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We were talking about making the static tracepoints more "dynamic". I'm not sure he's ever used eBPF with tracing.

So my concerns/objections are two-fold:

- I want only a single static tracepoint in the code.

- I want only a single 'event' associated with this in userspace.

(in particular I only see confusion happening when we have:
sched_switch_fair, sched_switch_rt, sched_switch_deadline events for the exact same event; people will forget to enable one or more and wonder WTF they have holes in their traces)

These are not strange constraints / demands in my book. Just turns out its 'difficult' to pull off or something. I'm in fact fine with simply adding bits to the one tracepoint we have; although others (that'd be you Steve) are not because expensive.

Further complications seem to stem from the fact that I use the tracefs interface exclusively. I don't know how to use perf or trace-cmd or any of that new fangled stuff to do tracing -- nor do I really care, it works for me (same why I'm happy with sysvinit, I don't _want_ to have to relearn my 20+ year old sysadmin skillz, there's better things in live to spend time on, that baby you mentioned for example).

So on that same vein, I'd be entirely helpless using eBPF to do tracing, that's even more complicated. That said, I don't typically need this crud anyway, I just change my kernel and rebuild, reboot and am happy, that's far easier than trying to figure out how eBPF works.

In any case, baby vomit is more fun that this subject :-)

#30 lore raw
In reply to Steven Rostedt
Quoted 2 lines · Steven Rostedt

I didn't realize you even used the tracepoints.

I don't.

But I absolutely refuse to see this kind of idiotic churn, and even more the idiotic "let's discuss it at ksummit" when we've had those tracing discussions for years and they apparently didn't result in anything workable.

Dammit, this stupid tracing issue really has been discussed for too long, and the answer hasn't changed. You can have a static tracepoint, but only for GENERIC stuff. Not random internal stuff that only makes sense to one scheduler implementation. We've made that mistake too many times before, we're not making it again.

Didn't that useless 'prio' field teach you anything? Instead, now you want to add new fields that are not generic, but specific to one very particular scheduling class. EXACTLTY like "prio" was/is.

Quoted 4 lines

You mean kprobes? Or perhaps eBPF?

The information about SCHED_DEADLINE is not trivial enough to extract with them.

You clearly shouldn't extract it unless you are a scheduler developer, and then you can damn well instrument that thing yourself with your own private tracepoints that you don't try to claim are generic and useful to anybody else.

Quoted 1 line

Well, the world does have people that use tools besides powertop.

I said "like powertop".

I did not say "only powertop".

The fact is, no generic scheduler tracing tool that is run by real people is ever going to care about some esoteric field that isn't even relevant to most schedulers out there.

Quoted 3 lines

Just one last note. I've tried very hard to keep tracing as contained as possible. That is, not to let implementation details and such creep into the rest of the kernel.

Yeah, and apparently it was a complete failure, if extending a single breakpoint isn't possible without breaking stuff, and if people STILL talk about the idiotic "oh, but it has that argument that doesn't make sense" issue.

And dammit, having that same stupid argument _again_ isn't going to improve things.

Why can't you just attach some eBPF script to the one tracepoint you already have? I know that has been at least discussed, and it seems to be the only reasonable way forward, since the existing thing clearly isn't working.

Linus

#31 lore raw
Quoted 4 lines · On Thu, 29 Jun 2017 18:38:11 -0700 Linus Torvalds…

Why can't you just attach some eBPF script to the one tracepoint you already have? I know that has been at least discussed, and it seems to be the only reasonable way forward, since the existing thing clearly isn't working.

I like the eBPF idea, unfortunately it suffers the same fate. It only has access to fields that have been exported via the tracepoint. Now we are back to what fields are we going to make visible, and what happens when they no longer make sense?

I guess you can argue that we need to keep modifying that eBPF script, for each kernel release.

-- Steve