tracepoints without user space interfaces
The topic came up again at (of all places) the Schedule Workloads Microconf at Linux Plumbers in LA last week. The addition of tracepoints in locations that maintainers don't want them, only because they don't want them to become an ABI for user space tools. Where these tools then must be supported indefinitely, and may prevent future development of the kernel. This includes the scheduler as well as VFS (mandated by Al Viro).
The current solution by Facebook (told to us by Josef Bacik) is to just hand write kprobes with BPF programs to the locations that they need. When they get a new kernel, they just rewrite the programs because the kprobes and BPF programs break at each new release (or can break).
First it was mentioned to add a hook to locations where it would be easier to get variables, as the compiler could optimize them out, and it becomes difficult even with BPF and kprobes to get the information one would like to have. It was asked if we could add a tracepoint hook in these locations that are not exported to user space where it runs the risk of becoming an ABI. It was pointed out that this mechanism already exists in the kernel.
A tracepoint is the hook in the kernel. The TRACE_EVENT() macro is built on top of a tracepoint to export it to user space. But the tracepoint itself can be manually added anywhere and there will be no creation of trace event files in the tracefs directory, nor would perf be able to access it. But the advantage of having this hook is that a kernel module could access it without a problem.
By adding tracepoints in the scheduler and VFS, without the TRACE_EVENT macros that export them to user space, it would be much easier for companies like Facebook, Red Hat and SuSE to add a module that can tap into these hooks and build their custom analysis tools on top.
Requiring an external and custom module to access the tracepoints on live systems (that is, an unmodified vanilla kernel or distro kernel) will help these companies implement advance analytical tools to monitor their production kernels, and because it requires a module, and it has been stated several times in the past that there is no KABI with module interfaces, the maintainers of these hooks should have no fear that they will become a stable interface.
Now, I will also point out that if one of the tracepoint hooks prove to be useful for a generic tool, then this could be an incentive to have the maintainer change the tracepoint hook into a full blown TRACE_EVENT() and upgrade it to an ABI, after having time to see how it is useful. This is a better method than having tens of trace events where one random one proves to be useful for tools and surprises the maintainer that the code it affects can no longer be changed.
Thoughts?
-- Steve
Cc’ing my personal address so I can reply with a sane email client.
On 9/20/17, 9:50 AM, "Steven Rostedt" <[email protected]> wrote:
The topic came up again at (of all places) the Schedule Workloads Microconf at Linux Plumbers in LA last week. The addition of tracepoints in locations that maintainers don't want them, only because they don't want them to become an ABI for user space tools. Where these tools then must be supported indefinitely, and may prevent future development of the kernel. This includes the scheduler as well as VFS (mandated by Al Viro).
The current solution by Facebook (told to us by Josef Bacik) is to just hand write kprobes with BPF programs to the locations that they need. When they get a new kernel, they just rewrite the programs because the kprobes and BPF programs break at each new release (or can break).
First it was mentioned to add a hook to locations where it would be easier to get variables, as the compiler could optimize them out, and it becomes difficult even with BPF and kprobes to get the information one would like to have. It was asked if we could add a tracepoint hook in these locations that are not exported to user space where it runs the risk of becoming an ABI. It was pointed out that this mechanism already exists in the kernel.
A tracepoint is the hook in the kernel. The TRACE_EVENT() macro is built on top of a tracepoint to export it to user space. But the tracepoint itself can be manually added anywhere and there will be no creation of trace event files in the tracefs directory, nor would perf be able to access it. But the advantage of having this hook is that a kernel module could access it without a problem.
By adding tracepoints in the scheduler and VFS, without the TRACE_EVENT macros that export them to user space, it would be much easier for companies like Facebook, Red Hat and SuSE to add a module that can tap into these hooks and build their custom analysis tools on top.
Requiring an external and custom module to access the tracepoints on live systems (that is, an unmodified vanilla kernel or distro kernel) will help these companies implement advance analytical tools to monitor their production kernels, and because it requires a module, and it has been stated several times in the past that there is no KABI with module interfaces, the maintainers of these hooks should have no fear that they will become a stable interface.
Now, I will also point out that if one of the tracepoint hooks prove to be useful for a generic tool, then this could be an incentive to have the maintainer change the tracepoint hook into a full blown TRACE_EVENT() and upgrade it to an ABI, after having time to see how it is useful. This is a better method than having tens of trace events where one random one proves to be useful for tools and surprises the maintainer that the code it affects can no longer be changed.
Thoughts?
-- Steve
Quoted 37 lines · Josef Bacik
Cc’ing my personal address so I can reply with a sane email client.
On 9/20/17, 9:50 AM, "Steven Rostedt" <[email protected]> wrote:
The topic came up again at (of all places) the Schedule Workloads Microconf at Linux Plumbers in LA last week. The addition of tracepoints in locations that maintainers don't want them, only because they don't want them to become an ABI for user space tools. Where these tools then must be supported indefinitely, and may prevent future development of the kernel. This includes the scheduler as well as VFS (mandated by Al Viro).
The current solution by Facebook (told to us by Josef Bacik) is to just hand write kprobes with BPF programs to the locations that they need. When they get a new kernel, they just rewrite the programs because the kprobes and BPF programs break at each new release (or can break).
First it was mentioned to add a hook to locations where it would be easier to get variables, as the compiler could optimize them out, and it becomes difficult even with BPF and kprobes to get the information one would like to have. It was asked if we could add a tracepoint hook in these locations that are not exported to user space where it runs the risk of becoming an ABI. It was pointed out that this mechanism already exists in the kernel.
A tracepoint is the hook in the kernel. The
TRACE_EVENT()macro is built on top of a tracepoint to export it to user space. But the tracepoint itself can be manually added anywhere and there will be no creation of trace event files in the tracefs directory, nor would perf be able to access it. But the advantage of having this hook is that a kernel module could access it without a problem.By adding tracepoints in the scheduler and VFS, without the
TRACE_EVENTmacros that export them to user space, it would be much easier for companies like Facebook, Red Hat and SuSE to add a module that can tap into these hooks and build their custom analysis tools on top.
Quoted 8 lines
Requiring an external and custom module to access the tracepoints on live systems (that is, an unmodified vanilla kernel or distro kernel) will help these companies implement advance analytical tools to monitor their production kernels, and because it requires a module, and it has been stated several times in the past that there is no KABI with module interfaces, the maintainers of these hooks should have no fear that they will become a stable interface.
The tricky part is we want to be able to access these from eBPF. I argue that eBPF is run in the kernel so it has the same rules as kernel modules. Others seem less convinced of this argument, so it would be good to get a definitive answer. Thanks,
Josef
Quoted 4 lines · On Wed, 20 Sep 2017 11:04:05 -0400 Josef Bacik <[email protected]> wrote:
The tricky part is we want to be able to access these from eBPF. I argue that eBPF is run in the kernel so it has the same rules as kernel modules. Others seem less convinced of this argument, so it would be good to get a definitive answer. Thanks,
Note, adding a module to let eBPF access these tracepoints would also be trivial. Would a module be of issue at FB? It could be easily added at boot up.
-- Steve
Hello,
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Wed, 20 Sep 2017 11:04:05 -0400 Josef Bacik <[email protected]> wrote:
The tricky part is we want to be able to access these from eBPF. I argue that eBPF is run in the kernel so it has the same rules as kernel modules. Others seem less convinced of this argument, so it would be good to get a definitive answer. Thanks,
Note, adding a module to let eBPF access these tracepoints would also be trivial. Would a module be of issue at FB? It could be easily added at boot up.
JFI, seems that Josef's reply didn't make it to the ksummit-discuss list. I'm interested in this topic, tho.
-ss
Quoted 18 lines · Josef Bacik
Quoted 17 lines · Josef Bacik
Cc’ing my personal address so I can reply with a sane email client.
On 9/20/17, 9:50 AM, "Steven Rostedt" <[email protected]> wrote:
The topic came up again at (of all places) the Schedule Workloads Microconf at Linux Plumbers in LA last week. The addition of tracepoints in locations that maintainers don't want them, only because they don't want them to become an ABI for user space tools. Where these tools then must be supported indefinitely, and may prevent future development of the kernel. This includes the scheduler as well as VFS (mandated by Al Viro).
The current solution by Facebook (told to us by Josef Bacik) is to just hand write kprobes with BPF programs to the locations that they need. When they get a new kernel, they just rewrite the programs because the kprobes and BPF programs break at each new release (or can break).
Quoted 7 lines
Quoted 7 lines
First it was mentioned to add a hook to locations where it would be easier to get variables, as the compiler could optimize them out, and it becomes difficult even with BPF and kprobes to get the information one would like to have. It was asked if we could add a tracepoint hook in these locations that are not exported to user space where it runs the risk of becoming an ABI. It was pointed out that this mechanism already exists in the kernel.
Aren't we beating the dead horse?
A year ago at the kernel summit:
https://lwn.net/Articles/705270/
"The session concluded with Linus saying that, in the history of kernel development, nobody has ever screamed about a change to a tracepoint. He allowed that this might happen as the use of tracepoints increases. But, he said, there is no point in making a big deal about that possibility before it proves to be a problem."
So instead of inventing trace markers and other new things that are just like existing tracepoints but without arguments how about adding normal tracepoints with one or two arguments task* and rq* bpf progs can walk whatever internals of these structs they need with probe_read() and that would be plenty of info for most users including kernel developers.
In that sense the only difference between these new sched tracepoints and existing kprobe-based scripts will be the speed and ease of access to task/rq pointers.
If pretty print of tracepoints into trace_pipe is an abi concern then don't print anything.
Existing sched tracepoints are not useful from bpf point of view, since they don't have pointers in arguments and instead print comm/pid/cpu which is not very interesting.
Dumb kprobe in enqueue_task_*() is more powerful
since progs can simply bpf_trace_printk("%d\n", rq->nr_running); btw I won't be in Prague, so best to discuss over email.
Quoted 1 line · On Fri, 29 Sep 2017 16:50:23 -0700 Alexei Starovoitov…
Aren't we beating the dead horse?
Not really, because this dead horse can still give quite a hell of a kick back.
Quoted 6 lines
A year ago at the kernel summit:
https://lwn.net/Articles/705270/
"The session concluded with Linus saying that, in the history of kernel development, nobody has ever screamed about a change to a tracepoint. He allowed that this might happen as the use of tracepoints increases. But, he said, there is no point in making a big deal about that possibility before it proves to be a problem."
We have two, possibly three instances that user space has caused tracepoint hell already. Yes, powertop screamed about changing a tracepoint. We have silly crap in sched_switch and sched_wakeup due to user space not wanting that to change. And there was just another tracepoint having to carry blank fields because userspace expects them to exist.
Quoted 2 lines
So instead of inventing trace markers and other new things that are just
There is no "inventing". They already exist. In fact, that's what TRACE_EVENT() macros are built on. In fact, what we are talking about was the original introduction of tracepoints. This is what is in tracepoint.h and is implemented in tracepoint.c. No new code needs to be done to implement this. All it would take is to put in the tracepoints by hand, without the use of the TRACE_EVENT macros.
Quoted 1 line
like existing tracepoints but without arguments how about
We are not talking about tracepoints without arguments.
Quoted 7 lines
adding normal tracepoints with one or two arguments task* and rq* bpf progs can walk whatever internals of these structs they need with
probe_read()and that would be plenty of info for most users including kernel developers.
In that sense the only difference between these new sched tracepoints and existing kprobe-based scripts will be the speed and ease of access to task/rq pointers.
That may be what we are talking about ;-)
Quoted 2 lines
If pretty print of tracepoints into
trace_pipeis an abi concern then don't print anything.
No, that's not the issue. The issue is what gets written into the binary buffers of perf or ftrace.
Quoted 3 lines
Existing sched tracepoints are not useful from bpf point of view, since they don't have pointers in arguments and instead print comm/pid/cpu which is not very interesting.
?? The sched tracepoints pass in the task pointers that they deal with.
Quoted 3 lines
Dumb kprobe in enqueue_task_*() is more powerful
since progs can simplybpf_trace_printk("%d\n", rq->nr_running); btw I won't be in Prague, so best to discuss over email.
Well this isn't just about bpf, it's also about tracing.
-- Steve