Feedback on user-events UAPI
Hi Beau,
I have queued a few questions I would like to discuss with respect to the proposed user events UAPI. I originally planned to expand further on them, but I now think it's best if I ask away right now and we clarify things through discussion:
First, I find it odd that the event enable bitmask and the event ID and payload type registration must be combined. I can think of various use-cases where other tracers would be interested to use the event-enable bitmask facility without polluting the event ID/payload registration data structures with useless data. Can those be split into two distinct independent ABIs ?
I can't help but notice that this new user-space instrumentation infrastructure/ABI can only be used for tracing user-space through kernel tracers. Considering that ABIs dictated by the kernel usually end up being de facto standards, I am concerned that if it is unable to allow purely user-space tracers to use it, then all applications will end up being statically instrumented in ways that prevent user-space tracers from hooking efficiently on the static instrumentation. As I have replied in an earlier thread, purely user-space tracers are not just marginally faster than kernel tracers for tracing user-space. They are an order of magnitude faster as soon as all the proper configuration steps are taken to ensure there are no system calls on the tracer fast path. Therefore, it would be sad to effectively dismiss efficient tracer implementations for the sake of easiness of implementation of today's user-event ABI. This will cause a precedent we will be stuck with later.
Linux kernel developers involved in implementation of instrumentation within Linux have spent a lot of effort to make sure the instrumentation is orthogonal to the tracing technology (tracepoints, kprobe, kretprobe...). I understand that making sure the user-space instrumentation ABI keeps this orthogonal is a lot more work, but nobody said that exposing ABIs to user-space was easy. ;-)
The other point I would like to raise is container awareness. I can't help but notice that the user events ABI is exposed to trace all containers, with the intent to be used (consumed) from some privileged namespace (e.g. root pid namespace). This works in use-cases where the user of the tracing data controls the entire machine (all containers), but not so much if the user is a single tenant within a multi-tenants systems. I would expect that a proper namespace-aware facility would take care of making sure that a trace consumer could observe what is instrumented within its own container, and within nested containers as well. The fact that all container questions are entirely dismissed, thus keeping a event-enable bitmask registry and event ID/type registry global to the entire system, is not compatible with consuming traces from a non-privileged container, and I suspect this may also be used as a side-channel to learn information about what other containers are doing in a multi-tenant system.
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 6 lines · Mathieu Desnoyers
Hi Beau,
I have queued a few questions I would like to discuss with respect to the proposed user events UAPI. I originally planned to expand further on them, but I now think it's best if I ask away right now and we clarify things through discussion:
Sounds good, thanks for taking the time to write down your thoughts.
Quoted 6 lines
First, I find it odd that the event enable bitmask and the event ID and payload type registration must be combined. I can think of various use-cases where other tracers would be interested to use the event-enable bitmask facility without polluting the event ID/payload registration data structures with useless data. Can those be split into two distinct independent ABIs ?
If the ABI were to expand beyond kernel hosted trace_events, then I think these could be split out. However, I believe the question of other tracers was asked previously [1] and was out of scope for this work.
I don't have a plan to use other tracers, but that doesn't mean I would close the door on the ability to have other tracers participate in the ABI.
I believe Steven would ultimately own if something hosted in tracefs would allow non-kernel tracers to be involved. If the events don't follow the kernel format, I could see this being pretty hard to get right while still allowing the kernel to filter them via ftrace, etc.
Masami should also have an opinion on this, regarding registering and formats within the ABI.
Steven and Masami, do you mind giving your opinion/guidance here?
Quoted 13 lines
I can't help but notice that this new user-space instrumentation infrastructure/ABI can only be used for tracing user-space through kernel tracers. Considering that ABIs dictated by the kernel usually end up being de facto standards, I am concerned that if it is unable to allow purely user-space tracers to use it, then all applications will end up being statically instrumented in ways that prevent user-space tracers from hooking efficiently on the static instrumentation. As I have replied in an earlier thread, purely user-space tracers are not just marginally faster than kernel tracers for tracing user-space. They are an order of magnitude faster as soon as all the proper configuration steps are taken to ensure there are no system calls on the tracer fast path. Therefore, it would be sad to effectively dismiss efficient tracer implementations for the sake of easiness of implementation of today's user-event ABI. This will cause a precedent we will be stuck with later.
Quoted 6 lines
Linux kernel developers involved in implementation of instrumentation within Linux have spent a lot of effort to make sure the instrumentation is orthogonal to the tracing technology (tracepoints, kprobe, kretprobe...). I understand that making sure the user-space instrumentation ABI keeps this orthogonal is a lot more work, but nobody said that exposing ABIs to user-space was easy. ;-)
I agree on a lot of these points. This is why user_events uses trace_events instead of rolling something custom. It just hooks the probes already attached and acts as a bridge between a user buffer and the various probes that are attached.
Since these are trace_events through the kernel we can re-use ways for grabbing the stack, hardware counters, etc. via perf. We can also apply filtering and debug things occurring very easily via ftrace (I do this all the time to help figure out if events are really registered and writing things out). Things just work without extra lines of code.
User events is using all the common framework that has been put into the kernel to make this work, very little is specific to user mode (beside what absolutely needs to be, like mmap).
Since the way to get data into user_events is via the write() syscall, I don't see how the data can be used purely on the user side (beside just the mmap part of it, which is likely why you want it split up, right?).
If the ABI were to split up between a pure status map and a data emit mechanism, how do you envision the data emit mechanism to look? Would it be the same we have now, or would there be some way to have the ABI switch between kernel tracers and user tracers avoiding the write syscall?
Quoted 5 lines
The other point I would like to raise is container awareness. I can't help but notice that the user events ABI is exposed to trace all containers, with the intent to be used (consumed) from some privileged namespace (e.g. root pid namespace). This works in use-cases where the user of the tracing data controls the entire machine (all containers), but not so much if the user is a single tenant within
Quoted 9 lines
a multi-tenants systems. I would expect that a proper namespace-aware facility would take care of making sure that a trace consumer could observe what is instrumented within its own container, and within nested containers as well. The fact that all container questions are entirely dismissed, thus keeping a event-enable bitmask registry and event ID/type registry global to the entire system, is not compatible with consuming traces from a non-privileged container, and I suspect this may also be used as a side-channel to learn information about what other containers are doing in a multi-tenant system.
This is the same as uprobes/kprobes/tracefs today, right? This seems much wider than just user_events.
We started a conversation [2] about getting tracefs more container/namespace aware. I think this would be a good thing, but I'm not sure it's ABI related.
My understanding is that we would use the current users cgroup/namespace to determine which pages to give back to them once tracefs had it's own cgroup/namespace, ideally.
Quoted 8 lines
Thanks,
Mathieu
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Thanks,
-Beau
Quoted 42 lines · ----- On May 12, 2022, at 11:47 AM, Mathieu Desnoyers…
Hi Beau,
I have queued a few questions I would like to discuss with respect to the proposed
user events UAPI. I originally planned to expand further on them, but I now think it's
best if I ask away right now and we clarify things through discussion:First, I find it odd that the event enable bitmask and the event ID and payload type registration must be combined. I can think of various use-cases where other tracers would be interested to use the event-enable bitmask facility without polluting the event ID/payload registration data structures with useless data. Can those be split into two distinct independent ABIs ?
I can't help but notice that this new user-space instrumentation infrastructure/ABI
can only be used for tracing user-space through kernel tracers. Considering that ABIs dictated by the kernel usually end up being de facto standards, I am concerned
that if it is unable to allow purely user-space tracers to use it, then all applications
will end up being statically instrumented in ways that prevent user-space tracers from
hooking efficiently on the static instrumentation. As I have replied in an earlier
thread, purely user-space tracers are not just marginally faster than kernel tracers
for tracing user-space. They are an order of magnitude faster as soon as all the proper
configuration steps are taken to ensure there are no system calls on the tracer fast path. Therefore, it would be sad to effectively dismiss efficient tracer implementations for the sake of easiness of implementation of today's user-event ABI. This will cause a precedent we will be stuck with later.Linux kernel developers involved in implementation of instrumentation within Linux
have spent a lot of effort to make sure the instrumentation is orthogonal to the tracing technology (tracepoints, kprobe, kretprobe...). I understand that making sure the user-space instrumentation ABI keeps this orthogonal is a lot more work,
but nobody said that exposing ABIs to user-space was easy. ;-)
Quoted 14 lines
The other point I would like to raise is container awareness. I can't help but notice that the user events ABI is exposed to trace all containers, with the intent
to be used (consumed) from some privileged namespace (e.g. root pid namespace). This works in use-cases where the user of the tracing data controls the entire machine (all containers), but not so much if the user is a single tenant within a multi-tenants systems. I would expect that a proper namespace-aware facility would take care of making sure that a trace consumer could observe what is instrumented within its own container, and within nested containers as well. The fact that all container questions are entirely dismissed, thus keeping a event-enable bitmask registry and event ID/type registry global to the entire system, is not compatible with consuming traces from a non-privileged container, and I suspect this may also be used as a side-channel to learn information about what other containers are doing in a multi-tenant system.
One more thought: I may have simply missed it, but is there any user events code which dynamically validates that the input from user-space writev() indeed match the event description layout ? I'm thinking about wrong size, too short strings, too long strings, missing null terminator and so on. Any user input that could make the trace unreadable should never reach the tracing buffers.
Thanks,
Mathieu
Quoted 9 lines
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Quoted 59 lines · Mathieu Desnoyers
Quoted 42 lines · ----- On May 12, 2022, at 11:47 AM, Mathieu Desnoyers…
Hi Beau,
I have queued a few questions I would like to discuss with respect to the proposed
user events UAPI. I originally planned to expand further on them, but I now think it's
best if I ask away right now and we clarify things through discussion:First, I find it odd that the event enable bitmask and the event ID and payload type registration must be combined. I can think of various use-cases where other tracers would be interested to use the event-enable bitmask facility without polluting the event ID/payload registration data structures with useless data. Can those be split into two distinct independent ABIs ?
I can't help but notice that this new user-space instrumentation infrastructure/ABI
can only be used for tracing user-space through kernel tracers. Considering that ABIs dictated by the kernel usually end up being de facto standards, I am concerned
that if it is unable to allow purely user-space tracers to use it, then all applications
will end up being statically instrumented in ways that prevent user-space tracers from
hooking efficiently on the static instrumentation. As I have replied in an earlier
thread, purely user-space tracers are not just marginally faster than kernel tracers
for tracing user-space. They are an order of magnitude faster as soon as all the proper
configuration steps are taken to ensure there are no system calls on the tracer fast path. Therefore, it would be sad to effectively dismiss efficient tracer implementations for the sake of easiness of implementation of today's user-event ABI. This will cause a precedent we will be stuck with later.Linux kernel developers involved in implementation of instrumentation within Linux
have spent a lot of effort to make sure the instrumentation is orthogonal to the tracing technology (tracepoints, kprobe, kretprobe...). I understand that making sure the user-space instrumentation ABI keeps this orthogonal is a lot more work,
but nobody said that exposing ABIs to user-space was easy. ;-)Quoted 14 lines
The other point I would like to raise is container awareness. I can't help but notice that the user events ABI is exposed to trace all containers, with the intent
to be used (consumed) from some privileged namespace (e.g. root pid namespace). This works in use-cases where the user of the tracing data controls the entire machine (all containers), but not so much if the user is a single tenant within a multi-tenants systems. I would expect that a proper namespace-aware facility would take care of making sure that a trace consumer could observe what is instrumented within its own container, and within nested containers as well. The fact that all container questions are entirely dismissed, thus keeping a event-enable bitmask registry and event ID/type registry global to the entire system, is not compatible with consuming traces from a non-privileged container, and I suspect this may also be used as a side-channel to learn information about what other containers are doing in a multi-tenant system.
Quoted 6 lines
One more thought: I may have simply missed it, but is there any user events code which dynamically validates that the input from user-space
writev()indeed match the event description layout ? I'm thinking about wrong size, too short strings, too long strings, missing null terminator and so on. Any user input that could make the trace unreadable should never reach the tracing buffers.
Yes, there are validators attached to events to ensure that the minimum size is written and ensure types that require certain safety gaurantees are met. Currently I believe the only such requirement is for variable length strings to have proper null termination (beside the min size requirement).
See user_event_validate() in kernel/trace/trace_events_user.c. This is called for both perf and ftrace buffers, buffers discard if the validation fails.
Related patch:
https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
Thanks,
Mathieu
Quoted 1 line
Quoted 13 lines
Quoted 8 lines
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.comSignature · 3 lines
Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com
Thanks,
-Beau