udiag - User mode to trace_event (ftrace, perf, eBPF) ABI
RFC PATCH 1/1 udiag - User mode to trace_event (ftrace, perf, eBPF) ABI
User mode processes that require emitting diagnostic data are currently limited to using uprobes to get data into trace_events. The udiag ABI offers a way for user mode processes to write diagnostic data to trace_events much faster than the uprobe die chain handler.
In addition a shared page is exposed out to registered user processes that is used to enable single branch checking for if the trace_event is being traced. This allows equivalent overhead as a uprobe site when tracing is not enabled.
User processes register a trace_event to use via a device exposed at /dev/udiag. System owners can write udev rules to decide the security boundary. udiag is limited to only a page size worth of trace_events that are isolated and put under the udiag subsystem. User processes write events out via /dev/udiag. This allows for many languages and processes to contribute the same events regardless of where in the code the event was generated. This allows common code to be shared and centrally processed on the machine within a eBPF program regardless how the code has evolved as long as the data within the event follows the same data format as before.
An example of this is common error conditions that can happen across a suite of processes. A single eBPF program can watch for the single event across all processes, regardless of binary location or language used to create the process. Once problems are found, additional eBPF programs can be launched to impose further tracing, run mitigations, etc.
--- kernel/trace/Kconfig | 15 + kernel/trace/Makefile | 1 + kernel/trace/trace_events_udiag.c | 650 ++++++++++++++++++++++++++++++ 3 files changed, 666 insertions(+) create mode 100644 kernel/trace/trace_events_udiag.c
kernel/trace/Kconfig +15 -0
@@ -658,6 +658,21 @@ config SYNTH_EVENTS If in doubt, say N. +config UDIAG_EVENTS + bool "User diagnostic trace events" + select TRACING + default n + help + User diagnostic events are user-defined trace events that + can combine many user diagnostic events into a single + trace event from multiple processes. User diagnostic + events are generated by writing to a device file. User + processes can determine if their diagnostic events should + be generated by memory mapping a device file and checking + for an associated byte being non-zero. + + If in doubt, say N. + config HIST_TRIGGERS bool "Histogram triggers" depends on ARCH_HAVE_NMI_SAFE_CMPXCHG
kernel/trace/Makefile +1 -0
@@ -79,6 +79,7 @@ obj-$(CONFIG_EVENT_TRACING) += trace_events_trigger.o obj-$(CONFIG_TRACE_EVENT_INJECT) += trace_events_inject.o obj-$(CONFIG_SYNTH_EVENTS) += trace_events_synth.o obj-$(CONFIG_HIST_TRIGGERS) += trace_events_hist.o +obj-$(CONFIG_UDIAG_EVENTS) += trace_events_udiag.o obj-$(CONFIG_BPF_EVENTS) += bpf_trace.o obj-$(CONFIG_KPROBE_EVENTS) += trace_kprobe.o obj-$(CONFIG_TRACEPOINTS) += error_report-traces.o
kernel/trace/trace_events_udiag.c new file +650 -0
650 lines not on this page · raw message
Signature · 1 line
2.17.1
Steven Rostedt
maintainerHi Beau,
BTW, feel free to Cc LKML too ([email protected])
Quoted 21 lines · On Tue, 27 Jul 2021 12:35:35 -0700 Beau Belgrave <[email protected]>…
User mode processes that require emitting diagnostic data are currently limited to using uprobes to get data into
trace_events. The udiag ABI offers a way for user mode processes to write diagnostic data totrace_eventsmuch faster than the uprobe die chain handler.In addition a shared page is exposed out to registered user processes that is used to enable single branch checking for if the
trace_eventis being traced. This allows equivalent overhead as a uprobe site when tracing is not enabled.User processes register a
trace_eventto use via a device exposed at/dev/udiag. System owners can write udev rules to decide the security boundary. udiag is limited to only a page size worth oftrace_eventsthat are isolated and put under the udiag subsystem. User processes write events out via/dev/udiag. This allows for many languages and processes to contribute the same events regardless of where in the code the event was generated. This allows common code to be shared and centrally processed on the machine within a eBPF program regardless how the code has evolved as long as the data within the event follows the same data format as before.
Quoted 8 lines
An example of this is common error conditions that can happen across a suite of processes. A single eBPF program can watch for the single event across all processes, regardless of binary location or language used to create the process. Once problems are found, additional eBPF programs can be launched to impose further tracing, run mitigations, etc.
Signed-off-by Beau Belgrave <[email protected]>
Can you provide user space code that would show a use case of this implementation. Understanding exactly what is expected on the user side will help tremendously with understanding the kernel side.
Thanks,
-- Steve
Quoted 39 lines · Steven Rostedt
Hi Beau,
BTW, feel free to Cc LKML too ([email protected])
Quoted 21 lines · On Tue, 27 Jul 2021 12:35:35 -0700 Beau Belgrave <[email protected]>…
User mode processes that require emitting diagnostic data are currently limited to using uprobes to get data into
trace_events. The udiag ABI offers a way for user mode processes to write diagnostic data totrace_eventsmuch faster than the uprobe die chain handler.In addition a shared page is exposed out to registered user processes that is used to enable single branch checking for if the
trace_eventis being traced. This allows equivalent overhead as a uprobe site when tracing is not enabled.User processes register a
trace_eventto use via a device exposed at/dev/udiag. System owners can write udev rules to decide the security boundary. udiag is limited to only a page size worth oftrace_eventsthat are isolated and put under the udiag subsystem. User processes write events out via/dev/udiag. This allows for many languages and processes to contribute the same events regardless of where in the code the event was generated. This allows common code to be shared and centrally processed on the machine within a eBPF program regardless how the code has evolved as long as the data within the event follows the same data format as before.Quoted 8 lines
An example of this is common error conditions that can happen across a suite of processes. A single eBPF program can watch for the single event across all processes, regardless of binary location or language used to create the process. Once problems are found, additional eBPF programs can be launched to impose further tracing, run mitigations, etc.
Signed-off-by Beau Belgrave <[email protected]>
Quoted 7 lines
Can you provide user space code that would show a use case of this implementation. Understanding exactly what is expected on the user side will help tremendously with understanding the kernel side.
Thanks,
-- Steve
Sure thing, and thanks for the reply! I appreciate it.
For clarity, would you like a resend with the user mode code in the description or would you like an in-thread example?
Thanks,
-Beau
Steven Rostedt
maintainerQuoted 2 lines · On Tue, 3 Aug 2021 15:52:00 -0700 Beau Belgrave <[email protected]>…
For clarity, would you like a resend with the user mode code in the description or would you like an in-thread example?
In thread example is fine. I'd just like to understand what exactly you plan on doing with it.
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 2 lines · On Tue, 3 Aug 2021 15:52:00 -0700 Beau Belgrave <[email protected]>…
For clarity, would you like a resend with the user mode code in the description or would you like an in-thread example?
In thread example is fine. I'd just like to understand what exactly you plan on doing with it.
-- Steve
Internally we have single trace_events that actually represent many events. We do this by having the payload start with an int that the eBPF programs always probes first to get a sub-event ID. The sub-event ID is then used to determine the actual payload format. We typically use this pattern to enable a single eBPF program to turn on a class of events that require tracing. An example is tracing out all the network related errors, of which there may be many events all with different payloads.
IE:
struct packeterror {
int id;
int packetnumber;
int errorcode;
};struct connerror {
int id;
int connnumber;
int ip4;
int errorcode;
};Both packeterror and connerror would be output to a trace_event with a name like ms.net.errors. packeterror might have an id of 0 while connerror would have an id of 1. eBPF or the trace decoder would check the first int and do further decoding on the payload (or skip it). udiag sends user data as an eBPF context struct, so user probing costs are delayed until the program sees something that warrants it. This limits system cost when tracing is enabled but only a subset of the events are wanted.
This pattern makes writing an eBPF program that is revolved around common data much easier. Internally we have macros that translate to the page check followed by the write syscall. These macros are also used post compile to auto-generate eBPF probe statements, etc. to make it harder for developers to get the code wrong on either side.
Here's a simple example showing the general working order and flow we use on top of udiag. The payload in this case is very simple (a single int). Payloads in general can be anything, the kernel side doesn't force a user to use the sub-event models. It works great for us, but some other users might not require it or have their ideas.
#include <unistd.h>
#include <stdio.h>
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <errno.h>
#include <fcntl.h>
#define DIAG_IOC_MAGIC '*'
#define DIAG_IOCSREG _IOW(DIAG_IOC_MAGIC, 0, char*) #define DIAG_IOCSDEL _IOW(DIAG_IOC_MAGIC, 1, char*)
int udiag_init(char **eventpage)
{
int ret;
int page_size = sysconf(_SC_PAGESIZE);
int fd = open("/dev/udiag", O_RDWR);
if (fd == -1) {
ret = errno;
goto out;
}
/* Map in single page for event enabled checking */
char *page = mmap(NULL, page_size, PROT_READ, MAP_SHARED, fd, 0);
if (page != MAP_FAILED) {
*eventpage = page;
ret = 0;
} else {
ret = errno;
}
close(fd);out:
return ret; }
int udiag_event_open(char *name, int *eventfd, int *eventindex)
{
int ret;
int fd;
long status;
fd = open("/dev/udiag", O_RDWR);
if (fd == -1) {
ret = errno;
goto out;
}
/* Make data written to this fd log under the named event */
status = ioctl(fd, DIAG_IOCSREG, name);
if (status < 0) {
ret = errno;
close(fd);
goto out;
}
/* Give caller back an fd for writing to this event */
*eventfd = fd;
/* Give caller back the index of the page to check if enabled */
*eventindex = (int)status;
ret = 0;out:
return ret; }
int main(int argc, char *argv[])
{
int err, fd, index, payload;
char *page;
err = udiag_init(&page);
if (err != 0) {
return err;
}
err = udiag_event_open("testevent", &fd, &index);
if (err != 0) {
return err;
}
payload = 0;
printf("Press enter to write a testevent\n");
fetch:
getchar();
/* Avoid write overhead when nothing is listening */
if (page[index]) {
payload++;
write(fd, &payload, sizeof(payload));
printf("Logged %d", payload);
} else {
printf("Event is not being traced currently, enable via:\n");
printf("echo 1 > /sys/kernel/debug/tracing/events/udiag/testevent/enable");
}
goto fetch;
return 0;
}Thanks,
-Beau