tracing: Support poll on event hist file

PATCH In review Masami Hiramatsu (Google) 1 participant 6 messages Dec 27, 2024 – Dec 29, 2024

Subsystems (2)

Kselftest Tracing

lore

Share & tools

lore

Masami Hiramatsu (Google)

author maintainer

2024-12-27 04:07 UTC #0 lore raw

v7 cover tracing: Support poll on event hist file

Hi,

Here is the 7th version of the series to support polling on event 'hist' file. The previous version is here;

https://lore.kernel.org/all/172907575534.470540.12941248697563459082.stgit@mhiramat.roam.corp.google.com/

This version updates descriptions, use guard() for mutex and fixes selftest problem.

Background
----------
There has been interest in allowing user programs to monitor kernel events in real time. Ftrace provides trace_pipe interface to wait on events in the ring buffer, but it is needed to wait until filling up a page with events in the ring buffer. We can also peek the trace file periodically, but that is inefficient way to monitor a randomely happening event.

Overview
--------
This patch set allows user to poll(or select, epoll) on event histogram interface. As you know each event has its own hist file which shows histograms generated by trigger action. So user can set a new hist trigger on any event you want to monitor, and poll on the hist file until it is updated.

There are 2 poll events are supported, POLLIN and POLLPRI. POLLIN means that there are any readable update on hist file and this event will be flashed only when you call read(). So, this is useful if you want to read the histogram periodically. The other POLLPRI event is for monitoring trace event. Like the POLLIN, this will be returned when the histogram is updated, but you don't need to read() the file and use poll() again.

Note that this waits for histogram update (not event arrival), thus you must set a histogram on the event at first.

Usage
-----
Here is an example usage:

----
TRACEFS=/sys/kernel/tracing
EVENT=$TRACEFS/events/sched/sched_process_free

# setup histogram trigger and enable event
echo "hist:key=comm" >> $EVENT/trigger
echo 1 > $EVENT/enable

# Wait for update
poll pri $EVENT/hist

# Event arrived.
echo "process free event is comming"
tail $TRACEFS/trace
----

The 'poll' command is in the selftest patch.

You can take this series also from here;

https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/event-hist-poll

Thank you,

---

Masami Hiramatsu (Google) (3):
tracing/hist: Add poll(POLLIN) support on hist file tracing/hist: Support POLLPRI event for poll on histogram selftests/tracing: Add hist poll() support test

 include/linux/trace_events.h                       |   14 +++
 kernel/trace/trace_events.c                        |   14 +++
 kernel/trace/trace_events_hist.c                   |   92 +++++++++++++++++++-
 tools/testing/selftests/ftrace/Makefile            |    2 
 tools/testing/selftests/ftrace/poll.c              |   74 ++++++++++++++++
 .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   74 ++++++++++++++++
 6 files changed, 267 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/poll.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc

--
Masami Hiramatsu (Google) <[email protected]>

Masami Hiramatsu (Google)

author maintainer

2024-12-27 04:08 UTC #1 lore raw

PATCH v7 1/3 tracing/hist: Add poll(POLLIN) support on hist file

From: Masami Hiramatsu (Google) <[email protected]>

Add poll syscall support on the hist file. The Waiter will be waken up when the histogram is updated with POLLIN.

Currently, there is no way to wait for a specific event in userspace. So user needs to peek the trace periodicaly, or wait on trace_pipe. But it is not a good idea to peek at the trace for an event that randomly happens. And trace_pipe is not coming back until a page is filled with events.

This allows a user to wait for a specific event on the hist file. User can set a histogram trigger on the event which they want to monitor and poll() on its hist file. Since this poll() returns POLLIN, the next poll() will return soon unless a read() happens on that hist file.

NOTE: To read the hist file again, you must set the file offset to 0, but just for monitoring the event, you may not need to read the histogram.

Signed-off-by Masami Hiramatsu (Google) <[email protected]>

Reviewed-by Tom Zanussi <[email protected]>

---
Changes in v7:
- Reword description according to Steve's comment.
- Use guard() for mutex.

---
 include/linux/trace_events.h     |   14 ++++++++
 kernel/trace/trace_events.c      |   14 ++++++++
 kernel/trace/trace_events_hist.c |   69 ++++++++++++++++++++++++++++++++++++--
 3 files changed, 94 insertions(+), 3 deletions(-)

include/linux/trace_events.h +14 -0

index 2a5df5b62cfc..47caca4e7ff4 100644
@@ -669,6 +669,20 @@ struct trace_event_file {
 	atomic_t		tm_ref;	/* trigger-mode reference counter */
 };
 
+#ifdef CONFIG_HIST_TRIGGERS
+extern struct irq_work hist_poll_work;
+extern wait_queue_head_t hist_poll_wq;
+
+static inline void hist_poll_wakeup(void)
+{
+	if (wq_has_sleeper(&hist_poll_wq))
+		irq_work_queue(&hist_poll_work);
+}
+
+#define hist_poll_wait(file, wait)	\
+	poll_wait(file, &hist_poll_wq, wait)
+#endif
+
 #define __TRACE_EVENT_FLAGS(name, value)				\
 	static int __init trace_init_flags_##name(void)			\
 	{								\

kernel/trace/trace_events.c +14 -0

index 77e68efbd43e..e34b4a88d8ec 100644
@@ -2972,6 +2972,20 @@ static bool event_in_systems(struct trace_event_call *call,
 	return !*p || isspace(*p) || *p == ',';
 }
 
+#ifdef CONFIG_HIST_TRIGGERS
+/*
+ * Wake up waiter on the hist_poll_wq from irq_work because the hist trigger
+ * may happen in any context.
+ */
+static void hist_poll_event_irq_work(struct irq_work *work)
+{
+	wake_up_all(&hist_poll_wq);
+}
+
+DEFINE_IRQ_WORK(hist_poll_work, hist_poll_event_irq_work);
+DECLARE_WAIT_QUEUE_HEAD(hist_poll_wq);
+#endif
+
 static struct trace_event_file *
 trace_create_new_event(struct trace_event_call *call,
 		       struct trace_array *tr)

kernel/trace/trace_events_hist.c +66 -3

117 lines not on this page · raw message

Masami Hiramatsu (Google)

author maintainer

2024-12-27 04:08 UTC #2 lore raw

PATCH v7 2/3 tracing/hist: Support POLLPRI event for poll on histogram

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Since POLLIN will not be flushed until the hist file is read, the user needs to repeatedly read() and poll() on the hist file for monitoring the event continuously. But the read() is somewhat redundant when the user is only monitoring for event updates.

Add POLLPRI poll event on the hist file so the event returns when a histogram is updated after open(), poll() or read(). Thus it is possible to wait for the next event without having to issue a read().

Signed-off-by Masami Hiramatsu (Google) <[email protected]>

Reviewed-by Tom Zanussi <[email protected]>

---
Changes in v7:
- Reword description according to Steve's comment.
- Use guard() in hist_poll_open().

---
 kernel/trace/trace_events_hist.c |   29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)

kernel/trace/trace_events_hist.c +26 -3

index e169f67c6088..15f76c681c6c 100644
@@ -5595,6 +5595,7 @@ static void hist_trigger_show(struct seq_file *m,
 struct hist_file_data {
 	struct file *file;
 	u64 last_read;
+	u64 last_act;
 };
 
 static u64 get_hist_hit_count(struct trace_event_file *event_file)

@@ -5632,6 +5633,11 @@ static int hist_show(struct seq_file *m, void *v)
 			hist_trigger_show(m, data, n++);
 	}
 	hist_file->last_read = get_hist_hit_count(event_file);
+	/*
+	 * Update last_act too so that poll()/POLLPRI can wait for the next
+	 * event after any syscall on hist file.
+	 */
+	hist_file->last_act = hist_file->last_read;
 
  out_unlock:
 	mutex_unlock(&event_mutex);

@@ -5644,6 +5650,8 @@ static __poll_t event_hist_poll(struct file *file, struct poll_table_struct *wai
 	struct trace_event_file *event_file;
 	struct seq_file *m = file->private_data;
 	struct hist_file_data *hist_file = m->private;
+	__poll_t ret = 0;
+	u64 cnt;
 
 	guard(mutex)(&event_mutex);

@@ -5653,10 +5661,15 @@ static __poll_t event_hist_poll(struct file *file, struct poll_table_struct *wai
 
 	hist_poll_wait(file, wait);
 
-	if (hist_file->last_read != get_hist_hit_count(event_file))
-		return EPOLLIN | EPOLLRDNORM;
+	cnt = get_hist_hit_count(event_file);
+	if (hist_file->last_read != cnt)
+		ret |= EPOLLIN | EPOLLRDNORM;
+	if (hist_file->last_act != cnt) {
+		hist_file->last_act = cnt;
+		ret |= EPOLLPRI;
+	}
 
-	return 0;
+	return ret;
 }
 
 static int event_hist_release(struct inode *inode, struct file *file)

@@ -5670,6 +5683,7 @@ static int event_hist_release(struct inode *inode, struct file *file)
 
 static int event_hist_open(struct inode *inode, struct file *file)
 {
+	struct trace_event_file *event_file;
 	struct hist_file_data *hist_file;
 	int ret;

@@ -5677,16 +5691,25 @@ static int event_hist_open(struct inode *inode, struct file *file)
 	if (ret)
 		return ret;
 
+	guard(mutex)(&event_mutex);
+
+	event_file = event_file_data(file);
+	if (!event_file)
+		return -ENODEV;
+
 	hist_file = kzalloc(sizeof(*hist_file), GFP_KERNEL);
 	if (!hist_file)
 		return -ENOMEM;
+
 	hist_file->file = file;
+	hist_file->last_act = get_hist_hit_count(event_file);
 
 	/* Clear private_data to avoid warning in single_open() */
 	file->private_data = NULL;
 	ret = single_open(file, hist_show, hist_file);
 	if (ret)
 		kfree(hist_file);
+
 	return ret;
 }

Masami Hiramatsu (Google)

author maintainer

2024-12-27 04:08 UTC #3 lore raw

PATCH v7 3/3 selftests/tracing: Add hist poll() support test

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add a testcase for poll() on hist file. This introduces a helper binary to the ftracetest, because there is no good way to reliably execute poll() on hist file.

Signed-off-by Masami Hiramatsu (Google) <[email protected]>

Reviewed-by Shuah Khan <[email protected]>

---
Changes in v7:
- Add comm histogram filter for specifying test sleep command.

---
 tools/testing/selftests/ftrace/Makefile            |    2 +
 tools/testing/selftests/ftrace/poll.c              |   74 ++++++++++++++++++++
 .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   74 ++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/poll.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc

tools/testing/selftests/ftrace/Makefile +2 -0

index a1e955d2de4c..49d96bb16355 100644
@@ -6,4 +6,6 @@ TEST_PROGS := ftracetest-ktap
 TEST_FILES := test.d settings
 EXTRA_CLEAN := $(OUTPUT)/logs/*
 
+TEST_GEN_PROGS = poll
+
 include ../lib.mk

tools/testing/selftests/ftrace/poll.c new file +74 -0

74 lines not on this page · raw message

tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc new file +74 -0

74 lines not on this page · raw message

Masami Hiramatsu (Google)

author maintainer

2024-12-29 13:09 UTC #4 lore raw

Re: [PATCH v7 3/3] selftests/tracing: Add hist poll() support test

Quoted 11 lines · On Fri, 27 Dec 2024 13:08:18 +0900 "Masami Hiramatsu (Google)"…

From: Masami Hiramatsu (Google) <[email protected]>
Add a testcase for poll() on hist file. This introduces a helper binary to the ftracetest, because there is no good way to reliably execute poll() on hist file.
Signed-off-by Masami Hiramatsu (Google) <[email protected]>
Reviewed-by Shuah Khan <[email protected]>
---
Changes in v7:
- Add comm histogram filter for specifying test sleep command.

I found I forgot to refresh the patch. (just not committed)

Sorry.

Quoted 180 lines

---
 tools/testing/selftests/ftrace/Makefile            |    2 +
 tools/testing/selftests/ftrace/poll.c              |   74 ++++++++++++++++++++
 .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   74 ++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/poll.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc

tools/testing/selftests/ftrace/Makefile +2 -0

index a1e955d2de4c..49d96bb16355 100644
@@ -6,4 +6,6 @@ TEST_PROGS := ftracetest-ktap
 TEST_FILES := test.d settings
 EXTRA_CLEAN := $(OUTPUT)/logs/*
 
+TEST_GEN_PROGS = poll
+
 include ../lib.mk

tools/testing/selftests/ftrace/poll.c new file +74 -0

new file mode 100644
index 000000000000..53258f7515e7
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Simple poll on a file.
+ *
+ * Copyright (c) 2024 Google LLC.
+ */
+
+#include <errno.h>
+#include <fcntl.h>
+#include <poll.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+
+#define BUFSIZE 4096
+
+/*
+ * Usage:
+ *  poll [-I|-P] [-t timeout] FILE
+ */
+int main(int argc, char *argv[])
+{
+	struct pollfd pfd = {.events = POLLIN};
+	char buf[BUFSIZE];
+	int timeout = -1;
+	int ret, opt;
+
+	while ((opt = getopt(argc, argv, "IPt:")) != -1) {
+		switch (opt) {
+		case 'I':
+			pfd.events = POLLIN;
+			break;
+		case 'P':
+			pfd.events = POLLPRI;
+			break;
+		case 't':
+			timeout = atoi(optarg);
+			break;
+		default:
+			fprintf(stderr, "Usage: %s [-I|-P] [-t timeout] FILE\n",
+				argv[0]);
+			return -1;
+		}
+	}
+	if (optind >= argc) {
+		fprintf(stderr, "Error: Polling file is not specified\n");
+		return -1;
+	}
+
+	pfd.fd = open(argv[optind], O_RDONLY);
+	if (pfd.fd < 0) {
+		fprintf(stderr, "failed to open %s", argv[optind]);
+		perror("open");
+		return -1;
+	}
+
+	/* Reset poll by read if POLLIN is specified. */
+	if (pfd.events & POLLIN)
+		do {} while (read(pfd.fd, buf, BUFSIZE) == BUFSIZE);
+
+	ret = poll(&pfd, 1, timeout);
+	if (ret < 0 && errno != EINTR) {
+		perror("poll");
+		return -1;
+	}
+	close(pfd.fd);
+
+	/* If timeout happned (ret == 0), exit code is 1 */
+	if (ret == 0)
+		return 1;
+
+	return 0;
+}

tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc new file +74 -0

new file mode 100644
index 000000000000..cbd01a71ecad
@@ -0,0 +1,74 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event trigger - test poll wait on histogram
+# requires: set_event events/sched/sched_process_free/trigger events/sched/sched_process_free/hist
+# flags: instance
+
+POLL=${FTRACETEST_ROOT}/poll
+
+if [ ! -x ${POLL} ]; then
+  echo "poll program is not compiled!"
+  exit_unresolved
+fi
+
+EVENT=events/sched/sched_process_free/
+
+# Check poll ops is supported. Before implementing poll on hist file, it
+# returns soon with POLLIN | POLLOUT, but not POLLPRI.
+
+# This must wait >1 sec and return 1 (timeout).
+set +e
+${POLL} -I -t 1000 ${EVENT}/hist
+ret=$?
+set -e
+if [ ${ret} != 1 ]; then
+  echo "poll on hist file is not supported"
+  exit_unsupported
+fi
+
+# Test POLLIN
+echo > trace
+echo "hist:key=comm" > ${EVENT}/trigger
+echo 1 > ${EVENT}/enable
+
+# This sleep command will exit after 2 seconds.
+sleep 2 &
+BGPID=$!
+# if timeout happens, poll returns 1.
+${POLL} -I -t 4000 ${EVENT}/hist
+echo 0 > tracing_on
+
+if [ -d /proc/${BGPID} ]; then
+  echo "poll exits too soon"
+  kill -KILL ${BGPID} ||:
+  exit_fail
+fi
+
+if ! grep -qw "sleep" trace; then
+  echo "poll exits before event happens"
+  exit_fail
+fi
+
+# Test POLLPRI
+echo > trace
+echo 1 > tracing_on
+
+# This sleep command will exit after 2 seconds.
+sleep 2 &
+BGPID=$!
+# if timeout happens, poll returns 1.
+${POLL} -P -t 4000 ${EVENT}/hist
+echo 0 > tracing_on
+
+if [ -d /proc/${BGPID} ]; then
+  echo "poll exits too soon"
+  kill -KILL ${BGPID} ||:
+  exit_fail
+fi
+
+if ! grep -qw "sleep" trace; then
+  echo "poll exits before event happens"
+  exit_fail
+fi
+
+exit_pass

Signature · 1 line

Masami Hiramatsu (Google) <[email protected]>

Masami Hiramatsu (Google)

author maintainer

2024-12-29 13:24 UTC #5 lore raw

PATCH v7 3/3 selftests/tracing: Add hist poll() support test

In reply to Masami Hiramatsu (Google)

From: Masami Hiramatsu (Google) <[email protected]>

Add a testcase for poll() on hist file. This introduces a helper binary to the ftracetest, because there is no good way to reliably execute poll() on hist file.

Signed-off-by Masami Hiramatsu (Google) <[email protected]>

Reviewed-by Shuah Khan <[email protected]>

---
Changes in v7:
- Add comm histogram filter for specifying test sleep command.

---
 tools/testing/selftests/ftrace/Makefile            |    2 +
 tools/testing/selftests/ftrace/poll.c              |   74 ++++++++++++++++++++
 .../ftrace/test.d/trigger/trigger-hist-poll.tc     |   74 ++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/poll.c
 create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc

tools/testing/selftests/ftrace/Makefile +2 -0

index a1e955d2de4c..49d96bb16355 100644
@@ -6,4 +6,6 @@ TEST_PROGS := ftracetest-ktap
 TEST_FILES := test.d settings
 EXTRA_CLEAN := $(OUTPUT)/logs/*
 
+TEST_GEN_PROGS = poll
+
 include ../lib.mk

tools/testing/selftests/ftrace/poll.c new file +74 -0

74 lines not on this page · raw message

tools/testing/selftests/ftrace/test.d/trigger/trigger-hist-poll.tc new file +74 -0

74 lines not on this page · raw message