Skip to contentSkip to conversation

perf record: Fix event fd races

PATCH Applied Ian Rogers 3 participants 5 messages Oct 24, 2022 perf events

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

PATCH 1/1 perf record: Fix event fd races

The write call may set errno which is problematic if occurring in a function also setting errno. Save and restore errno around the write call.

done_fd may be used after close, clear it as part of the close and check its validity in the signal handler.

---
 tools/perf/builtin-record.c | 41 ++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 16 deletions(-)
tools/perf/builtin-record.c +25 -16
index 52d254b1530c..e128b855ddde 100644
@@ -649,7 +649,7 @@ static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size)
 static volatile int signr = -1;
 static volatile int child_finished;
 #ifdef HAVE_EVENTFD_SUPPORT
-static int done_fd = -1;
+static volatile int done_fd = -1;
 #endif
 
 static void sig_handler(int sig)
@@ -661,19 +661,24 @@ static void sig_handler(int sig)
 
 	done = 1;
 #ifdef HAVE_EVENTFD_SUPPORT
-{
-	u64 tmp = 1;
-	/*
-	 * It is possible for this signal handler to run after done is checked
-	 * in the main loop, but before the perf counter fds are polled. If this
-	 * happens, the poll() will continue to wait even though done is set,
-	 * and will only break out if either another signal is received, or the
-	 * counters are ready for read. To ensure the poll() doesn't sleep when
-	 * done is set, use an eventfd (done_fd) to wake up the poll().
-	 */
-	if (write(done_fd, &tmp, sizeof(tmp)) < 0)
-		pr_err("failed to signal wakeup fd, error: %m\n");
-}
+	if (done_fd >= 0) {
+		u64 tmp = 1;
+		int orig_errno = errno;
+
+		/*
+		 * It is possible for this signal handler to run after done is
+		 * checked in the main loop, but before the perf counter fds are
+		 * polled. If this happens, the poll() will continue to wait
+		 * even though done is set, and will only break out if either
+		 * another signal is received, or the counters are ready for
+		 * read. To ensure the poll() doesn't sleep when done is set,
+		 * use an eventfd (done_fd) to wake up the poll().
+		 */
+		if (write(done_fd, &tmp, sizeof(tmp)) < 0)
+			pr_err("failed to signal wakeup fd, error: %m\n");
+
+		errno = orig_errno;
+	}
 #endif // HAVE_EVENTFD_SUPPORT
 }
 
@@ -2834,8 +2839,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 out_delete_session:
 #ifdef HAVE_EVENTFD_SUPPORT
-	if (done_fd >= 0)
-		close(done_fd);
+	if (done_fd >= 0) {
+		fd = done_fd;
+		done_fd = -1;
+
+		close(fd);
+	}
 #endif
 	zstd_fini(&session->zstd_data);
 	perf_session__delete(session);
Signature · 1 line
2.38.0.135.g90850a2211-goog
#1 lore raw

Hi Ian,

Quoted 18 lines · Ian Rogers

The write call may set errno which is problematic if occurring in a function also setting errno. Save and restore errno around the write call.

done_fd may be used after close, clear it as part of the close and check its validity in the signal handler.

---
 tools/perf/builtin-record.c | 41 ++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 16 deletions(-)
tools/perf/builtin-record.c +0 -0
index 52d254b1530c..e128b855ddde 100644

tools/perf/builtin-record.c static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size) · line 652

Quoted 5 lines
tools/perf/builtin-record.c +1 -1
@@ -649,7 +649,7 @@ static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size)
 static volatile int signr = -1;
 static volatile int child_finished;
 #ifdef HAVE_EVENTFD_SUPPORT
-static int done_fd = -1;
+static volatile int done_fd = -1;

Here is a bit suspecious for adding volatile qualifier. See the
document: process/volatile-considered-harmful.rst.

I know the document is mainly for kernel programming, but seems to me it's also valid for C programming in userspace.

I not sure what's the purpose for adding volatile for done_fd, if we really have concern for reading any stale value for done_fd, should we use WRITE_ONCE/READ_ONCE?

The rest changes look good to me.

Thanks,
Leo

Quoted 59 lines
tools/perf/builtin-record.c +24 -15
 #endif
 
 static void sig_handler(int sig)
@@ -661,19 +661,24 @@ static void sig_handler(int sig)
 
 	done = 1;
 #ifdef HAVE_EVENTFD_SUPPORT
-{
-	u64 tmp = 1;
-	/*
-	 * It is possible for this signal handler to run after done is checked
-	 * in the main loop, but before the perf counter fds are polled. If this
-	 * happens, the poll() will continue to wait even though done is set,
-	 * and will only break out if either another signal is received, or the
-	 * counters are ready for read. To ensure the poll() doesn't sleep when
-	 * done is set, use an eventfd (done_fd) to wake up the poll().
-	 */
-	if (write(done_fd, &tmp, sizeof(tmp)) < 0)
-		pr_err("failed to signal wakeup fd, error: %m\n");
-}
+	if (done_fd >= 0) {
+		u64 tmp = 1;
+		int orig_errno = errno;
+
+		/*
+		 * It is possible for this signal handler to run after done is
+		 * checked in the main loop, but before the perf counter fds are
+		 * polled. If this happens, the poll() will continue to wait
+		 * even though done is set, and will only break out if either
+		 * another signal is received, or the counters are ready for
+		 * read. To ensure the poll() doesn't sleep when done is set,
+		 * use an eventfd (done_fd) to wake up the poll().
+		 */
+		if (write(done_fd, &tmp, sizeof(tmp)) < 0)
+			pr_err("failed to signal wakeup fd, error: %m\n");
+
+		errno = orig_errno;
+	}
 #endif // HAVE_EVENTFD_SUPPORT
 }
 
@@ -2834,8 +2839,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 out_delete_session:
 #ifdef HAVE_EVENTFD_SUPPORT
-	if (done_fd >= 0)
-		close(done_fd);
+	if (done_fd >= 0) {
+		fd = done_fd;
+		done_fd = -1;
+
+		close(fd);
+	}
 #endif
 	zstd_fini(&session->zstd_data);
 	perf_session__delete(session);
Signature · 1 line
2.38.0.135.g90850a2211-goog
author reviewer
#2 lore raw
Quoted 31 lines · Leo Yan

Hi Ian,

Quoted 18 lines · Ian Rogers

The write call may set errno which is problematic if occurring in a function also setting errno. Save and restore errno around the write call.

done_fd may be used after close, clear it as part of the close and check its validity in the signal handler.

---
 tools/perf/builtin-record.c | 41 ++++++++++++++++++++++---------------
 1 file changed, 25 insertions(+), 16 deletions(-)
tools/perf/builtin-record.c +0 -0
index 52d254b1530c..e128b855ddde 100644

tools/perf/builtin-record.c static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size) · line 652

Quoted 5 lines
tools/perf/builtin-record.c +1 -1
@@ -649,7 +649,7 @@ static int record__pushfn(struct mmap *map, void *to, void *bf, size_t size)
 static volatile int signr = -1;
 static volatile int child_finished;
 #ifdef HAVE_EVENTFD_SUPPORT
-static int done_fd = -1;
+static volatile int done_fd = -1;

Here is a bit suspecious for adding volatile qualifier. See the
document: process/volatile-considered-harmful.rst.

Quoted 6 lines

I know the document is mainly for kernel programming, but seems to me it's also valid for C programming in userspace.

I not sure what's the purpose for adding volatile for done_fd, if we really have concern for reading any stale value for done_fd, should we use WRITE_ONCE/READ_ONCE?

We could just switch to C11 and stdatomic. The volatile is consistent with the code above and more consistent with the expectation of writing to a variable that is read in a signal handler.

Thanks,
Ian

Quoted 64 lines

The rest changes look good to me.

Thanks,
Leo

Quoted 59 lines
tools/perf/builtin-record.c +24 -16
 #endif
 
 static void sig_handler(int sig)
@@ -661,19 +661,24 @@ static void sig_handler(int sig)
 
      done = 1;
 #ifdef HAVE_EVENTFD_SUPPORT
-{
-     u64 tmp = 1;
-     /*
-      * It is possible for this signal handler to run after done is checked
-      * in the main loop, but before the perf counter fds are polled. If this
-      * happens, the poll() will continue to wait even though done is set,
-      * and will only break out if either another signal is received, or the
-      * counters are ready for read. To ensure the poll() doesn't sleep when
-      * done is set, use an eventfd (done_fd) to wake up the poll().
-      */
-     if (write(done_fd, &tmp, sizeof(tmp)) < 0)
-             pr_err("failed to signal wakeup fd, error: %m\n");
-}
+     if (done_fd >= 0) {
+             u64 tmp = 1;
+             int orig_errno = errno;
+
+             /*
+              * It is possible for this signal handler to run after done is
+              * checked in the main loop, but before the perf counter fds are
+              * polled. If this happens, the poll() will continue to wait
+              * even though done is set, and will only break out if either
+              * another signal is received, or the counters are ready for
+              * read. To ensure the poll() doesn't sleep when done is set,
+              * use an eventfd (done_fd) to wake up the poll().
+              */
+             if (write(done_fd, &tmp, sizeof(tmp)) < 0)
+                     pr_err("failed to signal wakeup fd, error: %m\n");
+
+             errno = orig_errno;
+     }
 #endif // HAVE_EVENTFD_SUPPORT
 }
 
@@ -2834,8 +2839,12 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 out_delete_session:
 #ifdef HAVE_EVENTFD_SUPPORT
-     if (done_fd >= 0)
-             close(done_fd);
+     if (done_fd >= 0) {
+             fd = done_fd;
+             done_fd = -1;
+
+             close(fd);
+     }
 #endif
      zstd_fini(&session->zstd_data);
      perf_session__delete(session);
--

2.38.0.135.g90850a2211-goog

Reviewed-by
#3 lore raw

On Sun, Oct 23, 2022 at 10:33:30PM -0700, Ian Rogers wrote:

[...]

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

+static volatile int done_fd = -1;

Here is a bit suspecious for adding volatile qualifier. See the
document: process/volatile-considered-harmful.rst.

I know the document is mainly for kernel programming, but seems to me it's also valid for C programming in userspace.

Quoted 7 lines
Quoted 3 lines

I not sure what's the purpose for adding volatile for done_fd, if we really have concern for reading any stale value for done_fd, should we use WRITE_ONCE/READ_ONCE?

We could just switch to C11 and stdatomic. The volatile is consistent with the code above and more consistent with the expectation of writing to a variable that is read in a signal handler.

Thanks for the info for C11 and stdatomic.h. The documentation [1] says the safe way is for accessing shared data in signal handler is:

  static volatile sig_atomic_t done_fd = -1;

It's fine if you want to use another patch to address this issue, this patch for fixing errno is fine for me:

[1] https://wiki.sei.cmu.edu/confluence/display/c/SIG31-C.+Do+not+access+shared+objects+in+signal+handlers

maintainer Applied
#4 lore raw

Em Mon, Oct 24, 2022 at 05:16:56PM +0800, Leo Yan escreveu:

Quoted 23 lines · Ian Rogers

[...]

Quoted 8 lines
Quoted 8 lines
Quoted 1 line

+static volatile int done_fd = -1;

Here is a bit suspecious for adding volatile qualifier. See the
document: process/volatile-considered-harmful.rst.

I know the document is mainly for kernel programming, but seems to me it's also valid for C programming in userspace.

Quoted 7 lines
Quoted 3 lines

I not sure what's the purpose for adding volatile for done_fd, if we really have concern for reading any stale value for done_fd, should we use WRITE_ONCE/READ_ONCE?

We could just switch to C11 and stdatomic. The volatile is consistent with the code above and more consistent with the expectation of writing to a variable that is read in a signal handler.

Thanks for the info for C11 and stdatomic.h. The documentation [1] says the safe way is for accessing shared data in signal handler is:

Quoted 6 lines
  static volatile sig_atomic_t done_fd = -1;

It's fine if you want to use another patch to address this issue, this patch for fixing errno is fine for me:

Thanks, applied.

- Arnaldo

Quoted 1 line

[1] https://wiki.sei.cmu.edu/confluence/display/c/SIG31-C.+Do+not+access+shared+objects+in+signal+handlers

Signature · 1 line
- Arnaldo