Skip to contentSkip to conversation

perf test: Handle fd gaps in test__dso_data_reopen

PATCH Applied Eirik Fuller 4 participants 5 messages Jun 26, 2021 – Aug 2, 2021 perf events

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf test: Handle fd gaps in test__dso_data_reopen

https://github.com/beaker-project/restraint/issues/215 describes a file descriptor leak which revealed the test failure described here.

The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the number of open file descriptors, but it actually limits newly opened file descriptors. When the file descriptor limit is reduced, file descriptors already open remain open regardless of the new limit. This test failure does not occur if open file descriptors are contiguous, beginning at zero.

The following command triggers this perf test failure.

perf test 'DSO data reopen' 3>/dev/null 8>/dev/null

This patch determines the file descriptor limit by opening four files and then closing them. The limit is set to the fourth file descriptor, leaving only the first three available because any newly opened file descriptor must be less than the limit.

---
 tools/perf/tests/dso-data.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tools/perf/tests/dso-data.c +12 -2
index 627c1aaf1c9e..43e1b01e5afc 100644
@@ -308,10 +308,20 @@ int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u
 	return 0;
 }
 
+static long new_limit(int count)
+{
+	int fd = open("/dev/null", O_RDONLY);
+	long ret = fd;
+	if (count > 0)
+		ret = new_limit(--count);
+	close(fd);
+	return ret;
+}
+
 int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
 {
 	struct machine machine;
-	long nr_end, nr = open_files_cnt();
+	long nr_end, nr = open_files_cnt(), lim = new_limit(3);
 	int fd, fd_extra;
 
 #define dso_0 (dsos[0])
@@ -334,7 +344,7 @@ int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_
 
 	/* Make sure we are able to open 3 fds anyway */
 	TEST_ASSERT_VAL("failed to set file limit",
-			!set_fd_limit((nr + 3)));
+			!set_fd_limit((lim)));
 
 	TEST_ASSERT_VAL("failed to create dsos\n", !dsos__create(3, TEST_FILE_SIZE));
 
Signature · 1 line
2.27.0
Acked-by on v1 1/1
#1 lore raw
Quoted 31 lines · Eirik Fuller

https://github.com/beaker-project/restraint/issues/215 describes a file descriptor leak which revealed the test failure described here.

The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the number of open file descriptors, but it actually limits newly opened file descriptors. When the file descriptor limit is reduced, file descriptors already open remain open regardless of the new limit. This test failure does not occur if open file descriptors are contiguous, beginning at zero.

The following command triggers this perf test failure.

perf test 'DSO data reopen' 3>/dev/null 8>/dev/null

This patch determines the file descriptor limit by opening four files and then closing them. The limit is set to the fourth file descriptor, leaving only the first three available because any newly opened file descriptor must be less than the limit.

---
 tools/perf/tests/dso-data.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tools/perf/tests/dso-data.c +0 -0
index 627c1aaf1c9e..43e1b01e5afc 100644
@@ -308,10 +308,20 @@ int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u
 	return 0;
 }

tools/perf/tests/dso-data.c int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u · line 319

Quoted 10 lines
tools/perf/tests/dso-data.c +9 -0
 
+static long new_limit(int count)
+{
+	int fd = open("/dev/null", O_RDONLY);
+	long ret = fd;
+	if (count > 0)
+		ret = new_limit(--count);
+	close(fd);
+	return ret;
+}

nice idea.. thanks

jirka

Quoted 21 lines
tools/perf/tests/dso-data.c +3 -2
+
 int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
 {
 	struct machine machine;
-	long nr_end, nr = open_files_cnt();
+	long nr_end, nr = open_files_cnt(), lim = new_limit(3);
 	int fd, fd_extra;
 
 #define dso_0 (dsos[0])
@@ -334,7 +344,7 @@ int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_
 
 	/* Make sure we are able to open 3 fds anyway */
 	TEST_ASSERT_VAL("failed to set file limit",
-			!set_fd_limit((nr + 3)));
+			!set_fd_limit((lim)));
 
 	TEST_ASSERT_VAL("failed to create dsos\n", !dsos__create(3, TEST_FILE_SIZE));
 
Signature · 1 line
2.27.0
Acked-by on v1 1/1
#2 lore raw
In reply to Eirik Fuller
Quoted 12 lines · Eirik Fuller

https://github.com/beaker-project/restraint/issues/215 describes a file descriptor leak which revealed the test failure described here.

The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the number of open file descriptors, but it actually limits newly opened file descriptors. When the file descriptor limit is reduced, file descriptors already open remain open regardless of the new limit. This test failure does not occur if open file descriptors are contiguous, beginning at zero.

The following command triggers this perf test failure.

Quoted 8 lines

perf test 'DSO data reopen' 3>/dev/null 8>/dev/null

This patch determines the file descriptor limit by opening four files and then closing them. The limit is set to the fourth file descriptor, leaving only the first three available because any newly opened file descriptor must be less than the limit.

Quoted 43 lines
---
 tools/perf/tests/dso-data.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tools/perf/tests/dso-data.c +12 -2
index 627c1aaf1c9e..43e1b01e5afc 100644
@@ -308,10 +308,20 @@ int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u
 	return 0;
 }
 
+static long new_limit(int count)
+{
+	int fd = open("/dev/null", O_RDONLY);
+	long ret = fd;
+	if (count > 0)
+		ret = new_limit(--count);
+	close(fd);
+	return ret;
+}
+
 int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
 {
 	struct machine machine;
-	long nr_end, nr = open_files_cnt();
+	long nr_end, nr = open_files_cnt(), lim = new_limit(3);
 	int fd, fd_extra;
 
 #define dso_0 (dsos[0])
@@ -334,7 +344,7 @@ int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_
 
 	/* Make sure we are able to open 3 fds anyway */
 	TEST_ASSERT_VAL("failed to set file limit",
-			!set_fd_limit((nr + 3)));
+			!set_fd_limit((lim)));
 
 	TEST_ASSERT_VAL("failed to create dsos\n", !dsos__create(3, TEST_FILE_SIZE));
 
Signature · 1 line
2.27.0
#3 lore raw
Quoted 15 lines · Michael Petlan
Quoted 14 lines · Eirik Fuller

https://github.com/beaker-project/restraint/issues/215 describes a file descriptor leak which revealed the test failure described here.

The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the number of open file descriptors, but it actually limits newly opened file descriptors. When the file descriptor limit is reduced, file descriptors already open remain open regardless of the new limit. This test failure does not occur if open file descriptors are contiguous, beginning at zero.

The following command triggers this perf test failure.

perf test 'DSO data reopen' 3>/dev/null 8>/dev/null

Quoted 8 lines
Quoted 6 lines

This patch determines the file descriptor limit by opening four files and then closing them. The limit is set to the fourth file descriptor, leaving only the first three available because any newly opened file descriptor must be less than the limit.

Hi Arnaldo,
kindly reminder, could this get merged?

Thank you.

Quoted 46 lines
Quoted 44 lines
---
 tools/perf/tests/dso-data.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tools/perf/tests/dso-data.c +12 -2
index 627c1aaf1c9e..43e1b01e5afc 100644
@@ -308,10 +308,20 @@ int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u
 	return 0;
 }
 
+static long new_limit(int count)
+{
+	int fd = open("/dev/null", O_RDONLY);
+	long ret = fd;
+	if (count > 0)
+		ret = new_limit(--count);
+	close(fd);
+	return ret;
+}
+
 int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
 {
 	struct machine machine;
-	long nr_end, nr = open_files_cnt();
+	long nr_end, nr = open_files_cnt(), lim = new_limit(3);
 	int fd, fd_extra;
 
 #define dso_0 (dsos[0])
@@ -334,7 +344,7 @@ int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_
 
 	/* Make sure we are able to open 3 fds anyway */
 	TEST_ASSERT_VAL("failed to set file limit",
-			!set_fd_limit((nr + 3)));
+			!set_fd_limit((lim)));
 
 	TEST_ASSERT_VAL("failed to create dsos\n", !dsos__create(3, TEST_FILE_SIZE));
 
Signature · 1 line
2.27.0
maintainer Applied
#4 lore raw

Em Fri, Jul 30, 2021 at 02:19:58PM +0200, Michael Petlan escreveu:

Quoted 21 lines · Michael Petlan
Quoted 20 lines · Eirik Fuller
Quoted 19 lines

https://github.com/beaker-project/restraint/issues/215 describes a file descriptor leak which revealed the test failure described here.

The 'DSO data reopen' perf test assumes that RLIMIT_NOFILE limits the number of open file descriptors, but it actually limits newly opened file descriptors. When the file descriptor limit is reduced, file descriptors already open remain open regardless of the new limit. This test failure does not occur if open file descriptors are contiguous, beginning at zero.

The following command triggers this perf test failure.

perf test 'DSO data reopen' 3>/dev/null 8>/dev/null

This patch determines the file descriptor limit by opening four files and then closing them. The limit is set to the fourth file descriptor, leaving only the first three available because any newly opened file descriptor must be less than the limit.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Hi Arnaldo,
kindly reminder, could this get merged?

Thanks, applied. Sorry for the delay.

- Arnaldo

Quoted 49 lines

Thank you.

Quoted 46 lines
Quoted 44 lines
---
 tools/perf/tests/dso-data.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tools/perf/tests/dso-data.c +12 -2
index 627c1aaf1c9e..43e1b01e5afc 100644
@@ -308,10 +308,20 @@ int test__dso_data_cache(struct test *test __maybe_unused, int subtest __maybe_u
 	return 0;
 }
 
+static long new_limit(int count)
+{
+	int fd = open("/dev/null", O_RDONLY);
+	long ret = fd;
+	if (count > 0)
+		ret = new_limit(--count);
+	close(fd);
+	return ret;
+}
+
 int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_unused)
 {
 	struct machine machine;
-	long nr_end, nr = open_files_cnt();
+	long nr_end, nr = open_files_cnt(), lim = new_limit(3);
 	int fd, fd_extra;
 
 #define dso_0 (dsos[0])
@@ -334,7 +344,7 @@ int test__dso_data_reopen(struct test *test __maybe_unused, int subtest __maybe_
 
 	/* Make sure we are able to open 3 fds anyway */
 	TEST_ASSERT_VAL("failed to set file limit",
-			!set_fd_limit((nr + 3)));
+			!set_fd_limit((lim)));
 
 	TEST_ASSERT_VAL("failed to create dsos\n", !dsos__create(3, TEST_FILE_SIZE));
 
Signature · 1 line
2.27.0
Signature · 1 line
- Arnaldo