Skip to contentSkip to conversation

osdep: add a qemu_close_all_open_fd() helper

PATCH Superseded Clément Léger 4 participants 5 messages Jul 17, 2024 – Jul 23, 2024 Networking

v4 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 osdep: add a qemu_close_all_open_fd() helper

Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking.

----
v4:
- Add a comment saying that qemu_close_all_fds() can take a NULL skip array and nskip == 0
- Added an assert in qemu_close_all_fds() to check for skip/nskip parameters
- Fix spurious tabs instead of spaces
- Applied checkpatch
- v3: https://lore.kernel.org/qemu-devel/[email protected]/

v3:
- Use STD*_FILENO defines instead of raw values
- Fix indentation of close_all_fds_after_fork()
- Check for nksip in fallback code
- Check for path starting with a '.' in qemu_close_all_open_fd_proc()
- Use unsigned for cur_skip
- Move ifdefs inside close_fds functions rather than redefining them
- Remove uneeded 'if(nskip)' test
- Add comments to close_range version
- Reduce range of skip fd as we find them in
- v2: https://lore.kernel.org/qemu-devel/[email protected]/

v2:
- Factorize async_teardown.c close_fds implementation as well as tap.c ones
- Apply checkpatch
- v1: https://lore.kernel.org/qemu-devel/[email protected]/

---
 include/qemu/osdep.h    |   8 +++
 net/tap.c               |  33 +++++-----
 system/async-teardown.c |  37 +----------
 util/osdep.c            | 132 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 160 insertions(+), 50 deletions(-)
include/qemu/osdep.h +8 -0
index 191916f38e..43a035d756 100644
@@ -757,6 +757,14 @@ static inline void qemu_reset_optind(void)
 
 int qemu_fdatasync(int fd);
 
+/**
+ * Close all open file descriptors except the ones supplied in the @skip array
+ *
+ * @skip: ordered array of distinct file descriptors that should not be closed
+ * @nskip: number of entries in the @skip array.
+ */
+void qemu_close_all_open_fd(const int *skip, unsigned int nskip);
+
 /**
  * Sync changes made to the memory mapped file back to the backing
  * storage. For POSIX compliant systems this will fallback
net/tap.c +19 -14
index 51f7aec39d..45cb963142 100644
@@ -385,6 +385,23 @@ static TAPState *net_tap_fd_init(NetClientState *peer,
     return s;
 }
 
+static void close_all_fds_after_fork(int excluded_fd)
+{
+    const int skip_fd[] = {STDIN_FILENO, STDOUT_FILENO, STDERR_FILENO,
+                           excluded_fd};
+    unsigned int nskip = ARRAY_SIZE(skip_fd);
+
+    /*
+     * skip_fd must be an ordered array of distinct fds, exclude
+     * excluded_fd if already included in the [STDIN_FILENO - STDERR_FILENO]
+     * range
+     */
+    if (excluded_fd <= STDERR_FILENO) {
+        nskip--;
+    }
+    qemu_close_all_open_fd(skip_fd, nskip);
+}
+
 static void launch_script(const char *setup_script, const char *ifname,
                           int fd, Error **errp)
 {
@@ -400,13 +417,7 @@ static void launch_script(const char *setup_script, const char *ifname,
         return;
     }
     if (pid == 0) {
-        int open_max = sysconf(_SC_OPEN_MAX), i;
-
-        for (i = 3; i < open_max; i++) {
-            if (i != fd) {
-                close(i);
-            }
-        }
+        close_all_fds_after_fork(fd);
         parg = args;
         *parg++ = (char *)setup_script;
         *parg++ = (char *)ifname;
@@ -490,17 +501,11 @@ static int net_bridge_run_helper(const char *helper, const char *bridge,
         return -1;
     }
     if (pid == 0) {
-        int open_max = sysconf(_SC_OPEN_MAX), i;
         char *fd_buf = NULL;
         char *br_buf = NULL;
         char *helper_cmd = NULL;
 
-        for (i = 3; i < open_max; i++) {
-            if (i != sv[1]) {
-                close(i);
-            }
-        }
-
+        close_all_fds_after_fork(sv[1]);
         fd_buf = g_strdup_printf("%s%d", "--fd=", sv[1]);
 
         if (strrchr(helper, ' ') || strrchr(helper, '\t')) {
system/async-teardown.c +1 -36
index 396963c091..9148ee8d04 100644
@@ -26,40 +26,6 @@
 
 static pid_t the_ppid;
 
-/*
- * Close all open file descriptors.
- */
-static void close_all_open_fd(void)
-{
-    struct dirent *de;
-    int fd, dfd;
-    DIR *dir;
-
-#ifdef CONFIG_CLOSE_RANGE
-    int r = close_range(0, ~0U, 0);
-    if (!r) {
-        /* Success, no need to try other ways. */
-        return;
-    }
-#endif
-
-    dir = opendir("/proc/self/fd");
-    if (!dir) {
-        /* If /proc is not mounted, there is nothing that can be done. */
-        return;
-    }
-    /* Avoid closing the directory. */
-    dfd = dirfd(dir);
-
-    for (de = readdir(dir); de; de = readdir(dir)) {
-        fd = atoi(de->d_name);
-        if (fd != dfd) {
-            close(fd);
-        }
-    }
-    closedir(dir);
-}
-
 static void hup_handler(int signal)
 {
     /* Check every second if this process has been reparented. */
@@ -85,9 +51,8 @@ static int async_teardown_fn(void *arg)
     /*
      * Close all file descriptors that might have been inherited from the
      * main qemu process when doing clone, needed to make libvirt happy.
-     * Not using close_range for increased compatibility with older kernels.
      */
-    close_all_open_fd();
+    qemu_close_all_open_fd(NULL, 0);
 
     /* Set up a handler for SIGHUP and unblock SIGHUP. */
     sigaction(SIGHUP, &sa, NULL);
util/osdep.c +132 -0

135 lines not on this page · raw message

Signature · 1 line
2.45.2
#1 lore raw

Hi Clément,

Quoted 14 lines · Clément Léger

Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking.

Quoted 3 lines

----

FYI git tools parse 3 '-', not 4.

Quoted 25 lines

v4:
- Add a comment saying that qemu_close_all_fds() can take a NULL skip

    array and nskip == 0

- Added an assert in qemu_close_all_fds() to check for skip/nskip parameters
- Fix spurious tabs instead of spaces
- Applied checkpatch
- v3: https://lore.kernel.org/qemu-devel/[email protected]/

util/osdep.c +15 -0
+void qemu_close_all_open_fd(const int *skip, unsigned int nskip)
+{
+    int open_max = sysconf(_SC_OPEN_MAX);
+    unsigned int cur_skip = 0;
+    int i;
+
+    assert(skip != NULL || nskip == 0);
+
+    if (qemu_close_all_open_fd_close_range(skip, nskip)) {
+        return;
+    }
+
+    if (qemu_close_all_open_fd_proc(skip, nskip)) {
+        return;
+    }

util/osdep.c int qemu_fdatasync(int fd) · line 747

Quoted 10 lines
util/osdep.c +10 -0
+
+    /* Fallback */
+    for (i = 0; i < open_max; i++) {
+        if (cur_skip < nskip && i == skip[cur_skip]) {
+            cur_skip++;
+            continue;
+        }
+        close(i);
+    }
+}

Build failure on windows:

../util/osdep.c: In function 'qemu_close_all_open_fd':
../util/osdep.c:725:20: error: implicit declaration of function 'sysconf'; did you mean 'swscanf'? [-Wimplicit-function-declaration]

  725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                    ^~~~~~~
      |                    swscanf
../util/osdep.c:725:20: error: nested extern declaration of 'sysconf' [-Werror=nested-externs]
../util/osdep.c:725:28: error: '_SC_OPEN_MAX' undeclared (first use in this function); did you mean 'FOPEN_MAX'?
  725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                            ^~~~~~~~~~~~
      |                            FOPEN_MAX

../util/osdep.c:725:28: note: each undeclared identifier is reported only once for each function it appears in

#2 lore raw
Quoted 17 lines · Philippe Mathieu-Daudé

Hi Clément,

Quoted 14 lines · Clément Léger

Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking.

Quoted 5 lines
Quoted 3 lines

----

FYI git tools parse 3 '-', not 4.

Hi Philippe,

Thanks for the info, I was not aware of that ! I'll use 3 '-' from now on.

Quoted 40 lines
Quoted 27 lines

v4:
  - Add a comment saying that qemu_close_all_fds() can take a NULL skip array and nskip == 0
  - Added an assert in qemu_close_all_fds() to check for skip/nskip parameters
  - Fix spurious tabs instead of spaces
  - Applied checkpatch
  - v3:
https://lore.kernel.org/qemu-devel/[email protected]/

util/osdep.c +15 -0
+void qemu_close_all_open_fd(const int *skip, unsigned int nskip)
+{
+    int open_max = sysconf(_SC_OPEN_MAX);
+    unsigned int cur_skip = 0;
+    int i;
+
+    assert(skip != NULL || nskip == 0);
+
+    if (qemu_close_all_open_fd_close_range(skip, nskip)) {
+        return;
+    }
+
+    if (qemu_close_all_open_fd_proc(skip, nskip)) {
+        return;
+    }

util/osdep.c int qemu_fdatasync(int fd) · line 747

Quoted 10 lines
util/osdep.c +10 -0
+
+    /* Fallback */
+    for (i = 0; i < open_max; i++) {
+        if (cur_skip < nskip && i == skip[cur_skip]) {
+            cur_skip++;
+            continue;
+        }
+        close(i);
+    }
+}

Build failure on windows:

Quoted 16 lines

../util/osdep.c: In function 'qemu_close_all_open_fd':
../util/osdep.c:725:20: error: implicit declaration of function 'sysconf'; did you mean 'swscanf'? [-Wimplicit-function-declaration] 725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                    ^~~~~~~
      |                    swscanf
../util/osdep.c:725:20: error: nested extern declaration of 'sysconf' [-Werror=nested-externs]
../util/osdep.c:725:28: error: '_SC_OPEN_MAX' undeclared (first use in this function); did you mean 'FOPEN_MAX'?
  725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                            ^~~~~~~~~~~~
      |                            FOPEN_MAX ../util/osdep.c:725:28: note: each undeclared identifier is reported only once for each function it appears in

Should I move this chunk of code in oslib-posix.c and stub that function for win32 ? It seems like this code was not built for windows before I added it in osdep, which means it is not needed for win32.

What's your advice on that ?

Thanks,

Clément

#3 lore raw
Quoted 87 lines · Clément Léger
Quoted 17 lines · Philippe Mathieu-Daudé

Hi Clément,

Quoted 14 lines · Clément Léger

Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking.

Quoted 5 lines
Quoted 3 lines

----

FYI git tools parse 3 '-', not 4.

Hi Philippe,

Thanks for the info, I was not aware of that ! I'll use 3 '-' from now on.

Quoted 40 lines
Quoted 27 lines

v4:
  - Add a comment saying that qemu_close_all_fds() can take a NULL skip array and nskip == 0
  - Added an assert in qemu_close_all_fds() to check for skip/nskip parameters
  - Fix spurious tabs instead of spaces
  - Applied checkpatch
  - v3:
https://lore.kernel.org/qemu-devel/[email protected]/

util/osdep.c +15 -0
+void qemu_close_all_open_fd(const int *skip, unsigned int nskip)
+{
+    int open_max = sysconf(_SC_OPEN_MAX);
+    unsigned int cur_skip = 0;
+    int i;
+
+    assert(skip != NULL || nskip == 0);
+
+    if (qemu_close_all_open_fd_close_range(skip, nskip)) {
+        return;
+    }
+
+    if (qemu_close_all_open_fd_proc(skip, nskip)) {
+        return;
+    }

util/osdep.c int qemu_fdatasync(int fd) · line 747

Quoted 10 lines
util/osdep.c +10 -0
+
+    /* Fallback */
+    for (i = 0; i < open_max; i++) {
+        if (cur_skip < nskip && i == skip[cur_skip]) {
+            cur_skip++;
+            continue;
+        }
+        close(i);
+    }
+}

Build failure on windows:

Quoted 16 lines

../util/osdep.c: In function 'qemu_close_all_open_fd':
../util/osdep.c:725:20: error: implicit declaration of function 'sysconf'; did you mean 'swscanf'? [-Wimplicit-function-declaration] 725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                    ^~~~~~~
      |                    swscanf
../util/osdep.c:725:20: error: nested extern declaration of 'sysconf' [-Werror=nested-externs]
../util/osdep.c:725:28: error: '_SC_OPEN_MAX' undeclared (first use in this function); did you mean 'FOPEN_MAX'?
  725 |     int open_max = sysconf(_SC_OPEN_MAX);
      |                            ^~~~~~~~~~~~
      |                            FOPEN_MAX ../util/osdep.c:725:28: note: each undeclared identifier is reported only once for each function it appears in

Quoted 3 lines

Should I move this chunk of code in oslib-posix.c and stub that function for win32 ? It seems like this code was not built for windows before I added it in osdep, which means it is not needed for win32.

Yes, I think that'll be OK. The fork/exec concept isn't present on Windows so we also don't need to close all FDs.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#4 lore raw
In reply to Clément Léger

Hi

Quoted 14 lines · Clément Léger

Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking.

Quoted 3 lines

GLib already implemented those kinds of portable facilities.

I wonder why launch_script() is not using glib gspawn API.

async-teardown should use g_clownfrom() when glib >= 2.80.

my 2c

Signature · 1 line
Marc-André Lureau