qemu/osdep: add a qemu_close_all_open_fd() helper
v6 cover qemu/osdep: add a qemu_close_all_open_fd() helper
Since commit 03e471c41d8b ("qemu_init: increase NOFILE soft limit on POSIX"), the maximum number of file descriptors that can be opened are raised to nofile.rlim_max. On recent debian distro, this yield a maximum of 1073741816 file descriptors. Now, when forking to start qemu-bridge-helper, this actually calls close() on the full possible file descriptor range (more precisely [3 - sysconf(_SC_OPEN_MAX)]) which takes a considerable amount of time. In order to reduce that time, factorize existing code to close all open files descriptors in a new qemu_close_all_open_fd() function. This function uses various methods to close all the open file descriptors ranging from the most efficient one to the least one. It also accepts an ordered array of file descriptors that should not be closed since this is required by the callers that calls it after forking. Since this function is not used for Win32, do not implement it to force an error at link time if used.
---
v6:
- Split patch in multiple commits
- Drop Richard Henderson Reviewed-by since there was a lot of modifications
- Remove useless #ifdef LINUX in qemu_close_all_open_fd_proc()
- v5: https://lore.kernel.org/qemu-devel/[email protected]/
v5:
- Move qemu_close_all_open_fd() to oslib-posix.c since it does not compile on windows and is not even used on it.
- v4: https://lore.kernel.org/qemu-devel/[email protected]/
v4:
- Add a comment saying that qemu_close_all_open_fd() can take a NULL skip array and nskip == 0
- Added an assert in qemu_close_all_open_fd() to check for skip/nskip parameters
- Fix spurious tabs instead of spaces
- Applied checkpatch
- v3: https://lore.kernel.org/qemu-devel/[email protected]/
v3:
- Use STD*_FILENO defines instead of raw values
- Fix indentation of close_all_fds_after_fork()
- Check for nksip in fallback code
- Check for path starting with a '.' in qemu_close_all_open_fd_proc()
- Use unsigned for cur_skip
- Move ifdefs inside close_fds functions rather than redefining them
- Remove uneeded 'if(nskip)' test
- Add comments to close_range version
- Reduce range of skip fd as we find them in
- v2: https://lore.kernel.org/qemu-devel/[email protected]/
v2:
- Factorize async_teardown.c close_fds implementation as well as tap.c ones
- Apply checkpatch
- v1: https://lore.kernel.org/qemu-devel/[email protected]/
Clément Léger (5):
qemu/osdep: Move close_all_open_fds() to oslib-posix qemu/osdep: Split qemu_close_all_open_fd() and add fallback net/tap: Factorize fd closing after forking qemu/osdep: Add excluded fd parameter to qemu_close_all_open_fd() net/tap: Use qemu_close_all_open_fd()
include/qemu/osdep.h | 11 ++++ net/tap.c | 34 ++++++----- system/async-teardown.c | 37 +----------- util/oslib-posix.c | 128 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 160 insertions(+), 50 deletions(-)
Signature · 1 line
2.45.2
PATCH v6 1/5 qemu/osdep: Move close_all_open_fds() to oslib-posix
Move close_all_open_fds() in oslib-posix, rename it qemu_close_all_open_fds() and export it.
--- include/qemu/osdep.h | 7 +++++++ system/async-teardown.c | 37 +------------------------------------ util/oslib-posix.c | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 36 deletions(-)
include/qemu/osdep.h +7 -0
@@ -757,6 +757,13 @@ static inline void qemu_reset_optind(void)int qemu_fdatasync (int fd); +/** +* qemu_close_all_open_fd: +* +* Close all open file descriptors +*/ +void qemu_close_all_open_fd (void ); +/** * Sync changes made to the memory mapped file back to the backing * storage. For POSIX compliant systems this will fallback
system/async-teardown.c +1 -36
@@ -26,40 +26,6 @@static pid_t the_ppid; -/* -* Close all open file descriptors. -*/ -static void close_all_open_fd (void ) -{ -struct dirent *de; -int fd, dfd; -DIR *dir; - -#ifdef CONFIG_CLOSE_RANGE -int r =close_range (0 , ~0U ,0 ); -if (!r) { -/* Success, no need to try other ways. */ -return ; - } -#endif - - dir =opendir ("/proc/self/fd" ); -if (!dir) { -/* If /proc is not mounted, there is nothing that can be done. */ -return ; - } -/* Avoid closing the directory. */ - dfd =dirfd (dir); - -for (de =readdir (dir); de; de =readdir (dir)) { - fd =atoi (de->d_name); -if (fd != dfd) { -close (fd); - } - } -closedir (dir); -} -static void hup_handler (int signal) {/* Check every second if this process has been reparented. */
@@ -85,9 +51,8 @@ static int async_teardown_fn(void *arg)/* * Close all file descriptors that might have been inherited from the * main qemu process when doing clone, needed to make libvirt happy. -* Not using close_range for increased compatibility with older kernels. */ -close_all_open_fd (); +qemu_close_all_open_fd ();/* Set up a handler for SIGHUP and unblock SIGHUP. */ sigaction (SIGHUP , &sa,NULL );
util/oslib-posix.c +34 -0
@@ -807,3 +807,37 @@ int qemu_msync(void *addr, size_t length, int fd)return msync (addr, length,MS_SYNC ); } + +/* +* Close all open file descriptors. +*/ +void qemu_close_all_open_fd (void ) +{ +struct dirent *de; +int fd, dfd; +DIR *dir; + +#ifdef CONFIG_CLOSE_RANGE +int r =close_range (0 , ~0U ,0 ); +if (!r) { +/* Success, no need to try other ways. */ +return ; + } +#endif + + dir =opendir ("/proc/self/fd" ); +if (!dir) { +/* If /proc is not mounted, there is nothing that can be done. */ +return ; + } +/* Avoid closing the directory. */ + dfd =dirfd (dir); + +for (de =readdir (dir); de; de =readdir (dir)) { + fd =atoi (de->d_name); +if (fd != dfd) { +close (fd); + } + } +closedir (dir); +}
Signature · 1 line
2.45.2
Re: [PATCH v6 1/5] qemu/osdep: Move close_all_open_fds() to oslib-posix
Quoted 9 lines · Clément Léger
Move
close_all_open_fds()in oslib-posix, rename itqemu_close_all_open_fds()and export it.Signed-off-by Clément Léger <[email protected]>--- include/qemu/osdep.h | 7 +++++++ system/async-teardown.c | 37 +------------------------------------ util/oslib-posix.c | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 36 deletions(-)
Re: [PATCH v6 1/5] qemu/osdep: Move close_all_open_fds() to oslib-posix
In reply to Clément LégerQuoted 9 lines · Clément Léger
Move
close_all_open_fds()in oslib-posix, rename itqemu_close_all_open_fds()and export it.Signed-off-by Clément Léger<[email protected]>--- include/qemu/osdep.h | 7 +++++++ system/async-teardown.c | 37 +------------------------------------ util/oslib-posix.c | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 36 deletions(-)
r~
PATCH v6 2/5 qemu/osdep: Split qemu_close_all_open_fd() and add fallback
In reply to Clément LégerIn order to make it cleaner, split qemu_close_all_open_fd() logic into multiple subfunctions (close with close_range(), with /proc/self/fd and fallback).
--- util/oslib-posix.c | 51 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-)
util/oslib-posix.c +38 -13
@@ -808,27 +808,16 @@ int qemu_msync(void *addr, size_t length, int fd)return msync (addr, length,MS_SYNC ); } -/* -* Close all open file descriptors. -*/ -void qemu_close_all_open_fd (void ) +static bool qemu_close_all_open_fd_proc (void ) {struct dirent *de;int fd, dfd;DIR *dir; -#ifdef CONFIG_CLOSE_RANGE -int r =close_range (0 , ~0U ,0 ); -if (!r) { -/* Success, no need to try other ways. */ -return ; - } -#endif - dir =opendir ("/proc/self/fd" );if (!dir) {/* If /proc is not mounted, there is nothing that can be done. */ -return ; +return false; }/* Avoid closing the directory. */ dfd =dirfd (dir);
@@ -840,4 +829,40 @@ void qemu_close_all_open_fd(void) } }closedir (dir); + +return true; +} + +static bool qemu_close_all_open_fd_close_range (void ) +{ +#ifdef CONFIG_CLOSE_RANGE +int r =close_range (0 , ~0U ,0 ); +if (!r) { +/* Success, no need to try other ways. */ +return true; + } +#endif +return false; +} + +/* +* Close all open file descriptors. +*/ +void qemu_close_all_open_fd (void ) +{ +int open_max =sysconf (_SC_OPEN_MAX); +int i; + +if (qemu_close_all_open_fd_close_range ()) { +return ; + } + +if (qemu_close_all_open_fd_proc ()) { +return ; + } + +/* Fallback */ +for (i =0 ; i < open_max; i++) { +close (i); + } }
Signature · 1 line
2.45.2
Re: [PATCH v6 2/5] qemu/osdep: Split qemu_close_all_open_fd() and add fallback
Quoted 11 lines · Clément Léger
util/oslib-posix.c +11
-0+/* +* Close all open file descriptors. +*/ +void qemu_close_all_open_fd (void ) +{ +int open_max =sysconf (_SC_OPEN_MAX); +int i; + +if (qemu_close_all_open_fd_close_range ()) { +return ; + }
util/oslib-posix.c void qemu_close_all_open_fd(void) · line 868
Quoted 10 lines
util/oslib-posix.c +9
-0+ +if (qemu_close_all_open_fd_proc ()) { +return ; + } + +/* Fallback */ +for (i =0 ; i < open_max; i++) { +close (i); + } }
Split out fallback too, because (so far) open_max is only required within the fallback. Then
if (!qemu_close_all_open_fd_close_range() &&
!qemu_close_all_open_fd_proc()) {
qemu_close_all_open_fd_fallback();
}r~
PATCH v6 3/5 net/tap: Factorize fd closing after forking
In reply to Clément LégerThe same code is used twice to actually close all open file descriptors after forking. Factorize it in a single place.
--- net/tap.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
net/tap.c +13 -14
@@ -385,6 +385,17 @@ static TAPState *net_tap_fd_init(NetClientState *peer,return s; } +static void close_all_fds_after_fork (int excluded_fd) +{ +int open_max =sysconf (_SC_OPEN_MAX), i; + +for (i =3 ; i < open_max; i++) { +if (i != excluded_fd) { +close (i); + } + } +} +static void launch_script (const char *setup_script,const char *ifname,int fd,Error **errp) {
@@ -400,13 +411,7 @@ static void launch_script(const char *setup_script, const char *ifname,return ; }if (pid ==0 ) { -int open_max =sysconf (_SC_OPEN_MAX), i; - -for (i =3 ; i < open_max; i++) { -if (i != fd) { -close (i); - } - } +close_all_fds_after_fork (fd); parg = args; *parg++ = (char *)setup_script; *parg++ = (char *)ifname;
@@ -490,17 +495,11 @@ static int net_bridge_run_helper(const char *helper, const char *bridge,return -1 ; }if (pid ==0 ) { -int open_max =sysconf (_SC_OPEN_MAX), i;char *fd_buf =NULL ;char *br_buf =NULL ;char *helper_cmd =NULL ; -for (i =3 ; i < open_max; i++) { -if (i != sv[1 ]) { -close (i); - } - } - +close_all_fds_after_fork (sv[1 ]); fd_buf =g_strdup_printf ("%s%d" ,"--fd=" , sv[1 ]);if (strrchr (helper,' ' ) ||strrchr (helper,'\t' )) {
Signature · 1 line
2.45.2
Re: [PATCH v6 3/5] net/tap: Factorize fd closing after forking
Quoted 7 lines · Clément Léger
The same code is used twice to actually close all open file descriptors after forking. Factorize it in a single place.
Signed-off-by Clément Léger <[email protected]>--- net/tap.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
Re: [PATCH v6 3/5] net/tap: Factorize fd closing after forking
In reply to Clément LégerQuoted 7 lines · Clément Léger
The same code is used twice to actually close all open file descriptors after forking. Factorize it in a single place.
Signed-off-by Clément Léger<[email protected]>--- net/tap.c | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-)
r~
PATCH v6 4/5 qemu/osdep: Add excluded fd parameter to qemu_close_all_open_fd()
In reply to Clément LégerIn order for this function to be usable by tap.c code, add a list of file descriptors that should not be closed.
--- include/qemu/osdep.h | 8 +++- system/async-teardown.c | 2 +- util/oslib-posix.c | 99 ++++++++++++++++++++++++++++++++++------- 3 files changed, 91 insertions(+), 18 deletions(-)
include/qemu/osdep.h +6 -2
@@ -760,9 +760,13 @@ int qemu_fdatasync(int fd);/** * qemu_close_all_open_fd: * -* Close all open file descriptors +* Close all open file descriptors except the ones supplied in the @skip array +* +* @skip: ordered array of distinct file descriptors that should not be closed +* if any, or NULL. +* @nskip: number of entries in the @skip array or 0 if @skip is NULL. */ -void qemu_close_all_open_fd (void ); +void qemu_close_all_open_fd (const int *skip,unsigned int nskip);/** * Sync changes made to the memory mapped file back to the backing
system/async-teardown.c +1 -1
@@ -52,7 +52,7 @@ static int async_teardown_fn(void *arg)* Close all file descriptors that might have been inherited from the * main qemu process when doing clone, needed to make libvirt happy. */ -qemu_close_all_open_fd (); +qemu_close_all_open_fd (NULL ,0 );/* Set up a handler for SIGHUP and unblock SIGHUP. */ sigaction (SIGHUP , &sa,NULL );
util/oslib-posix.c +84 -15
139 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [PATCH v6 4/5] qemu/osdep: Add excluded fd parameter to qemu_close_all_open_fd()
util/oslib-posix.c int qemu_msync(void *addr, size_t length, int fd) · line 812
Quoted 11 lines · Clément Léger
util/oslib-posix.c +2
-1@@ -808,11 +808,14 @@ int qemu_msync(void *addr, size_t length, int fd)return msync (addr, length,MS_SYNC ); } -static bool qemu_close_all_open_fd_proc (void ) + +static bool qemu_close_all_open_fd_proc (const int *skip,unsigned int nskip)
Extra whitespace.
util/oslib-posix.c int qemu_msync(void *addr, size_t length, int fd) · line 818
Quoted 6 lines
util/oslib-posix.c +2
-0{struct dirent *de;int fd, dfd; +bool close_fd;DIR *dir; +unsigned int i, skip_start =0 , skip_end = nskip;
You can narrow the scope of close_fd and i.
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 828
Quoted 4 lines
util/oslib-posix.c +0
-0@@ -823,8 +826,31 @@ static bool qemu_close_all_open_fd_proc(void) dfd =dirfd (dir);for (de =readdir (dir); de; de =readdir (dir)) {
bool close_fd = true;
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 835
Quoted 8 lines
util/oslib-posix.c +6
-1+if (de->d_name[0 ] =='.' ) { +continue ; + } fd =atoi (de->d_name); -if (fd != dfd) { + close_fd = true; +if (fd == dfd) { + close_fd = false;
continue, which avoids
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 836
Quoted 1 line
util/oslib-posix.c +1
-0+ }else {
the else and the subsequent indentation.
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 837
Quoted 1 line
util/oslib-posix.c +1
-0+for (i = skip_start; i < skip_end; i++) {
for (unsigned int i = ...)
Quoted 20 lines
util/oslib-posix.c +16
-0+if (fd < skip[i]) { +/* We are below the next skipped fd, break */ +break ; + }else if (fd == skip[i]) { + close_fd = false; +/* Restrict the range as we found fds matching start/end */ +if (i == skip_start) { + skip_start++; + }else if (i == skip_end) { + skip_end--; + } +break ; + } + } + } +if (close_fd) {close (fd); } }
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 863
Quoted 6 lines
util/oslib-posix.c +2
-1@@ -833,36 +859,79 @@ static bool qemu_close_all_open_fd_proc(void)return true; } -static bool qemu_close_all_open_fd_close_range (void ) +static bool qemu_close_all_open_fd_close_range (const int *skip, +unsigned int nskip)
Pass in open_max, so that you don't compute it a second time...
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 866
Quoted 9 lines
util/oslib-posix.c +1
-6{#ifdef CONFIG_CLOSE_RANGE -int r =close_range (0 , ~0U ,0 ); -if (!r) { -/* Success, no need to try other ways. */ -return true; - } -#endif +int max_fd =sysconf (_SC_OPEN_MAX) -1 ;
... here
Quoted 16 lines
util/oslib-posix.c +6
-3-void qemu_close_all_open_fd (void ) +void qemu_close_all_open_fd (const int *skip,unsigned int nskip) {int open_max =sysconf (_SC_OPEN_MAX); +unsigned int cur_skip =0 ;int i; -if (qemu_close_all_open_fd_close_range ()) { +assert (skip !=NULL || nskip ==0 ); + +if (qemu_close_all_open_fd_close_range (skip, nskip)) {return ; } -if (qemu_close_all_open_fd_proc ()) { +if (qemu_close_all_open_fd_proc (skip, nskip)) {
util/oslib-posix.c static bool qemu_close_all_open_fd_proc(void) · line 935
Quoted 10 lines
util/oslib-posix.c +4
-0return ; }/* Fallback */ for (i =0 ; i < open_max; i++) { +if (cur_skip < nskip && i == skip[cur_skip]) { + cur_skip++; +continue ; + }close (i);
And pass open_max to qemu_close_all_open_fd_fallback as well.
r~
PATCH v6 5/5 net/tap: Use qemu_close_all_open_fd()
In reply to Clément LégerInstead of using a slow implementation to close all open fd after forking, use qemu_close_all_open_fd().
--- net/tap.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
net/tap.c +12 -5
@@ -387,13 +387,20 @@ static TAPState *net_tap_fd_init(NetClientState *peer,static void close_all_fds_after_fork (int excluded_fd) { -int open_max =sysconf (_SC_OPEN_MAX), i; +const int skip_fd[] = {STDIN_FILENO ,STDOUT_FILENO ,STDERR_FILENO , + excluded_fd}; +unsigned int nskip =ARRAY_SIZE (skip_fd); -for (i =3 ; i < open_max; i++) { -if (i != excluded_fd) { -close (i); - } +/* +* skip_fd must be an ordered array of distinct fds, exclude +* excluded_fd if already included in the [STDIN_FILENO - STDERR_FILENO] +* range +*/ +if (excluded_fd <=STDERR_FILENO ) { + nskip--; } + +qemu_close_all_open_fd (skip_fd, nskip); } static void launch_script(const char *setup_script, const char *ifname,
Signature · 1 line
2.45.2
Re: [PATCH v6 5/5] net/tap: Use qemu_close_all_open_fd()
Quoted 7 lines · Clément Léger
Instead of using a slow implementation to close all open fd after forking, use
qemu_close_all_open_fd().Signed-off-by Clément Léger<[email protected]>--- net/tap.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-)
r~