nextfd(2)

Currently there is no reliable way to close all opened file descriptors (which daemons need and like to do):

* dumb close(fd) loop is slow, upper bound is unknown and can be arbitrary large,

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

BSD added closefrom(fd) which is OK for this exact purpose but suboptimal on the bigger scale. closefrom(2) does only close(2) (obviously :-) closefrom(2) siletly ignores errors from close(2) which in theory is not OK for userspace.

So, don't add closefrom(2), add nextfd(2).

int nextfd(int fd)

returns next opened file descriptor which is>= than fd or -1/ESRCH if there aren't any descriptors>= than fd.

Thus closefrom(3) can be rewritten through it in userspace:

	void closefrom(int fd)
	{
		while (1) {
			fd = nextfd(fd);
			if (fd == -1&&  errno == ESRCH)
				break;
			(void)close(fd);
			fd++;
		}
	}

Maybe it will grow other smart uses.

nextfd(2) doesn't change kernel state and thus can't fail which is why it should go in. Other means may fail or
may not be available or require linear time with only guessed upper boundaries (1024, getrlimit(RLIM_NOFILE), sysconf(_SC_OPEN_MAX).

Signed-off-by Alexey Dobriyan<[email protected]>

---

  arch/x86/syscalls/syscall_32.tbl |    1 +
  arch/x86/syscalls/syscall_64.tbl |    1 +
  fs/Makefile                      |    1 +
  fs/nextfd.c                      |   27 +++++++++++++++++++++++++++
  include/linux/syscalls.h         |    1 +
  5 files changed, 31 insertions(+)

arch/x86/syscalls/syscall_32.tbl +1 -0

@@ -355,3 +355,4 @@
  346	i386	setns			sys_setns
  347	i386	process_vm_readv	sys_process_vm_readv		compat_sys_process_vm_readv
  348	i386	process_vm_writev	sys_process_vm_writev		compat_sys_process_vm_writev
+349	i386	nextfd			sys_nextfd

arch/x86/syscalls/syscall_64.tbl +1 -0

@@ -318,6 +318,7 @@
  309	common	getcpu			sys_getcpu
  310	64	process_vm_readv	sys_process_vm_readv
  311	64	process_vm_writev	sys_process_vm_writev
+312	64	nextfd			sys_nextfd
  #
  # x32-specific system call numbers start at 512 to avoid cache impact
  # for native 64-bit operation.

fs/Makefile +1 -0

@@ -12,6 +12,7 @@ obj-y :=	open.o read_write.o file_table.o super.o \
  		seq_file.o xattr.o libfs.o fs-writeback.o \
  		pnode.o drop_caches.o splice.o sync.o utimes.o \
  		stack.o fs_struct.o statfs.o
+obj-y += nextfd.o
 
  ifeq ($(CONFIG_BLOCK),y)
  obj-y +=	buffer.o bio.o block_dev.o direct-io.o mpage.o ioprio.o

fs/nextfd.c new file +27 -0

@@ -0,0 +1,27 @@
+#include<linux/errno.h>
+#include<linux/fdtable.h>
+#include<linux/rcupdate.h>
+#include<linux/sched.h>
+#include<linux/syscalls.h>
+
+/* Return first opened file descriptor which is>= than the argument. */
+SYSCALL_DEFINE1(nextfd, unsigned int, fd)
+{
+	struct files_struct *files = current->files;
+	struct fdtable *fdt;
+
+	rcu_read_lock();
+	fdt = files_fdtable(files);
+	while (fd<  fdt->max_fds) {
+		struct file *file;
+
+		file = rcu_dereference_check_fdtable(files, fdt->fd[fd]);
+		if (file) {
+			rcu_read_unlock();
+			return fd;
+		}
+		fd++;
+	}
+	rcu_read_unlock();
+	return -ESRCH;
+}

include/linux/syscalls.h +1 -1

@@ -857,5 +857,6 @@ asmlinkage long sys_process_vm_writev(pid_t pid,
  				      const struct iovec __user *rvec,
  				      unsigned long riovcnt,
  				      unsigned long flags);
+asmlinkage long sys_nextfd(unsigned int fd);
 
  #endif
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 3 lines · Alexey Dobriyan

So, don't add closefrom(2), add nextfd(2).

int nextfd(int fd)

Can we add "pid" argument to be able to search next fd in other task? Together with sys_kcmp() this will be very useful for checkpoint/restore.

Can we add "pid" argument to be able to search next fd in other task? Together with sys_kcmp() this will be very useful for checkpoint/restore.

Quoted 2 lines

Can we add "pid" argument to be able to search next fd in other task? Together with sys_kcmp() this will be very useful for checkpoint/restore.

That would raise all sorts of security questions unless protected, plus its got races. If you want to do that and you have the rights you can do it via procfs anyway.

Quoted 6 lines · Alan Cox

Quoted 2 lines

Can we add "pid" argument to be able to search next fd in other task? Together with sys_kcmp() this will be very useful for checkpoint/restore.

That would raise all sorts of security questions unless protected, plus its got races. If you want to do that and you have the rights you can do it via procfs anyway.

Don't worry, I just asked. =) Seems like crtools dumps fd in the target-task context, so the current version is suitable.

fs/nextfd.c new file +12 -0

+
+/* Return first opened file descriptor which is >= than the argument. */
+SYSCALL_DEFINE1(nextfd, unsigned int, fd)
+{
+	struct files_struct *files = current->files;
+	struct fdtable *fdt;
+
+	rcu_read_lock();
+	fdt = files_fdtable(files);
+	while (fd < fdt->max_fds) {
+		struct file *file;
+

Quoted 12 lines · Alexey Dobriyan

fs/nextfd.c new file +12 -0

+
+/* Return first opened file descriptor which is >= than the argument. */
+SYSCALL_DEFINE1(nextfd, unsigned int, fd)
+{
+	struct files_struct *files = current->files;
+	struct fdtable *fdt;
+
+	rcu_read_lock();
+	fdt = files_fdtable(files);
+	while (fd < fdt->max_fds) {
+		struct file *file;
+

fs/nextfd.c line 27

Quoted 10 lines

fs/nextfd.c new file +10 -0

+		file = rcu_dereference_check_fdtable(files, fdt->fd[fd]);
+		if (file) {
+			rcu_read_unlock();
+			return fd;
+		}
+		fd++;
+	}
+	rcu_read_unlock();
+	return -ESRCH;
+}

Interesting idea but what about using fdt->open_fds bitmap to have a fast search and less cache pollution ?

alloc_fd(start, flags) uses find_next_zero_bit(), you could use find_next_bit().

fs/nextfd.c line 27

Quoted 2 lines

fs/nextfd.c new file +2 -0

+	return -ESRCH;
+}

Interesting idea but what about using fdt->open_fds bitmap to have a fast search and less cache pollution

BSD added closefrom(fd) which is OK for this exact purpose but suboptimal on the bigger scale. closefrom(2) does only close(2) (obviously :-) closefrom(2) siletly ignores errors from close(2) which in theory is not OK for userspace.

So, don't add closefrom(2), add nextfd(2).

So, don't add closefrom(2), add nextfd(2).

int nextfd(int fd)

Sun, Apr 01, 2012 at 15:57:42, adobriyan wrote about "[PATCH] nextfd(2)":

Quoted 3 lines

So, don't add closefrom(2), add nextfd(2).

int nextfd(int fd)

Is it really needed here to create syscall? One shall update all arch lists for it. Instead, adding a new option for fcntl() (named e.g. F_NEXTFD) solves for all platforms simultaneously.

 arch/x86/syscalls/syscall_32.tbl |    1 +
 arch/x86/syscalls/syscall_64.tbl |    1 +
 fs/Makefile                      |    1 +
 fs/nextfd.c                      |   27 +++++++++++++++++++++++++++
 include/linux/syscalls.h         |    1 +
 5 files changed, 31 insertions(+)

Quoted 6 lines · Alexey Dobriyan

 arch/x86/syscalls/syscall_32.tbl |    1 +
 arch/x86/syscalls/syscall_64.tbl |    1 +
 fs/Makefile                      |    1 +
 fs/nextfd.c                      |   27 +++++++++++++++++++++++++++
 include/linux/syscalls.h         |    1 +
 5 files changed, 31 insertions(+)

I don't have any comments on the syscall itself, but when you add one, please also make the change to include/asm-generic/unistd.h so it appears in the various architectures using the generic syscall table.

Quoted 6 lines · Alexey Dobriyan

 arch/x86/syscalls/syscall_32.tbl |    1 +
 arch/x86/syscalls/syscall_64.tbl |    1 +
 fs/Makefile                      |    1 +
 fs/nextfd.c                      |   27 +++++++++++++++++++++++++++
 include/linux/syscalls.h         |    1 +
 5 files changed, 31 insertions(+)

I don't have any comments on the syscall itself, but when you add one, please also make the change to include/asm-generic/unistd.h so it appears in the various architectures using the generic syscall table.

Arnd

Quoted 8 lines · Arnd Bergmann

Quoted 6 lines · Alexey Dobriyan

 arch/x86/syscalls/syscall_32.tbl |    1 +
 arch/x86/syscalls/syscall_64.tbl |    1 +
 fs/Makefile                      |    1 +
 fs/nextfd.c                      |   27 +++++++++++++++++++++++++++
 include/linux/syscalls.h         |    1 +
 5 files changed, 31 insertions(+)

Quoted 3 lines

I don't have any comments on the syscall itself, but when you add one, please also make the change to include/asm-generic/unistd.h so it appears in the various architectures using the generic syscall table.

Arnd: do you have any interest in leveraging the syscall scripts I did for x86? I have tried to make them as generic as possible, with the hope of getting more and more of syscall information into more easily processed form.

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

Quoted 7 lines · Alexey Dobriyan

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

This is really the motivation for this... the real question is how much functionality is actually available in the system without /proc mounted, and in particular if this particular subcase is worth optimizing ... after all, if someone is maliciously setting rlimit, we can just abort (if someone can set an rlimit they can also force an abort) or revert to the slow path.

- There is a huge backwards compatibility problem with this for a substantial transition period; using /proc/self/fd has worked for a very long time already.

- Your nextfd() system call will require more system calls that the typical case for reading /proc/self/fd, because each getdents() system call handles multiple readdir() invocations.

If I've rlimited you to no file handles then you already lost whichever approach you use. Just abort.

Quoted 3 lines · Alexey Dobriyan

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place.

Quoted 1 line

Looking at /proc/self/fd requires opening directory

Quoted 2 lines

which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

(if someone can set an rlimit they can also force an abort) or revert to the slow path.

Quoted 5 lines · Alexey Dobriyan

* /proc/self/fd is unreliable:
  proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
  which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

This is really the motivation for this... the real question is how much functionality is actually available in the system without /proc mounted, and in particular if this particular subcase is worth optimizing ... after all, if someone is maliciously setting rlimit, we can just abort (if someone can set an rlimit they can also force an abort) or revert to the slow path.

Quoted 5 lines · H. Peter Anvin

Quoted 4 lines · Alexey Dobriyan

* /proc/self/fd is unreliable:
  proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory which may not be available due to malicious rlimit drop or ENOMEM

situations.

Quoted 1 line

Quoted 1 line

  Not opening directory is equivalent to dumb close(2) loop except

slower.

Quoted 2 lines

This is really the motivation for this... the real question is how

much

Quoted 1 line

functionality is actually available in the system without /proc

mounted,

Quoted 2 lines

and in particular if this particular subcase is worth optimizing ... after all, if someone is maliciously setting rlimit, we can just

abort

Quoted 1 line

(if someone can set an rlimit they can also force an abort) or revert

to

Quoted 1 line

the slow path.

Well, I imagine one typical usecase for closing all FDs is for security isolation purposes (EG: chroot()+etc), and in a great deal of chroot environments you don't have /proc available. In particular /proc has been a source of a lot of privilege escalations in the past, so avoiding mounting it in a chroot is good security policy if possible.

Cheers,
Kyle Moffett

Well, I imagine one typical usecase for closing all FDs is for security isolation purposes (EG: chroot()+etc),

and in a great deal of
chroot environments you don't have /proc available.  In particular /proc has been a source of a lot of privilege escalations in the past, so avoiding mounting it in a chroot is good security policy if possible.

Quoted 7 lines · Alexey Dobriyan

* /proc/self/fd is unreliable:
  proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
  which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

This is really the motivation for this... the real question is how much functionality is actually available in the system without /proc mounted, and in particular if this particular subcase is worth optimizing ...

But I find it little hypocritical that kernel developers add CONFIG_PROC_FS, fix compilation problems associated with it, do not mount proc by default, do not mark it unmountable somehow and
then say procless setups aren't worth it.

I haven't seen personally procless environments
but several people mentioned them including on this very list.

I agree, this particular changelog may be somewhat out of line.

But I find it little hypocritical that kernel developers add CONFIG_PROC_FS, fix compilation problems associated with it, do not mount proc by default, do not mark it unmountable somehow and
then say procless setups aren't worth it.

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches.

Quoted 3 lines · Alexey Dobriyan

I agree, this particular changelog may be somewhat out of line.

Quoted 5 lines

But I find it little hypocritical that kernel developers add CONFIG_PROC_FS, fix compilation problems associated with it, do not mount proc by default, do not mark it unmountable somehow and
then say procless setups aren't worth it.

Aren't worth *optimizing for*. But yes, CONFIG_PROC_FS is pretty much a historic relic at this point, and probably should just be dropped.

Quoted 2 lines

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches.

What? Please explain how on Earth this would "lose several failure branches."

Quoted 5 lines · H. Peter Anvin

Quoted 4 lines · Alexey Dobriyan

I agree, this particular changelog may be somewhat out of line.

But I find it little hypocritical that kernel developers add

CONFIG_PROC_FS,

Quoted 1 line

Quoted 1 line

fix compilation problems associated with it, do not mount proc by

default,

Quoted 5 lines

Quoted 3 lines

do not mark it unmountable somehow and
then say procless setups aren't worth it.

Aren't worth *optimizing for*. But yes, CONFIG_PROC_FS is pretty

much a

Quoted 1 line

historic relic at this point, and probably should just be dropped.

What to do with automounting /proc so it availablility would match syscall availability?

Quoted 1 line

Quoted 1 line

Without proc knowledge about fdtable is gathered linearly and still

unreliable.

Quoted 1 line

Quoted 1 line

With nextfd(2), even procful environments could lose several

failure branches.

Quoted 3 lines

What? Please explain how on Earth this would "lose several failure branches."

closefrom(3) written via nextfd(2) loop is reliable and doesn't fail. closefrom(3) written via /proc/self/fd is reliable and can fail (including ENOMEM).
closefrom(3) written via close(fd++) is unreliable.

If programmer adds nextfd(2) loop before any closefrom(3) code he currently uses, there will be less failures.

closefrom(3) written via nextfd(2) loop is reliable and doesn't fail. closefrom(3) written via /proc/self/fd is reliable and can fail (including ENOMEM). closefrom(3) written via close(fd++) is unreliable.

Quoted 5 lines · Alexey Dobriyan

closefrom(3) written via nextfd(2) loop is reliable and doesn't fail. closefrom(3) written via /proc/self/fd is reliable and can fail (including ENOMEM). closefrom(3) written via close(fd++) is unreliable.

I call shenanigans on this.  There is no reason to ENOMEM on the second written using the fdwalk() implementation I already posted, for example.

Quoted 7 lines · H. Peter Anvin

Quoted 5 lines · Alexey Dobriyan

closefrom(3) written via nextfd(2) loop is reliable and doesn't fail. closefrom(3) written via /proc/self/fd is reliable and can fail (including ENOMEM). closefrom(3) written via close(fd++) is unreliable.

Quoted 2 lines

I call shenanigans on this.  There is no reason to ENOMEM on the second written using the fdwalk() implementation I already posted, for example.

open("/proc/self/fd") can fail with ENOMEM.

Quoted 8 lines · On Thu, 12 Apr 2012 13:54:25 +0300 Alexey Dobriyan <[email protected]> wrote:

Quoted 7 lines · H. Peter Anvin

Quoted 5 lines · Alexey Dobriyan

closefrom(3) written via nextfd(2) loop is reliable and doesn't fail. closefrom(3) written via /proc/self/fd is reliable and can fail (including ENOMEM). closefrom(3) written via close(fd++) is unreliable.

Quoted 4 lines

Quoted 2 lines

I call shenanigans on this.  There is no reason to ENOMEM on the second written using the fdwalk() implementation I already posted, for example.

open("/proc/self/fd") can fail with ENOMEM.

Any syscall can fail with a process kill due to a stack extend on out of memory in most configurations. That includes your nextfd stuff

This whole thing is getting stupid. "Perfection is the enemy of success".

Your code will also fail when the cat pees on the computer, when the power fails and when disk dies. I suspect that other than the cat these are all more likely real world cases than your ENOMEM.

You can't see I'm talking about difference only, can you? Have you read what hpa wrote exactly and what exactly I was answering to?

He has and he is right. You're trying to catch pointless corner cases.

This whole thing is getting stupid. "Perfection is the enemy of success".

Your code will also fail when the cat pees on the computer, when the power fails and when disk dies. I suspect that other than the cat these are all more likely real world cases than your ENOMEM.

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches. And they can keep old dumb fd++ or smart /proc/self/fd loops for a change.

Quoted 5 lines · Alexey Dobriyan

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches. And they can keep old dumb fd++ or smart /proc/self/fd loops for a change.

Incidentally, if we were to create a system call for this -- which I so far see no reason for -- I would make it return a select-style bitmask of file descriptors in use, not a "next fd" which would require a system call per iteration.

Quoted 11 lines · H. Peter Anvin

Quoted 5 lines · Alexey Dobriyan

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches. And they can keep old dumb fd++ or smart /proc/self/fd loops for a change.

Incidentally, if we were to create a system call for this -- which I so far see no reason for -- I would make it return a select-style bitmask of file descriptors in use, not a "next fd" which would require a system call per iteration.

It's already possible to do something a little like that with the existing "poll" system call:

#include <stdio.h>
#include <sys/poll.h>

int
main(void)
{
 enum { N_FDS = 1024 };
 struct pollfd fds[N_FDS];

Quoted 5 lines · Alexey Dobriyan

Without proc knowledge about fdtable is gathered linearly and still unreliable. With nextfd(2), even procful environments could lose several failure branches. And they can keep old dumb fd++ or smart /proc/self/fd loops for a change.

Incidentally, if we were to create a system call for this -- which I so far see no reason for -- I would make it return a select-style bitmask of file descriptors in use, not a "next fd" which would require a system call per iteration.

I know the reason. fcntl(F_NEXT) is one of a proposal of next SUS enhancement.

http://austingroupbugs.net/view.php?id=149

nextfd() has a semantics of F_NEXT.

Next, why shoundn't we implement fcntl(F_NEXT) in our kernel? I think we have two reason.

1) As linus pointed out, linux specific "flags" argument may be useful.
2) The name of F_NEXT is not fixed yet. another url of the austin says it is FD_NEXT.
So, we can't choose right name yet. Moreover, A meanings of 3rd
argument of F_NEXT
haven't been fixed.

But it still has the same braindamage: one system call per loop invocation, and we can do better.  I would much rather see fdwalk() in SUS.

Quoted 3 lines · H. Peter Anvin

But it still has the same braindamage: one system call per loop invocation, and we can do better. I would much rather see fdwalk() in SUS.

Why would we bother to do better?

System calls are cheap, and usually you actually do want to do something about the fd, so you actually want to iterate over them.

I'd much rather have simple cheap interfaces than anything else. If SuS has a F_NEXT fcntl, let's just do that thing. Much simpler than doing something more complex and then just having to emulate the simple thing in user space anyway.

If a standard interface exists, we should just use it.

Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux (new interfaces such as pipe2( ), accept4( ), mkostemps( ), ..., and extensions like fopen(,"we")) to guarantee the atomic creation of file descriptors with the cloexec bit already set, as was already done in the standard with O_CLOEXEC in open( ) and F_DUPFD_CLOEXEC in fcntl( ). See also 0000368 for a related proposal to require CLOEXEC on hidden descriptors.

Quoted 8 lines

Quoted 2 lines

But it still has the same braindamage: one system call per loop invocation, and we can do better.  I would much rather see fdwalk() in SUS.

Why would we bother to do better?

System calls are cheap, and usually you actually do want to do something about the fd, so you actually want to iterate over them.

Quoted 7 lines

I'd much rather have simple cheap interfaces than anything else. If SuS has a F_NEXT fcntl, let's just do that thing. Much simpler than doing something more complex and then just having to emulate the simple thing in user space anyway.

If a standard interface exists, we should just use it.

I went back and looked at the post, and also the discussion on the SUS mailing list.

The proposal for FD_NEXT was rejected with some serious vitriol.

fdwalk() was considered just more palatable since there is an existing implementation (in Solaris) and since it might be possible to provide a way to hide specific fds from fdwalk(), but a much bigger issue raised is that *ALL* of these interfaces are inherently broken.  Closing random file descriptors is:

a) inherently racy in a multithreaded environment;

b) unsafe because there might be file descriptors used by libc itself.

Instead, from the resolution text:

Quoted 9 lines

Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux (new interfaces such as pipe2( ), accept4( ), mkostemps( ), ..., and extensions like fopen(,"we")) to guarantee the atomic creation of file descriptors with the cloexec bit already set, as was already done in the standard with O_CLOEXEC in open( ) and F_DUPFD_CLOEXEC in fcntl( ). See also 0000368 for a related proposal to require CLOEXEC on hidden descriptors.

I say we ask the new glibc people to provide fdwalk() since it already has an implementation history (and because it can be implemented without new system calls, thereby working on old kernels), but the *big* takeaway from this is that if there is way to create a file descriptor so that it doesn't have O_CLOEXEC set from the very beginning, *that* is what we need to fix.

Quoted 2 lines

a) inherently racy in a multithreaded environment;

I would say two things. 1) I know and I agree we _can_ misuse the interface. but many already existed interface also can be misused. 2) As I already explained
this can be used correctly.

So, I have a question. Why do you bother a possibility of misuse? Of if you didn't point out misuse, can you please point out a real world use case of multi threads + fd interation?

Quoted 2 lines

b) unsafe because there might be file descriptors used by libc itself.

I agree this. Even though almost developer don't use libc message catalogue and we can avoid such issue by using nextfd() + fcntl(O_CLOEXEC).

Yeah, I don't think fdwalk() is problematic. It's an option if I understand Alexey's mail
correctly. but I disagree almost all developers should fix a design and rewrite their
applications. In theory, they can avoid glibc or they can rewrite all of their code or
avoid linux. but there is one problem. unrealistic.

Quoted 7 lines

Quoted 1 line

a) inherently racy in a multithreaded environment;

I would say two things. 1) I know and I agree we _can_ misuse the interface. but many already existed interface also can be misused. 2) As I already explained
this can be used correctly.

Quoted 3 lines

So, I have a question. Why do you bother a possibility of misuse? Of if you didn't point out misuse, can you please point out a real world use case of multi threads + fd interation?

This were brought up in the POSIX discussion as part of why these interfaces were considered undesirable.

so a future revision of the standard may indeed add fdwalk( ), although no one in the meeting was willing to draft a proposal for fdwalk( ) at this time

Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux

Quoted 4 lines

Quoted 1 line

b) unsafe because there might be file descriptors used by libc itself.

I agree this. Even though almost developer don't use libc message catalogue and we can avoid such issue by using nextfd() + fcntl(O_CLOEXEC).

No, that's exactly the point that we cannot.

Quoted 8 lines

Yeah, I don't think fdwalk() is problematic. It's an option if I understand Alexey's mail
correctly. but I disagree almost all developers should fix a design and rewrite their
applications. In theory, they can avoid glibc or they can rewrite all of their code or
avoid linux. but there is one problem. unrealistic.

The problem -- as was brought up in the POSIX discussion -- is that you actually end up breaking *properly functioning programs*.

Hmmm.... I'm sorry I don't find "considered undesirable". Maybe because my English is not very good. can you please help me clarify?

And all these programs and systems are wrong.

There is no guarantee that one of the fds isn't used behind the scenes for something important which is still running as part of the fork/exec code. It's completely unacceptable to build into the interfaces the assumption that the programmer knows all the file descriptors.

This is why using CLOEXEC is the only correct way to deal with this and now there is no exceuse anymore whatsoever. Every fd-creating interface can use CLOEXEC.

This text says,

Quoted 2 lines

so a future revision of the standard may indeed add fdwalk( ), although no one in the meeting was willing to draft a proposal for fdwalk( ) at this time

and, later says after noting F_NEXT and O_CLOEXEC,

Quoted 4 lines

Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux

Do you think latter override former?

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

b) unsafe because there might be file descriptors used by libc itself.

I agree this. Even though almost developer don't use libc message catalogue and we can avoid such issue by using nextfd() + fcntl(O_CLOEXEC).

No, that's exactly the point that we cannot.

I thknk we are talking different aspect. I'm talking practical issue. say, ruby hit the exact same issue
because valgrind uses internal fds and they don't think their exec() case don't need fd
inheritance. Even though it close libc internal fds, invoked new executable may open them
again at process strtup code. Therefore, they are using O_CLOEXEC. In the other hands,
you seems talking about it is corner case. If so, I agree. I was not argue it. I only say, I
haven't seen real world application require it.

Personally, I'm only interesting real world issue.

Quoted 2 lines

The problem -- as was brought up in the POSIX discussion -- is that you actually end up breaking *properly functioning programs*.

But the url only talk about a possibility of misuse.

Quoted 4 lines · KOSAKI Motohiro

Hmmm.... I'm sorry I don't find "considered undesirable". Maybe because my English is not very good. can you please help me clarify?

I also went and read the mailing list discussion on the topic.

Ulrich, for example (in his usual mild-mannered style), commented:

Quoted 17 lines

And all these programs and systems are wrong.

There is no guarantee that one of the fds isn't used behind the scenes for something important which is still running as part of the fork/exec code. It's completely unacceptable to build into the interfaces the assumption that the programmer knows all the file descriptors.

This is why using CLOEXEC is the only correct way to deal with this and now there is no exceuse anymore whatsoever. Every fd-creating interface can use CLOEXEC.

This text says,

Quoted 2 lines

so a future revision of the standard may indeed add fdwalk( ), although no one in the meeting was willing to draft a proposal for fdwalk( ) at this time

Quoted 8 lines

and, later says after noting F_NEXT and O_CLOEXEC,

Quoted 4 lines

Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux

Do you think latter override former?

Yes.

Quoted 19 lines

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

b) unsafe because there might be file descriptors used by libc itself.

I agree this. Even though almost developer don't use libc message catalogue and we can avoid such issue by using nextfd() + fcntl(O_CLOEXEC).

No, that's exactly the point that we cannot.

I thknk we are talking different aspect. I'm talking practical issue. say, ruby hit the exact same issue
because valgrind uses internal fds and they don't think their exec() case don't need fd
inheritance. Even though it close libc internal fds, invoked new executable may open them
again at process strtup code. Therefore, they are using O_CLOEXEC. In the other hands,
you seems talking about it is corner case. If so, I agree. I was not argue it. I only say, I
haven't seen real world application require it.

Quoted 1 line

Personally, I'm only interesting real world issue.

These are real-world issues.

Quoted 3 lines

Quoted 2 lines

The problem -- as was brought up in the POSIX discussion -- is that you actually end up breaking *properly functioning programs*.

Quoted 1 line

But the url only talk about a possibility of misuse.

There are concrete examples on the mailing list.

Anyway, fdwalk() at least exists as an interface. There is absolutely no momentum for FD_NEXT that I can see.

I know the reason. fcntl(F_NEXT) is one of a proposal of next SUS enhancement.

http://austingroupbugs.net/view.php?id=149

nextfd() has a semantics of F_NEXT.

Quoted 7 lines · KOSAKI Motohiro

I know the reason. fcntl(F_NEXT) is one of a proposal of next SUS enhancement.

  http://austingroupbugs.net/view.php?id=149

nextfd() has a semantics of F_NEXT.

Looking at that bug report, it sounds like that proposal was rejected.

If I understand correctly, above URL says, latest satatus is

Resolution: Accepted As Marked

Please see toppest issue summary. Or if I am missing something, can you please point out which line are you seeing?

Quoted 7 lines · KOSAKI Motohiro

If I understand correctly, above URL says, latest satatus is

  Resolution: Accepted As Marked

Please see toppest issue summary. Or if I am missing something, can you please point out which line are you seeing?

You need to read through the entire set of notes for it to make sense.

The note described several downsides. but I have no seen a rejection.

Quoted 1 line · KOSAKI Motohiro

The note described several downsides. but I have no seen a rejection.

The Aardvark resolution "Accept as Marked" means "make the changes in the markup."  The markup doesn't include any new interfaces, but just a set of warnings.

Quoted 6 lines · H. Peter Anvin

Quoted 1 line · KOSAKI Motohiro

The note described several downsides. but I have no seen a rejection.

The Aardvark resolution "Accept as Marked" means "make the changes in the markup." The markup doesn't include any new interfaces, but just a set of warnings.

Hm. Now are you saying "make the changes in the markup" mean rejection? I'm sorry I haven't caught your point.

Currently there is no reliable way to close all opened file descriptors (which daemons need and like to do):

* dumb close(fd) loop is slow, upper bound is unknown and can be arbitrary large,

* /proc/self/fd is unreliable:
 proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
 which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

BSD added closefrom(fd) which is OK for this exact purpose but suboptimal on the bigger scale. closefrom(2) does only close(2) (obviously :-) closefrom(2) siletly ignores errors from close(2) which in theory is not OK for userspace.

So, don't add closefrom(2), add nextfd(2).

       int nextfd(int fd)

returns next opened file descriptor which is >= than fd or -1/ESRCH if there aren't any descriptors >= than fd.

Thus closefrom(3) can be rewritten through it in userspace:

       void closefrom(int fd)
       {
               while (1) {
                       fd = nextfd(fd);
                       if (fd == -1 && errno == ESRCH) break; (void)close(fd);
                       fd++;
               }
       }

Maybe it will grow other smart uses.

nextfd(2) doesn't change kernel state and thus can't fail which is why it should go in. Other means may fail or
may not be available or require linear time with only guessed upper boundaries (1024, getrlimit(RLIM_NOFILE), sysconf(_SC_OPEN_MAX).

Sorry for the long delay comment. I realized this thread now. I think /proc no mount case is not good explanation for the worth of this patch. The problem is, we can't use opendir() after fork() if an app has multi threads.

SUS clearly say so,
http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html

we can only call async-signal-safe functions after fork() when multi threads and opendir() call malloc() internally.

As far as I know, OpenJDK has a such fork-readdir-exec code and it can make deadlock
when spawnning a new process. Unfortunately Java language perfeter to make a lot of threads rather than other language.

This patch can solve such multi threaded case.

offtopic, glibc malloc is a slightly clever. It reinitialize its
internal lock when fork by using thread_atfork() hook. It mean glibc malloc can be used after fork() and the technique can avoid this issue. But, glibc malloc still has several performance problem and many people prefer to use jemalloc or google malloc instead. Then, they hit an old issue, bah.

and I've received a request that linux aim fdwalk() several times. Example,

You can deal with this in one of two ways:

2. Fix your malloc().
1. Use the low level open()/getdents()/close() functions instead of opendir()/readdir()/closedir().

Quoted 3 lines · H. Peter Anvin

You can deal with this in one of two ways:

Quoted 3 lines

2. Fix your malloc().
1. Use the low level open()/getdents()/close() functions instead of opendir()/readdir()/closedir().

And if their is concern about using opendir() then we can add something to posix_spawn. And attribute like POSIX_SPAWN_CLOSEALL and you then would have to explicitly add dup and open requests for the descriptors you want to have open.

As far as I understand, any major open source project don't use posix_spawn().
Please remind, I'm talking about real world issue.

Quoted 3 lines · KOSAKI Motohiro

As far as I understand, any major open source project don't use posix_spawn().
Please remind, I'm talking about real world issue.

This doesn't mean they shouldn't. If you require code to be changed anyway let them change to something which doesn't require more cruft in the kernel. The limitations you cited are irrelevant for posix_spawn. And perhaps there will be actually spawn support in the kernel which would make dealing with OOM situations and non-overcommit much easier.

Umm... I'm sorry. I haven't catch why OOM is related topic. Could you please elaborate more?

Quoted 2 lines · KOSAKI Motohiro

Umm... I'm sorry. I haven't catch why OOM is related topic. Could you please elaborate more?

With fork you always have some copy-on-write (and worse for overcommit) just to then execute exec. With a real spawn implementation you wouldn't have that. A big problem if you, for instance, have to spawn a small helper from a gigantic process.

Quoted 16 lines · KOSAKI Motohiro

Sorry for the long delay comment. I realized this thread now. I think /proc no mount case is not good explanation for the worth of this patch. The problem is, we can't use opendir() after fork() if an app has multi threads.

SUS clearly say so,
http://pubs.opengroup.org/onlinepubs/009695399/functions/fork.html

we can only call async-signal-safe functions after fork() when multi threads and opendir() call malloc() internally.

As far as I know, OpenJDK has a such fork-readdir-exec code and it can make deadlock
when spawnning a new process. Unfortunately Java language perfeter to make a lot of threads rather than other language.

Quoted 8 lines

This patch can solve such multi threaded case.

offtopic, glibc malloc is a slightly clever. It reinitialize its
internal lock when fork by using thread_atfork() hook. It mean glibc malloc can be used after fork() and the technique can avoid this issue. But, glibc malloc still has several performance problem and many people prefer to use jemalloc or google malloc instead. Then, they hit an old issue, bah.

OK, so what you're saying here is:

Linux doesn't actually have a problem unless:
1. You use the library implementation of opendir/readdir/closedir;
2. You use a nonstandard malloc for the platform which doesn't

    correctly set up fork hooks (which I would consider a bug);

You can deal with this in one of two ways:

2. Fix your malloc().
1. Use the low level open()/getdents()/close() functions instead of

    opendir()/readdir()/closedir().

Quoted 1 line

and I've received a request that linux aim fdwalk() several times. Example,

It doesn't sound very hard to implement fdwalk() in terms of open/getdents/close without using malloc; since the fdwalk() interface lets you use the stack for storage. You can then implement closefrom() in terms of fdwalk(). Something like this (untested):

int fdwalk(int (*func)(void *, int), void *cd)

{
	char buf[4096];	/* ... could be less... */
	const char *p, *q;
	const struct linux_dirent *dp
	int dfd, fd;
	unsigned char c;
	int rv = 0;
	int sz;

	dfd = open("/proc/self/fd", O_RDONLY|O_DIRECTORY|O_CLOEXEC);
	if (dfd<  0)
		return -1;

	/*** XXX: may want to check for procfs magic here ***/

	while ((sz = getdents(dfd, buf, sizeof buf))>  0) {
		p = buf;

		while (sz>  offsetof(struct linux_dirent, d_name)) {
			dp = (const struct linux_dirent *)p;

			if (sz<  dp->d_reclen)
				break;

			q   = dp->d_name;
			p  += dp->d_reclen;
			sz -= dp->d_reclen;

			fd = 0;
			while (q<  p&&  (c = *q++)) {
				c -= '0';
				if (c>= 10)
					goto skip;
				fd = fd*10 + c;
			}

			if (fd != dfd)
				rv = func(cd, fd);
		skip:
			;

		}
	}

	if (close(dfd))
		return -1;

	return rv;
}

Ideally possible. but practically impossible. 2) people don't use a their own malloc. they only uses open sources alternative malloc. And, I think you have too narrowing concern. Even though malloc people adds a workaround, the standard inhibit to use it

and people may continue to use more dangerous
RLIM_NOFILE loop. 1) I haven't seen _practical_ userland software uses such linux internal hacking. Almost all major software can run on multiple OSs.

The thing is, even if it were added today, since we need to run on old kernels, we'd have to carry the code to use the /proc trick approximately forever. And in the end all nextfd() would accomplish would be a *third* case in the already messy ifdefs/fallbacks in the various implementations of process spawning.

Quoted 4 lines · KOSAKI Motohiro

Ideally possible. but practically impossible. 2) people don't use a their own malloc. they only uses open sources alternative malloc. And, I think you have too narrowing concern. Even though malloc people adds a workaround, the standard inhibit to use it

What do you mean? If as hpa says, the maintainer of e.g. google tcmalloc added a call to pthread_atfork(), then code which uses opendir() would start working.

Quoted 3 lines

and people may continue to use more dangerous
RLIM_NOFILE loop. 1) I haven't seen _practical_ userland software uses such linux internal hacking. Almost all major software can run on multiple OSs.

Except that if you're using /proc/self/fd, you're already relying on Linux-specific functionality. So it's not burdensome to use "struct linux_dirent" and O_DIRECTORY either.

In GLib we're presently doing the regular /proc+opendir() under #ifdef __linux__:
http://git.gnome.org/browse/glib/tree/glib/gspawn.c#n932

Now I'd happily switch to hpa's fdwalk() implementation if I was aware of someone using glib in combination with an alternative malloc hitting this problem.

Basically I think hpa is right here, and it's not really worth adding a new system call.

The thing is, even if it were added today, since we need to run on old kernels, we'd have to carry the code to use the /proc trick approximately forever. And in the end all nextfd() would accomplish would be a *third* case in the already messy ifdefs/fallbacks in the various implementations of process spawning.

Quoted 7 lines

OK, so what you're saying here is:

Linux doesn't actually have a problem unless:
1. You use the library implementation of opendir/readdir/closedir;
2. You use a nonstandard malloc for the platform which doesn't

    correctly set up fork hooks (which I would consider a bug);

Right. but I'm argue "correctly set up" term because SUS/POSIX don't require it.
It is only a workaround of buggy userland in glibc. SUS still says you can't
use opendir and typical userland people don't want ignore SUS as far as possible.

It can. but more ugly. no?

we can only call async-signal-safe functions after fork() when multi threads and opendir() call malloc() internally.

As far as I know, OpenJDK has a such fork-readdir-exec code and it can make deadlock
when spawnning a new process. Unfortunately Java language perfeter to make a lot of threads rather than other language.

This patch can solve such multi threaded case.

offtopic, glibc malloc is a slightly clever. It reinitialize its internal lock when fork by
using thread_atfork() hook. It mean glibc malloc can be used after fork() and the
technique can avoid this issue.

Currently there is no reliable way to close all opened file descriptors (which daemons need and like to do):

* dumb close(fd) loop is slow, upper bound is unknown and can be arbitrary large,

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

BSD added closefrom(fd) which is OK for this exact purpose but suboptimal on the bigger scale. closefrom(2) does only close(2) (obviously :-) closefrom(2) siletly ignores errors from close(2) which in theory is not OK for userspace.

So, don't add closefrom(2), add nextfd(2).

Quoted 12 lines · Alexey Dobriyan

Currently there is no reliable way to close all opened file descriptors (which daemons need and like to do):

* dumb close(fd) loop is slow, upper bound is unknown and can be arbitrary large,

* /proc/self/fd is unreliable:
proc may be unconfigured or not mounted at expected place. Looking at /proc/self/fd requires opening directory
which may not be available due to malicious rlimit drop or ENOMEM situations. Not opening directory is equivalent to dumb close(2) loop except slower.

Quoted 6 lines

BSD added closefrom(fd) which is OK for this exact purpose but suboptimal on the bigger scale. closefrom(2) does only close(2) (obviously :-) closefrom(2) siletly ignores errors from close(2) which in theory is not OK for userspace.

So, don't add closefrom(2), add nextfd(2).

Or unshare(CLONE_FILES_EMPTY) to steal an idea from rfork(2) (Plan 9 one, that is - I don't remember if its *BSD analog has that). Basically, they allow 3 kinds of behaviour on clone(2) analog (and unshare(2) is part of the same thing there):
1) share descriptor table with parent (default for rfork(2))
2) copy descriptor table from parent (RFFDG is set in flags)
3) give child an empty descriptor table (RFCFDG is set in flags)
They have something similar for namespace, BTW - the same share/copy/clean triple.