Skip to contentSkip to conversation

Add dummy definition of O_CLOEXEC

PATCH In review Thomas De Schampheleire 3 participants 10 messages Sep 30, 2014 – Oct 7, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

This patch is much more like a workaround, since it may need fcntl() use FD_CLOEXEC to replace.

This problem was reported by "Ting Liu <[email protected]>"

[Thomas De Schampheleire <[email protected]:
- move dummy definition from libkmod-internal.h to missing.h
- update commit title]

---
 libkmod/missing.h |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
libkmod/missing.h +4 -0
index 4c0d136..e123e98 100644
@@ -19,6 +19,10 @@
 # define __NR_finit_module -1
 #endif
 
+#ifndef O_CLOEXEC
+#define O_CLOEXEC 0
+#endif
+
 #ifndef HAVE_FINIT_MODULE
 #include <errno.h>
 
Signature · 1 line
1.7.1
#1 lore raw
Quoted 4 lines · Thomas De Schampheleire

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

Humn... Do we really want to support kernels older than 2.6.23?

Adding a workaround like this IMO will just hide bugs because we rely on O_CLOEXEC semantics. Doing nothing is not really what we want. Maybe if ancient downstream distros want the workaround they can define O_CLOEXEC by themselves during build... passing it in CFLAGS should work

Signature · 1 line
Lucas De Marchi
#2 lore raw
Quoted 1 line · Lucas De Marchi

Humn... Do we really want to support kernels older than 2.6.23?

No, not even pre-systemd udev supports them.

Signature · 2 lines
ciao,
Marco
#3 lore raw
In reply to Lucas De Marchi

Lucas De Marchi <[email protected]> schreef:

Quoted 7 lines
Quoted 4 lines · Thomas De Schampheleire

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

Quoted 8 lines

Humn... Do we really want to support kernels older than 2.6.23?

Adding a workaround like this IMO will just hide bugs because we rely on O_CLOEXEC semantics. Doing nothing is not really what we want. Maybe if ancient downstream distros want the workaround they can define O_CLOEXEC by themselves during build... passing it in CFLAGS should work

This is the same type of distro for which the
implementation of be32toh was needed: RHEL5.
Ancient, yes, but still supported and used in corporate
environments, also to build modern systems, for example using Buildroot or Openembedded.

The patch comes from openembedded and is about
to be integrated in Buildroot too, but it's far more advantageous to have such changes integrated upstream.

Thanks,
Thomas

#4 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Quoted 17 lines · Thomas De Schampheleire

Lucas De Marchi <[email protected]> schreef:

Quoted 7 lines
Quoted 4 lines · Thomas De Schampheleire

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

Quoted 8 lines

Humn... Do we really want to support kernels older than 2.6.23?

Adding a workaround like this IMO will just hide bugs because we rely on O_CLOEXEC semantics. Doing nothing is not really what we want. Maybe if ancient downstream distros want the workaround they can define O_CLOEXEC by themselves during build... passing it in CFLAGS should work

Quoted 2 lines

This is the same type of distro for which the
implementation of be32toh was needed: RHEL5.

Similar, but not the same. For the implementation of be32toh we depend on *libc* having it. As I said in the original email, I was surprised a libc could possibly miss it, but I was ok with adding a fallback implementation (maybe we should even go one step further and do what systemd does to check our use cases with sparse).

Quoted 3 lines

Ancient, yes, but still supported and used in corporate
environments, also to build modern systems, for example using Buildroot or Openembedded.

That's my worry. If you are building modern systems and you don't have O_CLOEXEC (and assuming you are not trying to put 2.6.22 in your embedded system), I'd say you are using the wrong headers, from host instead of target.

Quoted 4 lines

The patch comes from openembedded and is about
to be integrated in Buildroot too, but it's far more advantageous to have such changes integrated upstream.

They shouldn't be defining O_CLOEXEC to 0 in the first place. I don't want to support leaking file descriptors to child process. You are silently giving different behavior to your target depending on the machine it was built with :-o.

CC'ing buildroot mailing list. Peter, do you do this for other packages as well?

Signature · 1 line
Lucas De Marchi
#5 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Quoted 29 lines · Lucas De Marchi
Quoted 17 lines · Thomas De Schampheleire

Lucas De Marchi <[email protected]> schreef:

Quoted 7 lines
Quoted 4 lines · Thomas De Schampheleire

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

Quoted 8 lines

Humn... Do we really want to support kernels older than 2.6.23?

Adding a workaround like this IMO will just hide bugs because we rely on O_CLOEXEC semantics. Doing nothing is not really what we want. Maybe if ancient downstream distros want the workaround they can define O_CLOEXEC by themselves during build... passing it in CFLAGS should work

Quoted 2 lines

This is the same type of distro for which the
implementation of be32toh was needed: RHEL5.

Similar, but not the same. For the implementation of be32toh we depend on *libc* having it. As I said in the original email, I was surprised a libc could possibly miss it, but I was ok with adding a fallback implementation (maybe we should even go one step further and do what systemd does to check our use cases with sparse).

Quoted 8 lines
Quoted 3 lines

Ancient, yes, but still supported and used in corporate
environments, also to build modern systems, for example using Buildroot or Openembedded.

That's my worry. If you are building modern systems and you don't have O_CLOEXEC (and assuming you are not trying to put 2.6.22 in your embedded system), I'd say you are using the wrong headers, from host instead of target.

In Buildroot, we're not leaking host kernel headers into the target. The kernel headers used for target compilation are those coming with the toolchain, which are recent kernel headers supporting O_CLOEXEC.

The problem I'm encountering (and people on OpenEmbedded apparently have too) is when building host-kmod, that is the kmod tools that run on the host system (depmod --> kmod). These are (in buildroot at least) built using the toolchain on the host system, and thus with the kernel headers from the host system.
On RHEL5 systems these are old kernel headers from pre-O_CLOEXEC times.

Quoted 11 lines
Quoted 5 lines

The patch comes from openembedded and is about
to be integrated in Buildroot too, but it's far more advantageous to have such changes integrated upstream.

They shouldn't be defining O_CLOEXEC to 0 in the first place. I don't want to support leaking file descriptors to child process. You are silently giving different behavior to your target depending on the machine it was built with :-o.

As explained above, this is not what is happening. The O_CLOEXEC dummy definition to 0 is only relevant when building host tools (depmod) on old systems like RHEL5. These host tools are effectively run on the same host where they are built. When building for the target, or when building on modern host systems, the dummy does not set in and there is anyway no impact.

Hope this clarifies,

Thomas

#6 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Quoted 63 lines · Thomas De Schampheleire
Quoted 29 lines · Lucas De Marchi
Quoted 17 lines · Thomas De Schampheleire

Lucas De Marchi <[email protected]> schreef:

Quoted 7 lines
Quoted 4 lines · Thomas De Schampheleire

From: Robert Yang <[email protected]>

O_CLOEXEC is introduced from Linux 2.6.23, so old kernel doesn't have it, we need check before use.

Quoted 8 lines

Humn... Do we really want to support kernels older than 2.6.23?

Adding a workaround like this IMO will just hide bugs because we rely on O_CLOEXEC semantics. Doing nothing is not really what we want. Maybe if ancient downstream distros want the workaround they can define O_CLOEXEC by themselves during build... passing it in CFLAGS should work

Quoted 2 lines

This is the same type of distro for which the
implementation of be32toh was needed: RHEL5.

Similar, but not the same. For the implementation of be32toh we depend on *libc* having it. As I said in the original email, I was surprised a libc could possibly miss it, but I was ok with adding a fallback implementation (maybe we should even go one step further and do what systemd does to check our use cases with sparse).

Quoted 8 lines
Quoted 3 lines

Ancient, yes, but still supported and used in corporate
environments, also to build modern systems, for example using Buildroot or Openembedded.

That's my worry. If you are building modern systems and you don't have O_CLOEXEC (and assuming you are not trying to put 2.6.22 in your embedded system), I'd say you are using the wrong headers, from host instead of target.

In Buildroot, we're not leaking host kernel headers into the target. The kernel headers used for target compilation are those coming with the toolchain, which are recent kernel headers supporting O_CLOEXEC.

The problem I'm encountering (and people on OpenEmbedded apparently have too) is when building host-kmod, that is the kmod tools that run on the host system (depmod --> kmod). These are (in buildroot at least) built using the toolchain on the host system, and thus with the kernel headers from the host system.
On RHEL5 systems these are old kernel headers from pre-O_CLOEXEC times.

Quoted 11 lines
Quoted 5 lines

The patch comes from openembedded and is about
to be integrated in Buildroot too, but it's far more advantageous to have such changes integrated upstream.

They shouldn't be defining O_CLOEXEC to 0 in the first place. I don't want to support leaking file descriptors to child process. You are silently giving different behavior to your target depending on the machine it was built with :-o.

Quoted 8 lines

As explained above, this is not what is happening. The O_CLOEXEC dummy definition to 0 is only relevant when building host tools (depmod) on old systems like RHEL5. These host tools are effectively run on the same host where they are built. When building for the target, or when building on modern host systems, the dummy does not set in and there is anyway no impact.

Hope this clarifies,

Yep, this clarifies, thanks.

However it only makes sense for this specific scenario, i.e. you don't care for depmod leaking fds on the host... It's not something acceptable to do on upstream, sorry.

Signature · 1 line
Lucas De Marchi
#7 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Hi Lucas,

Quoted 12 lines · Lucas De Marchi
Quoted 12 lines
Quoted 4 lines

They shouldn't be defining O_CLOEXEC to 0 in the first place. I don't want to support leaking file descriptors to child process. You are silently giving different behavior to your target depending on the machine it was built with :-o.

As explained above, this is not what is happening. The O_CLOEXEC dummy definition to 0 is only relevant when building host tools (depmod) on old systems like RHEL5. These host tools are effectively run on the same host where they are built. When building for the target, or when building on modern host systems, the dummy does not set in and there is anyway no impact.

Quoted 7 lines
Quoted 1 line

Hope this clarifies,

Yep, this clarifies, thanks.

However it only makes sense for this specific scenario, i.e. you don't care for depmod leaking fds on the host... It's not something acceptable to do on upstream, sorry.

I'm trying to understand the reason you object to this patch.

We both agree that the dummy implementation of O_CLOEXEC will only take effect when building kmod using kernel headers < 2.6.23, correct?

In such a case, there are three paths:

1. Don't make any change (the current situation). In this case, kmod cannot be compiled due to the missing O_CLOEXEC definition and thus cannot be used@all.

2. Add a dummy definition (as proposed by the patch) to allow compilation of kmod. In this case, the O_CLOEXEC flag will not take effect, indeed with the leaking file descriptors into child processes, but only in this exceptional case where kmod is built for older kernels.

3. Use explicit fcntl(..., FD_CLOEXEC...) calls after the open's. However, this is not easily done without impact on the standard case with modern headers. I assume you will like this even less.

To me, case 2 seems a valid solution to an otherwise unusable kmod on systems with old kernels / kernel headers. The impact of the leaked file descriptors is to be accepted in this case.

However, if you still feel this is not upstreamable, then I will back off and hope that Peter will accept the patch in Buildroot :)

Thanks,
Thomas

#8 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Quoted 21 lines · Thomas De Schampheleire

Hi Lucas,

Quoted 16 lines · Lucas De Marchi
Quoted 5 lines
Quoted 4 lines

They shouldn't be defining O_CLOEXEC to 0 in the first place. I don't want to support leaking file descriptors to child process. You are silently giving different behavior to your target depending on the machine it was built with :-o.

Quoted 8 lines

As explained above, this is not what is happening. The O_CLOEXEC dummy definition to 0 is only relevant when building host tools (depmod) on old systems like RHEL5. These host tools are effectively run on the same host where they are built. When building for the target, or when building on modern host systems, the dummy does not set in and there is anyway no impact.

Hope this clarifies,

Yep, this clarifies, thanks.

Quoted 8 lines
Quoted 3 lines

However it only makes sense for this specific scenario, i.e. you don't care for depmod leaking fds on the host... It's not something acceptable to do on upstream, sorry.

I'm trying to understand the reason you object to this patch.

We both agree that the dummy implementation of O_CLOEXEC will only take effect when building kmod using kernel headers < 2.6.23, correct?

yes. But I don't want to pretend that kmod works fine there. It doesn't.

Quoted 11 lines

In such a case, there are three paths:

1. Don't make any change (the current situation). In this case, kmod cannot be compiled due to the missing O_CLOEXEC definition and thus cannot be used at all.

2. Add a dummy definition (as proposed by the patch) to allow compilation of kmod. In this case, the O_CLOEXEC flag will not take effect, indeed with the leaking file descriptors into child processes, but only in this exceptional case where kmod is built for older kernels.

The problem here is that you are silently accepting the misbehave.

Quoted 8 lines

3. Use explicit fcntl(..., FD_CLOEXEC...) calls after the open's. However, this is not easily done without impact on the standard case with modern headers. I assume you will like this even less.

To me, case 2 seems a valid solution to an otherwise unusable kmod on systems with old kernels / kernel headers. The impact of the leaked file descriptors is to be accepted in this case.

I'm all for accepting the leaked file descriptors *in this case*. Not in the general case in which one takes kmod and realizes it works fine in < 2.6.23. Because it doesn't and pretending it does is just asking for invalid bug reports.

So as I said, I don't think this is material for upstream.

What we could do is to hide the definition of O_CLOEXEC behind a configure flag... but then the amount of ifdef, the benefits from it and having to maintain that on build sys don't pay off IMO.

Quoted 2 lines

However, if you still feel this is not upstreamable, then I will back off and hope that Peter will accept the patch in Buildroot :)

I do think this could be accepted there. AFAICS he already acked on it. Just make sure this is a patch that gets applied on host-kmod only, not the one running on target. Let's try this approach and come back to what I suggested above if it becomes too troublesome to downstream?

Signature · 1 line
Lucas De Marchi
#9 lore raw

PATCH 1/1 Add dummy definition of O_CLOEXEC

Hi Lucas,

Quoted 42 lines · Lucas De Marchi
Quoted 5 lines

I'm trying to understand the reason you object to this patch.

We both agree that the dummy implementation of O_CLOEXEC will only take effect when building kmod using kernel headers < 2.6.23, correct?

yes. But I don't want to pretend that kmod works fine there. It doesn't.

Quoted 11 lines

In such a case, there are three paths:

1. Don't make any change (the current situation). In this case, kmod cannot be compiled due to the missing O_CLOEXEC definition and thus cannot be used at all.

2. Add a dummy definition (as proposed by the patch) to allow compilation of kmod. In this case, the O_CLOEXEC flag will not take effect, indeed with the leaking file descriptors into child processes, but only in this exceptional case where kmod is built for older kernels.

The problem here is that you are silently accepting the misbehave.

Quoted 8 lines

3. Use explicit fcntl(..., FD_CLOEXEC...) calls after the open's. However, this is not easily done without impact on the standard case with modern headers. I assume you will like this even less.

To me, case 2 seems a valid solution to an otherwise unusable kmod on systems with old kernels / kernel headers. The impact of the leaked file descriptors is to be accepted in this case.

I'm all for accepting the leaked file descriptors *in this case*. Not in the general case in which one takes kmod and realizes it works fine in < 2.6.23. Because it doesn't and pretending it does is just asking for invalid bug reports.

So as I said, I don't think this is material for upstream.

What we could do is to hide the definition of O_CLOEXEC behind a configure flag... but then the amount of ifdef, the benefits from it and having to maintain that on build sys don't pay off IMO.

Quoted 8 lines
Quoted 2 lines

However, if you still feel this is not upstreamable, then I will back off and hope that Peter will accept the patch in Buildroot :)

I do think this could be accepted there. AFAICS he already acked on it. Just make sure this is a patch that gets applied on host-kmod only, not the one running on target. Let's try this approach and come back to what I suggested above if it becomes too troublesome to downstream?

Thanks for your further explanation.
I will push the patch to buildroot, as suggested.

Best regards,
Thomas